3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13)

Taxonomies of User Experience (UX) Evaluation Methods

S. Rajeshkumar Department of Graphics and Multimedia

College of Information Technology Universiti Tenaga Nasional

E-mail: rajeshburn_6313@yahoo.com.my

Ridha Omar Department of Graphics and Multimedia

College of Information Technology Universiti Tenaga Nasional

E-mail: ridha@uniten.edu.my

Murni Mahmud Department of Information Systems

Kulliyyah of Information and Communication Technology International Islamic University Malaysia

P.O Box 10, 50728, Kuala Lumpur, Malaysia. E-mail: murni@iium.edu.my

Abstract – User Experiences (UX) and Usability are tightly coupled. They are yardsticks for successful interaction designs. Various methods are available to evaluate UX and usability respectively. Many arguments are presented in order to either distinguish or identify common grounds between the two. These evaluation methods are used in various different contexts and environments, and they fall into different types of taxonomy. Several UX and usability evaluation methods were reviewed and summarized in this paper. The result of analysis and observation indicates that there are correlations and disassociations between UX and Usability.

Keywords-User Experience, UX, Usability, User Experience Evaluation Methods, Correlation, Disassociation

I. INTRODUCTION Technology for interaction and interactivity has evolved

exponentially over the last three decades, since the early days of the Xerox Star workstation. The User Interface Design (UID) process has skewed towards human-centred design. This is clearly shown through the development of Human Factors studies, Ergonomics, the Human-Computer Interaction (HCI) discipline and the concept of Usability. According to Dan Saffer [1], all these disciplines fall under one “big” umbrella known as User Experience (UX). The term User Experience was introduced by Donald Norman to cover some of the critical aspects of human interface research and application at Apple Computer Inc. [2]. UX is one of the most important principles used when designing, describing, or improving how users feel when interfacing with a “system”. System may be defined as computer applications, software, or web pages.

The objectives of this paper are to examine, list and categorize the various UX evaluation methods into certain defined taxonomies. This will, enable UX researchers and practitioners to easily identify, and hence choose, the evaluation methods/parameters that best suit their development needs and research requirements. This paper is

divided into four main parts. Following this introduction, subsequent parts give a brief overview of UX; discuss the various concepts and theories pertinent to UX and the basis for classification; and discuss the list of examined UX Evaluation Methods (UXEM).

II. UX – A CONCISE OVERVIEW The evolution of technology has prompted UX designers

and HCI researchers to design user interfaces which make it easier for users to accomplish their goals, and they have acknowledged the importance of UX in interface design. UX includes the users’ emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviours and accomplishments that occur before, during and after the use of a system [3]. In addition to UX, Usability is one of main disciplines for the application/interface design process. Usability has been defined in ISO 9241-11as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [4]. Cooper, Reimann, and Cronin have defined Interaction Design as “the practice of designing interactive digital products, environments, systems and services” [5]. Figure 1 illustrates the multitude of disciplines covered within the UX domain. According to visual.ly1 there are five essential factors of good UX design: Visual Design (the interface’s look and feel); Information Architecture; Market Research (understanding customers’ reactions and experiences); Interaction Design; and Usability. Some classic examples of renowned corporations which have benefited from good UX design are IBM, Amazon.com and McAfee. Amazon.com for example saw gains of USD300 million a year through improved UX.

1 http://visual.ly/ux-101-what-user-experience

533

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13)

As shown in Figure 1, the major disciplines, involved in

the User Experience Design are Interaction Design, Human-Computer Interaction, Industrial Design, Architecture, Information Architecture, Content Design and Visual Design. UX Design also covers some areas of Mechanical and Electrical Engineering via the Industrial Design domain. By examining Figure 1, we can conclude that UX is an important factor which covers all integrative and synergistic aspects of designing good user interface experience.

A. The importance of UX in Interface Design In an interactive, digital and ubiquitous environment,

users are not merely looking at and using the functionalities of products or services. The “feelings” of using these functionalities are also of importance. Developers and interface designers need to find out more about the users – what they enjoy doing, their interests and what they need. This will ensure that the products or services rendered are not only effective and useful, but also provide an enjoyable and positive experience for the users. When we enjoy using something, then it will be easier for us to use it. Thus, products or services that are pleasurable to use and exceed users’ expectations will result in enhanced enjoyment so that they will continue using these products or services.

B. Spot the differences – UX and Usability According to Nigel Bevan, there is no fundamental

difference between usability and UX, but there are some differences in terms of task performance and pleasure during the development phase. Usability is used for the following reasons [6]:

• Testing effectiveness, satisfaction and efficiency • Making the interface/product easy to use • Improving learnability

UX is used to achieve the following aims [7]:

• Understand the users, what they do and what they want

• Recognition, suggestion and related emotional responses.

III. CLASSIFYING THE UX EVALUATION METHODS UX is used in many areas by UX researchers and

practitioners. UX evaluation methods (UXEMs) play an important role in ensuring that the product’s or service’s development phase is going in the right direction. As a result, the end product will meet the expectations of users and gives appropriate positive experience while using it. Many traditional usability evaluation methods do exist, but UX is different from usability. Consequently UXEM and usability evaluation methods are not the same. Usability evaluation methods can normally be measured using stopwatches or logging, but UXEM is much more subjective in nature [8].

Numerous UXEMs are currently been used and explored by UX researchers and practitioners. Thus, it is important to identify and classify them based on certain, well defined taxonomy.

A. UXEM Taxonomies Black defines taxonomy as “the practice and study of

classification of things or concepts” [18]. According to dictionary.com, taxonomy is “A classification into ordered categories”. UXEM taxonomy simply means the classification of UXEMs into specific domains. The core benefits of having UXEM taxonomy to:

• Ease the process of identification and validation of UXEMs.

• Simplify the process of selecting an appropriate/suitable UXEM.

• Identify advantages or detect any “deficiencies” in the current list. UXEMs are specific to the needs of certain projects.

• Assist UX researchers and practitioners in formulating a new UXEM to meet the requirements of their project.

The UXEMs presented here (Table 3, in Appendix 1) is by no means exhaustive list of those currently being used by UX researchers and practitioners. A total of 89 UXEMs have been identified in the literature [9] [14] [15]. Examples include “UX expert evaluation”, “Immersion”, “Heartbeat monitoring”, “Attrak-work questionnaire”. In order to provide a better understanding of these evaluation methods, a taxonomy was developed. The parameters used by the authors to categorize UXEMs are as follow:

• Classifying the evaluation methods based on research/study type.

• Classifying the evaluation methods based on development phase.

• Classifying the evaluation methods based on the type of research conducted.

Figure 1: Various disciplines within the domain of UX [1]

534

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13)

• Classifying the evaluation methods based on the type of users/evaluators.

• Classifying the evaluation methods based on time restriction factor.

• Classifying the evaluation methods based on the “period” of experience.

B. The UXEM taxonomy criteria This section will elaborate each of the above

classification criteria.

1) Classification based on research/study type The research/study type criterion is based on several

parameters such as: how the research is conducted, where the research takes place, and in what situation the research is conducted. This first grouping type will help UX researchers and practitioners to easily identify the correct UXEM to meet their research’s requirements. For example, if the product needs background testing before it reaches the end user, then it is more suitable to do a (controlled) laboratory study than a field study type of UXEM. The following methods are used as parameters for classification criteria of the first category of UXEM taxonomy:

• Laboratory Studies – exhaustive investigations or experimentations with a specific aim to discover new facts or correct interpretations [9]. This type of investigation can be divided into two categories, either group-based or an individual-based testing. A group-based evaluation normally requires working with a focus group. The number of focus groups and the participants within each group may vary. These numbers depend on various factors, such as the subject matter of the research, incentives given to participants, or location of the study.

• Field Study – the result is gained from observing certain situations (or scenarios) happening in real environments [10]. This study/research is conducted on the real users (in their normal, “real” environment).

• Survey – is conducted with the aim of questioning someone in order to collect data for analysis purposes [11]. It can be conducted either online or through dissemination of “manual” questionnaires.

• Expert Evaluation – a group(s) or an individual expert or specialist of a certain domain will evaluate the products or services based on their predefined domain matrix and parameters. This is to ensure the products or services meet their domain’s requirements. For example, an expert from the psychology domain is needed to test and evaluate users’ emotions while using the product.

2) Classification based on development phase The product development process may comprise (the

generic) five distinct stages as shown in Figure 2, and may vary from one project to another. These five distinct stages of the development phase are [12]:

• Concept • Analysis • Design • Prototype • Implementation and deployment These stages may also have sub-stages. They stages

provide developers with a clear idea of what output is expected at each stage (sometimes even before the start of the project).

The five distinct stages of the development process are:

• Concept/Requirement/What is wanted – This is the first stage, where the requirements to develop the product are determined.

• Analysis – During this stage, the results from surveys and interviews done in the previous stage are analysed. The results will help to set and to identify the course of action to be taken in the next step of the development process.

• Design/Development – In this stage the design and development processes begin. This stage is integrated or coupled with the prototyping stage forming some sort of continuous “improvement” cycle.

• Prototype – Is a working model of the incomplete product used for testing. In this stage, real users/experts will test the prototype. These prototypes are mainly developed for testing purposes and eliminating bugs.

• Implementation and deployment – In this final stage, the completed product is ready for use and deployed to the end-users.

UX researchers and practitioners should start from the very early stage of the development phases in order to ensure the product meets the user requirements and can be easily

Figure 2: an example of a waterfall model of software development process [12]

535

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13)

understood by users, thus creating desirable user experiences.

3) Classification based on the type of research

conducted Generally, two types of data are gathered: quantitative

and qualitative data. Quantitative data seeks explanatory laws, whereas qualitative aims at in-depth descriptions [13].

• Quantitative Research – A quantitative study is predetermined, and usually involves a large number of respondents. The measurements will be objective, quantitative and statistically valid (e.g. percentage from a survey).

• Qualitative Research – Involves collecting, analysing, and interpreting data by observing what people do and say. The measurement is somewhat more subjective (e.g. in an interview, identify and understand the person’s perception and feeling. Feeling and perception cannot be measured using numbers, or cannot be quantified).

4) Classification based on the type of users/evaluators There are two types of user/evaluator. These two

categories are: classification based on “normal” users, i.e. user oriented. Secondly, classification based on subject-matter experts, i.e. expert oriented. Real users’ perceptions and experts’ perceptions may differ. Experts will view things from a more technical perspective, and have deeper technical “feelings” while using the product.

5) Classification based on time restriction factor

Time restriction influences and differentiates UXEMs according to the duration needed to conduct the evaluation process. Some may be conducted within a single day,while others may take several months to complete.

6) Classification based on the “period” of experience

This classification scheme categorizes experience according to momentary, episodic, or overall UX. Similar to these three categories, [14] categorizes periods of experience into four types: (i) Before using the system/applications, (ii) Snapshots during interaction, (iii) Based on an experience (of a task or an activity), and (iv) Based on long-term UX.

C. Summary of UXEM categories A total of 89 evaluation methods were studied, and they

are categorized according to the criteria mentioned in section – II A.UXEM Taxonomies.

Out of the 89 UXEMs examined, only 20 exhibited the involvement of experts in the evaluation process. Hence, it can be concluded that a majority of UXEMs are user-centred. This may be due to the fact that UX is “for” users and their pertinent (satisfactory) experience.

Another significant characteristic of most UXEMs surveyed is the requirement of prototypes during the evaluation process. Nevertheless, some UXEMs are carried out during the very early stage of the development cycle.

Tables 1 and 2 show the total number of UXEMs corresponding to each criterion of the taxonomy. The number of UXEMs shows the total number in that specific class. A few evaluation methods fall within more than one type. For example, some types can be used in both field and laboratory study.

Type Taxonomy Number of UXEM

Study Type Field Study 54 Lab Study 71

Survey 39Expert Evaluation 3

Type of research

Qualitative Research 52

Quantitative Research 69Type of

user/evaluatorUser-oriented 79

Expert-Oriented 20

Development and

Experience Taxonomy

Number of UXEM

Development Phase

Concept 21

Analysis 9

Design 19

Prototype 78

Implement & deploy 65

Time Restriction

1 day of evaluation 17

1 Week of evaluation 72

Months of evaluation 14

Period of Experience

Moment of whole application

16

Episode of whole

application 74

Overall of whole

application 41

Table 1: Summary of Part I (based on type)

Table 2: Summary of Part II (based on development and experience factors)

536

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13)

IV. DISCUSSION & CONCLUSION

A. Descriptive Analysis Overall 89 evaluation methods were identified from the

literature and six UXEM taxonomies were derived. Section – II C. Summary of UXEM categories describes the characteristics of each of the taxonomies. For every category several suitable evaluation methods have been classified. All evaluation methods are mostly user-oriented since they focus on the user’s feeling and emotions. For example, one of the evaluation methods is the “Attrak-Work Questionnaire”. The questionnaire is specifically designed for evaluating the UX, and can be filled in immediately after the participant/user has used the system. It can be classified under the Study-type categories, and sub-Study-type, “Survey”. As for the Period of Experience category, the Attrak-Work Questionnaire can be classified under the sub-category of “Episodic” or “Overall Experience”.

The taxonomy has simplified the process of identifying a suitable UXEM for a particular scenario depending on specific requirements. For example, the suitable UXEM during the design phase (to test a prototype), which is required for a certain “episodic event” and must be done in a controlled environment-laboratory study, is the “Perspective-Based Inspection” evaluation method. Appendix 1 provides a full list of the UXEMs examined.

B. Discussion and Conclusion UX has evolved and addressed issues of design and

evaluation for interaction between users, interface and computers beyond the plateau of usability. The development and interest in UX rises significantly as the needs of users become more complex and diversify. UX is advancing in HCI which include multiple interconnected devices, new device designs and social needs such as mobile, ubiquitous, social and tangible computing technologies. UX has also gone beyond traditional Usability engineering. It includes users’ feelings, motivations and values which are given more emphasis in addition to the efficiency, effectiveness and basic satisfaction which defined the “traditional” meaning of Usability [17].

UX covers the software development process, website and product design; it is therefore important to combine the interest of different stakeholders. The field of UX is expected to cover the holistic perspective on how a person or user feels about using a system or product. The focus is on pleasure and values rather than on performance [18].

The study finds that the type and development phase categories are most popular (i.e. they have the greatest frequency), and thus make the biggest impact. This is because, based on these categories, UX researchers and practitioners can easily identify the most suitable UXEM for their research.

ACKNOWLEDGMENTS This work is supported by the UNITEN Seed Fund –

Grant No: J510050508

REFERENCES

[1] Dan Saffer, “Design for Interaction: Creating Innovative Application and Design”

[2] Donald Norman, Jim Miller, Austin Henderson, “What You See, Some of What's in the Future, And How We Go About Doing It”

[3] Panagoitis D. Ritsos, Dennis P. Ritsos, Antonis S. Gougoulis “Standards for Augmented Reality: a User Experience perspective”

[4] SO/IEC, 9241-11 Ergonomic requirements for office work with visual display terminals (VDT) - Part 11 Guidance on usability. 1998: ISO/IEC 9241-11: 1998 (E).

[5] Cooper, Alan; Reimann, Robert; Cronin, Dave (2007). About Face 3: The Essentials of Interaction Design

[6] Kaisa Vaananen-Vainio-Mattila, Virpi Roto, Marc Hassenzahl, “Towards Practical User Experience Evaluation Methods”

[7] Nigel Bevan, “What is the difference between the purpose of Usability and User experience evaluation methods?”

[8] Virpi Roto, Marianna Obrist, Kaisa Vaananen-Vaino-Mattila, “User Experience Evaluation Methods in Academic and Industrial Contexts”

[9] Webster, 3d ed. Sensagent Dictionary. Retrieved July 31, 2013, from http://dictionary.sensagent.com/laboratory%20research/en-en/

[10] Webster, 3d ed. Sensagent Dictionary. Retrieved July 31, 2013, from http://dictionary.sensagent.com/field%20study/en-en/

[11] T. Mathiyazhagan & Deoki Nandan, “Survey research method” [12] Alan Dix, Janet Finalay, Gregory D.Abowd, Russell Beale, “Human-

Computer Interaction” Third Edition [13] John D. Anderson, “Qualitative Quantitative research” [14] Diana Margarita Mundo Spataro, Dr Sergi Jorda, Sebastian Mealla,

“A Physiological Approach for User Experience Evaluation” [15] Virpi Roto, Ming Lee, Kari Pihkala, Brenda Castro. All about UX

Evaluation Methods. Retrieved 8th August , 2013 from http://www.allaboutux.org

[16] COST Action IC0904-Twintide “Towards the integration of IT Design and Evaluation”

[17] User Experience In Wikipedia. Retrieved August 13th 2013, from http://en.wikipedia.org/wiki/User_experience.

[18] Beth Blacl, “Critical Thinking – a definition and taxonomy for Cambridge Assessment: supporting validity arguments about Critical Thinking assessments administered by Cambridge Assessment”

537

3rd International Conference on Research and Innovation in Information Systems – 2013 (ICRIIS’13)

Appendix 1 Table 3: List of UXEMs examined and, a snippet of the complete taxonomy table

538