HUMAN CENTERED DESIGN PROCESS ASSESSMENT

HUMAN CENTERED DESIGN PROCESS ASSESSMENT

Rüdiger Heimgärtner Intercultural User Interface Consulting (IUIC)

Lehderstraße 71, 13086 Berlin, Germany

ABSTRACT

Process assessment is a disciplined project-oriented evaluation of an organizational unit’s processes with respect to a process assessment model. In this paper, the approach according to ISO 33000 is presented of how to assess HCD processes. In detail, the standard approaches according to ISO 9241-210 and to ISO 9241-220 are considered. It is explained, how an assessment of HCD processes should be prepared, executed, and followed up.

KEYWORDS

Capability, Maturity, Process Assessment Model, Process Reference Model, ISO 9241-220, Human Centered Design Processes

1. INTRODUCTION

Experience from various fields of technology shows that it is possible for a development process recognized as mature to produce products of higher quality. Process assessments and improvement are an everyday business. Each project team member must know the current process and its optimization potential. Self-assessments can be used for continuous process improvement. Each project team member can subsequently recognize what is important for successful projects and what is also asked in assessments. Therefore, time is saved sustainably, and productivity is increased in projects. However, assessing processes is a complex endeavor and comprises those integrative parts necessary to obtain valid objective and comparable results. In this paper, the structure and content of a process assessment model (PAM) is explained and exemplified to yield an impression of how to determine the capability of the human centered design (HCD) processes.

The term "process" can be understood at three levels of abstraction. These levels of abstraction are not meant to define a strict black-or-white split, nor is it the aim to provide a scientific classification schema – the message here is to understand that, in practice, when it comes to the term "process" there are different abstraction levels, and that a PAM resides at the highest.

Capturing experience acquired during product development (i.e., at the DOING level) to share this experience with others means creating a HOW level. However, a HOW is always specific to a context such as a company, an organizational unit, or a product line. For example, the HOW of a project, organizational unit, or company A is potentially not applicable to a project, organizational unit, or company B. However, both might be expected to adhere to the principles represented by PAM indicators for process outcomes and process attribute achievements. These indicators are at the WHAT level while deciding on solutions for concrete templates, proceedings, and tooling, etc. is left to the HOW level.

2. PROCESS CAPABILITY DETERMINATION

The concept of process capability determination by using a process assessment model (PAM) is based on a two-dimensional framework. The process dimension (first dimension) is provided by processes defined in a process reference model (PRM). The capability dimension (second dimension) consists of capability levels that are further subdivided into process attributes. The process attributes provide the measurable characteristics of process capability. The PAM selects processes from a PRM and supplements it with indicators. These indicators support the collection of objective evidence which enable an assessor to assign ratings for processes according to the capability dimension.

ISBN: 978-989-8704-31-3 © 2021

116

2.1 Process Reference Model

Processes are grouped by process category and at a second level into process groups according to the type of

activity they address. In general, there are at least three process categories: primary life cycle processes,

organizational life cycle processes and supporting life cycle processes. Each process is described in terms of a

purpose statement. The purpose statement contains the unique functional objectives of the process when

performed in an environment. For each purpose statement a list of specific outcomes is mapped, as a list of

expected positive results of the process performance. For the process dimension, the HCD PRM provides the

set of processes in ISO 9241-220:2019-03, whereas HCP.3 is divided into further process groups according to

the structured main HCD activities described in ISO 9241-210 and considered in ISO 9241-220:2019-03 as

shown in Figure 1.

Figure 1. Relationship between the HCD processes (including the sub-processes in HCP.3) from ISO 9241-220:2019-03

2.1.1 Organizational and Supporting Life Cycle Processes Category

The organizational life cycle processes category consists of processes that develop process, product, and

resource assets which, when used by projects in the organization, will help the organization achieve its business

goals. The organizational life cycle processes category consists of the following groups:

• the HCD.1 ensure process group;

• the HCD.2 enable process group.

The management processes may be used by anyone who manages any type of HCD project or process

within the life cycle belonging to the supporting life cycle process category.

2.1.2 Primary Life Cycle Processes Category

The primary life cycle processes category consists of processes that may be used by the customer when

acquiring products from a supplier, and by the supplier when responding and delivering products to the

customer including the engineering processes needed for specification, design, development, integration, and

testing. The primary life cycle processes category consists of the following groups:

• the HCP.3 Execute process group;

• the HCP.4 Introduce & operate process group.

The HCP.3 Execute process group consists of processes addressing the elicitation and management of

customer and internal requirements, the definition of the system architecture and the integration and testing on

the system level.

International Conferences Interfaces and Human Computer Interaction 2021; and Game and Entertainment Technologies 2021

117

3. MEASUREMENT FRAMEWORK

The measurement framework provides the necessary requirements and rules for the capability dimension. It

defines a schema which enables an assessor to determine the capability level of a given process. These

capability levels are defined as part of the measurement framework. To enable the rating, the measurement

framework provides process attributes defining a measurable property of process capability. Each process

attribute is assigned to a specific capability level. The extent of achievement of a certain process attribute is

represented by means of a rating based on a defined rating scale. The rules from which an assessor can derive

a final capability level for a given process are represented by a process capability level model. The HCD PAM

uses the measurement framework defined in ISO/IEC 33020:2015.

3.1 Process Capability Levels

Process capability levels and process attributes are identical to those defined in ISO/IEC 33020:2015. Rough

descriptions of the capability levels and the corresponding process attributes can be found in section 5 in the

paper at hand. Process attributes are features of a process that can be evaluated on a scale of achievement,

providing a measure of the capability of the process. They are applicable to all processes. A capability level is

a set of process attribute(s) that work together to provide a major enhancement in the capability to perform a

process. Each attribute addresses a specific aspect of the capability level. The levels constitute a rational way

of progressing through the improvement of the capability of any process. According to ISO/IEC 33020:2015

there are six capability levels (Figure 2), incorporating nine process attributes (PA) according to the levels.

Figure 2. Process capability levels according to ISO/IEC 33020:2015

3.2 Process Attribute Rating

To support the rating of process attributes, the ISO/IEC 33020:2015 measurement framework provides a

defined rating scale with an option for refinement, different rating methods and different aggregation methods

depending on the class of the assessment (e.g., required for organizational maturity assessments). Within this

process measurement framework, a process attribute is a measurable property of process capability. A process

attribute rating is a judgement of the degree of achievement of the process attribute for the assessed process.

The rating scale is defined by ISO/IEC 33020:2015 as shown in Figure 3.

ISBN: 978-989-8704-31-3 © 2021

118

Figure 3. Rating scale according to ISO/IEC 33020:2015

The rating and aggregation method in ISO/IEC 33020:2015 provides the following definitions. A process

outcome is the observable result of successful achievement of the process purpose. A process attribute outcome

is the observable result of achievement of a specified process attribute. Process outcomes and process attribute

outcomes may be characterized as an intermediate step to providing a process attribute rating. When performing

rating, the rating method employed shall be specified relevant to the class of assessment. The use of rating

method may vary according to the class, scope, and context of an assessment. The lead assessor shall decide

which (if any) rating method to use. The selected rating method(s) shall be specified in the assessment input

and referenced in the assessment report. Further information can be taken from ISO/IEC 33020:2015.

4. PROCESS ASSESSMENT MODEL

A process assessment model (PAM) offers indicators to identify whether the process outcomes and the process

attribute outcomes (achievements) are present or absent in the instantiated processes of projects and

organizational units. These indicators provide guidance for assessors in accumulating the necessary objective

evidence to support judgments of capability. They are not intended to be regarded as a mandatory set of

checklists to be followed. To judge the presence or absence of process outcomes and process achievements an

assessment delivers objective evidence. All such evidence comes from the examination of work products and

the repository content of the assessed processes, and from testimony provided by the performers and managers

of the assessed processes. This evidence is mapped to the PAM indicators to allow the establishment of a

correspondence to the relevant process outcomes and process attribute achievements. There are two types of

indicators. Process performance indicators, which apply exclusively to capability level 1, provide an indication

of the extent of fulfillment of the process outcomes. Process capability indicators, which apply to capability

levels 2 to 5, provide an indication of the extent of fulfillment of the process attribute achievements.

Assessment indicators are used to confirm that certain practices were performed, as shown by evidence

collected during an assessment. All such evidence comes either from the examination of work products of the

processes assessed, or from statements made by the performers and managers of the processes. The existence

of base practices and work products pro-vide evidence of the performance of the processes associated with

them. Similarly, the existence of process capability indicators provides evidence of process capability. The

evidence obtained should be recorded in a form that clearly relates to an associated indicator, in order that

support for the assessor’s judgment can be confirmed or verified as required by ISO/IEC 33002:2015.

4.1 Process Performance Indicators

Types of process performance indicators are base practices (BP) and work products (WP). Both BPs and WPs

relate to one or more process outcomes. Consequently, BPs and WPs are always process-specific and not

generic. BPs represent activity-oriented indicators. WPs represent result-oriented indicators. Both BP and WP

are used for judging objective evidence that an assessor is to collect, and accumulate, in the performance of an

assessment. In that respect BPs and WPs are alternative indicator sets the assessor can use. A PAM offers a set

of work product characteristics (WPC) for each WP. These are meant to offer a good practice and


119

state-of-the-art knowledge guide for the assessor. Therefore, WP and WPC are supposed to be a quickly

accessible information source during an assessment. In that respect WPs and WPCs only represent an example

structure. They are neither a "strict must" nor are they normative for organizations. Instead, the actual structure,

form and content of concrete work products and documents for the implemented processes must be de-fined

by the project and organization, respectively. The project and/or organization ensures that the work products

are appropriate for the intended purpose and needs, and in relation to the development goals.

4.2 Process Capability Indicators

Types of process capability indicators are generic practice (GP) and generic re-source (GR). Both GPs and

GRs relate to one or more PA achievements. In contrast to process performance indicators, however, they are

of generic type, i.e., they apply to any process. The difference between GP and GR is that the former represents

activity-oriented indicators while the latter represent infrastructure-oriented indicators for judging objective

evidence. An assessor must collect and accumulate evidence supporting process capability indicators during

an assessment.

5. PROCESS CAPABILITY LEVELS AND PROCESS ATTRIBUTES

Process capability indicators are the means of achieving the capabilities addressed by the considered process

attributes. Evidence of process capability indicators supports the judgment of the degree of achievement of the

process attributes. The capability dimension of the PAM consists of six capability levels matching the

capability levels defined in ISO/IEC 33020:2015. The process capability indicators for the 9 process attributes

included in the capability dimension for process capability level 1 to 5 are described in section 5.3. The generic

practices address the characteristics from each process attribute. The generic resources relate to the process

attribute. Process capability level 0 does not include any type of indicators, as it reflects a non-implemented

process or a process which fails to partially achieve any of its outcomes.

5.1 Process Capability Level 0: Incomplete Process

The process is not implemented or fails to achieve its process purpose. At this level there is little or no evidence

of any systematic achievement of the process purpose.

5.2 Process Capability Level 1: Performed Process

The implemented process achieves its process purpose. The following process at-tribute demonstrates the

achievement of a performed process representing process capability level 1. The process performance process

attribute for level 1 is a measure of the extent to which the process purpose is achieved. If the process achieves

its defined outcomes requested by the generic practice GP 1.1.1 “Achieve the process outcomes”, then level 1

is reached. This means, i.e., that the intent of the process specific base practices is met, and the work products

are produced (evidencing the process outcomes) using the generic resource “Resources are used to achieve the

intent of process specific base practices”.

5.2.1 Process Reference Model and Performance Indicators (Level 1)

The processes in the process dimension can be drawn from the HCD process reference model (based on ISO

9241-220:2019-03). Each table related to one process in the process dimension contains the process reference

model and the process performance indicators necessary to define the processes in the process assessment

model (cf. Figure 4). The process references model describes the processes by ID, name, purpose, benefit, and

outcomes. The process performance indicators consist of base practices and output work products.

ISBN: 978-989-8704-31-3 © 2021

120

Figure 4. Template for the process description

5.2.2 Output work Products (Level 1)

Work product characteristics (WPC) can be used when reviewing potential outputs such as work products of

process implementation. The characteristics are provided as guidance for the attributes to look for, in a sample

work product (WP) and to provide objective evidence supporting the assessment of a process. A documented

process and assessor judgment are needed to ensure that the process context (application domain, business

purpose, development methodology, size of the organization, etc.) is considered when using this information.

WPs and their characteristics should be considered as a starting point for considering whether, given the

context, they con-tribute to the intended purpose of the process, and not as a checklist of what every

organization must have.

5.3 Process Reference Model and Performance Indicators (Level 2-5)

In contrast to level 1, the performance indicators on level 2-5, represented by generic practices and resources,

relate to all process in the process reference model.

If the performed process is implemented in a managed fashion (planned, monitored, and adjusted) and its

work products are appropriately established, controlled, and maintained, then it is called a “managed process".

Thereby the performance management process attribute (PA 2.1) and the work product management process

attribute (PA 2.2), together with the previously defined performance process attribute (PA 1.1), demonstrate

the achievement of a managed process representing process capability level 2.

If the managed process is implemented using a defined process that can achieve its process outcomes, then

it is called a “established process”. The process definition process attribute (PA 3.1) and the process

deployment process attribute (PA 3.2), together with the process attributes for the performed (PA 1.1) and the

managed process (PA 2.1 and PA 2.2), demonstrate the achievement of an established process representing

process capability level 3.

If the established process operates predictively within defined limits to achieve its process outcomes, then

it is called a “predictive process”. Quantitative management needs are identified, measurement data are

collected and analyzed to identify assignable causes of variation. Corrective action is taken to address

assignable causes of variation. The quantitative analysis process attribute (PA 4.1) as well as the quantitative

control process attribute (PA 4.2), together with the process attributes for the performed (PA 1.1), managed

(PA 2.1 and PA 2.2) and established (PA 3.1 and PA 3.2) process, demonstrate the achievement of a predictable

process representing process capability level 4.

If the predictable process is continually improved to respond to change aligned with organizational goals,

then it is called a “innovating process”. The process innovation process attribute (PA 5.1) as well as the process

innovation implementation process attribute (PA 5.2), together with the process attributes for the performed

(PA 1.1), managed (PA 2.1 and PA 2.2), established (PA 3.1 and PA 3.2) and predicted (PA 4.1 and PA 4.2)

process, demonstrate the achievement of an ‘innovating process’ representing process capability level 5.


121

6. RATING IN THE PROCESS ASSESSMENT

The assessment procedure contains at least the following steps:

• Explain process purpose and benefit and most important requirements of the process to the audience,

• Interview the audience regarding process life based on purpose, base practices and work products and

collect documented evidence,

• Rate the evidence according to the rating scale,

• Accumulate the ratings for one process (e.g., HCP.3.4.2) to an overall rating considering:

• Achievement of process purpose,

• Achievement of all base practices,

• Achievement of work products.

As an example, the summary of the results of the rating up to level 2 could appear as presented in Figure 5.

Depending on the rating, the HCD processes of the executing process group (HCP.3) are ratings up to level 2

in this figure addressing the performance process attribute (level 1) and the management process attributes

(level 2). The only process that reached level 2 in this example is HCP.3.4.2 (Produce and refine user interface

design solutions) indicating that all base practices are achieved and the work products of this process (such as

system interface requirement, architectural design description and user interface specification) are available

and reviewed, and the process is managed in a proper way. Other processes such as HCP.3.1.1 (Establish

human-centered quality objectives) were not performed and managed properly, and the work products are

mostly missing.

Figure 5. Example Rating of the HCP.3 Process Group from ISO 9241-220:2019-03

ISBN: 978-989-8704-31-3 © 2021

122

7. DISCUSSION

With a seamless HCD development process, mistakes and unproductivity can be prevented and product quality

rises with high quality process output contributing to a high-quality interactive product. Behind such

experiences a causal relationship be-tween process and product quality is assumed, which consists in the fact

that a more mature process with a lower probability of error produces higher quality products than a less mature

process. Since quality cannot be achieved without a sustainable focus on user requirements, usability

engineering is also referred to as a human-centric development process. Requirements for a human-centered

development process are defined in DIN EN ISO 9241-210:2011-01. The aim of usability engineering is to do

everything necessary in a development process to steer it towards an optimal quality of use of the final product.

Following a PRM like that derived from ISO 9241-220:2019-03 ensures the involvement of users as

requirement donors and as evaluators of

a) the achieved quality of use,

b) the application of suitable usability engineering methods,

c) the documentation of design decisions with relevance for the quality of use,

d) the establishment of a quality management system for the quality of use and

e) setting up process roles with suitably qualified personnel (cf. DIN SPEC 92412:2015-11).

8. CONCLUSION

The presented PAM for the development process of interactive products serves to assess the process quality in

development projects based on objective indicators for process capability and thereby to achieve quality of use

referring to the HCD processes defined in ISO 9241-220:2019-03. Even if the presented HCD-PAM is still in

a working draft state and, hence, must be improved and worked out in detail in accordance with all other

relevant standards, the principles of this HCD-PAM and the presented assessment procedure have been

confirmed for years by assessments in the automotive industry regarding software engineering processes

(cf. VDA QMC WG 13, 2017). The presented HCD-PAM can be applied in conjunction with the process

reference model (PRM) for human centered design (HCD) processes derived from the processes in

ISO 9241-220:2019-03 to a similar benefit. The refinement and application of the suggested HCD-PAM will

show if it can be used to prove that the process has the capability of justifying trust in the HCD processes by

all stakeholders and causes that the assessments can be productively conducted.

REFERENCES

DIN EN ISO 9241-210:2011-01, Ergonomics of human-system interaction - Part 210: Hu-man-centred design for

interactive systems.

DIN SPEC 92412:2015-11, Ergonomics of human-system interaction - Auditing procedure for the development of

interactive products based on DIN EN ISO 9241-210.

ISO 9241-220:2019-03, Ergonomics of human-system interaction - Part 220: Processes for enabling, executing and

assessing human-centred design within organizations.

ISO/IEC 15504-5:2006, Information Technology – Process assessment – Part 5: An exem-plar Process Assessment Model.

ISO/IEC 33002:2015, Information technology -- Process assessment – Requirements for performing process assessment.

ISO/IEC 33020:2015, Information technology -- Process assessment – Process measure-ment framework for assessment

of process capability.

VDA QMC Working Group 13 / Automotive SIG, 2017, Automotive SPICE Process Assessment / Reference Model,

URL=http://www.automotivespice.com/fileadmin/software-download/AutomotiveSPICE_PAM_31.pdf, last access

06-30-2021.


123

Documents

HUMAN CENTERED DESIGN PROCESS ASSESSMENT