Embed Size (px)
Citation preview
University of Gothenburg
Chalmers University of Technology
Department of Computer Science and Engineering
Göteborg, Sweden, February 2013
Framework for Measuring Perceived Quality in
Technical Documentation
Bachelor of Science Thesis in Software Engineering and Management
BISHARE SUFI ABDI
The Author grants to Chalmers University of Technology and University of Gothenburg
the non-exclusive right to publish the Work electronically and in a non-commercial
purpose make it accessible on the Internet.
The Author warrants that he/she is the author to the Work, and warrants that the Work
does not contain text, pictures or other material that violates copyright law.
The Author shall, when transferring the rights of the Work to a third party (for example a
publisher or a company), acknowledge the third party about this agreement. If the Author
has signed a copyright agreement with a third party regarding the Work, the Author
warrants hereby that he/she has obtained any necessary permission from this third party to
let Chalmers University of Technology and University of Gothenburg store the Work
electronically and make it accessible on the Internet.
Framework for Measuring Perceived Quality in
Technical Documentation
BISHARE SUFI ABDI
© BISHARE SUFI ABDI, February 2013.
Examiner: LARS PARETO
University of Gothenburg
Chalmers University of Technology
Department of Computer Science and Engineering
SE-412 96 Göteborg
Sweden
Telephone + 46 (0)31-772 1000
Cover:
Department of Computer Science and Engineering
Göteborg, Sweden February 2013
, 2013
3 | P a g e
Framework for Measuring Perceived Quality in Technical Documentation
Bishare Sufi
University of Gothenburg
Department of Computer Science and
Engineering
Gothenburg, Sweden
E-mail: [email protected]
Abstract
Customer product information (CPI) provides
essential information to a user about how to use a
product. Given the importance of such technical
documentation to enable a more effective use of
technology, there is very little research conducted in
the domain of improving document quality. This
study intends to fill this gap by developing a tentative
framework for how to measure user perceived quality
(PQ) of technical documentation. Data collection is
based on a literature review and interviews with
practitioners. The advantages and disadvantages of
the approach are evaluated and suggestions for future
studies outlined. The implication of this research is
that it allows companies that produce technical
documentation to measure and thus improve
document quality more effectively.
Keywords: benchmarking, checklists, data quality, key
performance indicators, perceived quality, performance
measurement, surveys and user satisfaction.
1 INTRODUCTION
In today’s globalized business market organizations are
increasingly striving to adopt a more customer-focused
approach to remain competitive. This is deemed as
important to improve the quality of products and services.
Companies within the information system (IS) sector,
especially those who specialize in technical
documentation are increasingly recognizing the
importance of measuring quality more effectively to stay
competitive. The quality of technical documents and user
manuals form an important part of perceived product
quality. Generally, users turn to product documentation in
order to learn more about the product (Wingkvist, et al.,
2010).
Thus, technical documents quality plays a pivotal role
both for the usability of the document itself and the
product. In simple language, in order to achieve
satisfaction of customer requirements, a product must do
what the customer expects it to do. From this perspective,
the ability to initially measure and eventually control
customer perceived quality, is a major success factor in
software business (Xenos & Christodoulakis, 1997).
In the software engineering field the customer is assumed
to have a central role in improving internal activities and
thus quality of products and services (Fogelström, et al.,
2009). To achieve this there needs to be a close fit
between stated requirements and the product itself. From
this perspective, the interaction between the user and the
enterprise is important to improve the overall quality of
products and services.
The idea of increased customer collaboration can be
applied for improving the development of technical
documentation as data coming from user’s feedback is
essential to improve document quality. However, this
requires a systematic approach for measuring the
perceived quality in such products.
The Swedish company Sigma Kudos develops one type of
technical document labeled customer product information
(CPI). In brief, the CPI contains all relevant information
for how to use a product or system. In practice, the CPI
supports the user in terms of how to use complex systems
like serving GPRS support node-mobility management
entity (SGSN-MME). The system handles the registration
of the mobile to the GPRS network and takes care of its
mobility management. Sigma Kudos intends to develop a
more rigorous approach to measuring quality in their
technical documentation, with a special focus on CPI.
1.1 Research aim
The aim of this study is to develop a tentative framework
for how to measure the perceived quality of technical
documentation by using the pre-defined key performance
indicators (KPIs) (see Table 1). The KPIs have been
developed in a recent study Amanpreet (in press) and
Sigma Kudos specifically for technical documents.
According to this study the KPIs are created by focusing
on quality attributes of the document and they can be used
during the performance measurement process. Table 1
, 2013
4 | P a g e
displays the KPIs suitable for measurement of CPI
documents quality:
KPI Quality Attributes
Structure Understandable, well-presented, well-
documented, concise representation,
representation consistency,
interpretability
Contextual Value-added, appropriate amount of
data, relevance, completeness
Accuracy Accuracy, believability, objectivity
Accessibility Accessibility, easily traced, user
friendly, ease to retrieval
Table 1: KPIs for documents source (Amanpreet, in
press).
Current research shows that while there is much written
about quality improvement in products and services, there
is none that specifically focus on document quality.
Additionally, there is little guidance as to the practice of
measuring perceived quality in documents. The current
study is an attempt to address this shortcoming by
providing some useful insights into the basic components
of measuring perceived quality in technical documents. In
view of this, the guiding research question for the study
is: How can perceive quality be measured in technical
documentation? Thus, this is the impetus for this
investigation and resulting tentative framework.
To accomplish the research objective a qualitative
research design is used in a two-pronged approach: 1) a
literature search and review to map out the current
knowledge of performance measurement and 2)
interviews with Sigma Kudos staff to capture practitioners
view on document quality. The outcome from the data
collection is summarized and presented in the tentative
performance measurement framework on page 10.
1.2 Delimitation
The study focuses mainly on developing a tentative
measurement approach for perceived quality of technical
documents. Due to time and access constraints it is also
outside the scope of the study to test the resulting
framework into a real-world setting. Instead the study
discusses the advantages and disadvantages of the
framework and its usefulness in various contexts.
1.3 Overview
The rest of this paper is outlined as follows: section 2
provides an overview of related research as well as
summarizing the literature into a tentative framework for
the case. Section 3 describes the research approach and
process. Then, findings are presented in section 4. Finally,
in section 5 the findings are discussed and suggestions for
future work are outlined in section 6. The paper ends with
conclusion section.
2 RELATED RESEARCH
This section reviews the literature on perceived quality,
surveys, data quality and performance measurement with
the view to develop a tentative framework for measuring
quality in documents.
Wingkvist, et al. (2010) state that in order to determine
the quality of a product metrics need to be defined,
weighted and many of these metrics are based on
checklists. Checklist is a valid technique for evaluating
software product quality. Further, it is suggested to be an
“outstanding instrument” for handling the problem of
measurement procedure for qualitative determination
(Punter, 1997). For example, it allows choosing suitable
indicators and measures which determine a quality
characteristic.
According to Xenos & Christodoulakis (1995) an
approach is required for the measurement and evaluation
of users’ opinion. The purpose of this is to evaluate the
user’s opinion with respect to computers in general and to
use this information to improve software quality.
They suggest that the most efficient way to collect user’s
opinion is devising and performing structured
questionnaires and surveys (Xenos & Christodoulakis,
1995, 1997). This perspective can be transferred to
improving quality of documents and is thus also
beneficial for developing a viable measurement process
for such products.
, 2013
5 | P a g e
2.1 Perceived quality
According to Wingkvist, et al. (2010) the notion of
quality is most often viewed as an “intangible trait,
something that can be subjectively felt or judged, but
often not exactly measured or weighted”. They argue that
terms like good or bad quality are deliberately vague and
used with no intention of ever being an exact science.
This certainly creates confusion as with respect to
defining the concept of quality.
Aaker (1991) states that perceived quality (PQ) is the
customer’s perception of the overall quality of the product
with respect to its intended purpose. PQ can be associated
with price premiums, brand usage and stock return.
Further, it has the important attribute of being applicable
across product classes.
In order to elicit and understand user PQ both user
satisfaction and all functions of a business need to be
linked. Xenos & Christodoulakis (1997) further state that
the ability to initially measure and control customer PQ is
a fundamental factor in software business, therefore an
approach is needed to systematically measure this. The
approach includes continuous involvement of the
customer into company’s business activities.
Fogelström, et al. (2009) argues that cooperation between
user and organization is a prerequisite for improving
product quality. Measurement of PQ about the product
requires the identification of quality attributes (QA) of the
product.
According to Xenos & Christodoulakis (1995) the
following QA sufficiently describe all the aspects of the
user’s critique regarding a product:
Efficiency
Expandability
Functionality
Maintainability
Portability
Reliability
Simplicity
Usability
These attributes are the most commonly used within
software engineering to identify quality attributes of a
product. There are some empirical studies regarding the
measurement of PQ of software products where the
measurement process is performed by applying an
approach, which relates internal measureable quantities
with external quality attributes. These can be found in the
software measurement literature (Masayna, et al., 2007).
For instance, Xenos & Christodoulakis (1997) mention
function points which are used for estimating product
cost. Further, they suggest that cyclomatic complexity is
used for estimation of software complexity. This one is
software metric and measures the number of linearly
independent paths through a program’s source code. Also,
that effort estimator could be used to identify required
effort (Xenos & Christodoulakis, 1997).
2.2 Surveys
Surveys are capable of obtaining information from a large
population. Furthermore, surveys can also elicit
information about attitudes that are otherwise difficult to
capture using observational techniques (Glasow, 2005).
Electronic surveys are a good example of how user
opinions can be gathered efficiently. However, surveys
are not unproblematic. The problems associated with
surveys have been raised in many studies and include
(Xenos & Christodoulakis, 1997):
Subjectivity of measurements.
Difficulty of statistically analyzing results.
Lack of a weighing technique.
Frequency of errors.
According to Xenos & Christodoulakis (1997) these
issues produce unreliable data and therefore decrease the
quality of the outcomes. In addition, they also point to the
fact that undesired results are often related to the
participants. In other words, respondents are hard to
control and prone to mistakes. For instance, they answer
the questions subjectively; sometimes they are not the
‘right’ people for answering the questionnaires and fail to
answer the questions properly by giving unexpected
answers (Xenos & Christodoulakis, 1995).
This demonstrates that although there is a clear goal for
what type of data one needs to collect, the results could be
difficult to manage or predict. Therefore, well-planned
and structured surveys are required in order to avoid
problems.
, 2013
6 | P a g e
2.2.1 Handling the problems
Subjectivity of measurements Xenos & Christodoulakis
(1997) argue that subjective judgments bring in a degree
of error. In their view, subjectivity of measurement will
remain a problem, regardless of the measurement
methodology.
However, according to Lahlou, et al. (1992) the
application of a number of simple rules when designing
the questionnaire can improve the quality of the
measurement. They propose the following guideline for
structuring a questionnaire (Lahlou, et al., 1992):
Describe the aim of the questionnaire and relate
the first question with this aim.
The questionnaire should be attractive to the user
and the size must be kept short.
The questionnaire should be well structured and
the questions have to follow a logical order
without referring to each other.
Avoid open question if it is possible.
Questions should be objective to avoid affecting
user judgment.
Avoid to use confusing concepts such us
probability.
The quality manager should apply the above mentioned
guidelines in order to decrease error relate to human
judgment (Xenos & Christodoulakis, 1995, 1997).
Scale types statistical analysis refers to a collection of
methods used to process large amount of data. According
to Xenos & Christodoulakis (1997) statistical analysis has
a vital role in quality improvement activities because it
provides ways to objectively report the status of the
product based on data collection. Further, it is particularly
useful when dealing with complex data coming from
different sources.
There are four standard measurement scales to use as a
basis for developing a survey. Which one to apply
depends on the type of information contained in the
measurement results, so addressing the most suitable one
is crucial and enhances the success of measurement
analysis (Xenos & Christodoulakis, 1995, 1997).
Interval scale. Quantitative attributes are all
measurable on interval scales. It is about the
order of data points, and the size of the intervals
in between data points (Stevens, 1946).
Nominal scale. Represents the most unrestricted
assignment of numerals and used only as labels
or type numbers. For instance the use of
numerals as names for classes is an example of
the assignment of numerals according to rule.
The rule is: Do not assign the same numeral to
different classes (Stevens, 1946).
Ordinal scale. In this scale type, the numbers
assigned to objects or events represent the rank
order. An example of ordinal scale is the scale of
hardness of mineral. Other instances are found
among scales of intelligence or quality of leather
(Stevens, 1946).
Ratio scale. Most measurement in the physical
sciences and engineering is done on ratio scales.
There are two types of ratio scales: fundamental
and derived. Fundamental scales are represented
by length, weight and electrical resistance.
Derived scales are represented by density, force
and elasticity. The scale type takes its name from
the fact that measurement is the estimation of the
ratio between a magnitude of a continuous
quantity and a unit magnitude of the same kind
(Stevens, 1946).
One of the main problems with survey measurement is
that survey data based on ordinal scale cannot be
statistically analyzed using formal statistical methods. For
example if the questionnaire has multiple choices, choice
bars can be the solution, also an instruction that explains
the choices are in interval scale and equal distance to each
other should be mentioned (Xenos & Christodoulakis,
1997).
Analysis: Weighing user opinions is important because
all users are different and each specific one should be
accordingly evaluated (Xenos & Christodoulakis, 1995,
1997). For instance all users are not equal, they not have
the same background of knowledge and they do not have
, 2013
7 | P a g e
the same needs. An organization should take into
consideration all these factors for assessment of
measurement process and weighing user opinions based
on users’ qualification can be a way to increase the
quality of the survey.
Xenos & Christodoulakis (1995) visualized a process to
evaluate user’s knowledge considering three parts,
personal background, syntactic knowledge and semantic
knowledge. The personal background contributes the half
of the rest equally contributing parts here below is the
graphic showing the visualization:
Semantic 40% Syntactic 40%
Personal background 20%
Figure 1: Evaluating user’s qualification source Xenos &
Christodoulakis (1995).
Personal background should be a collection of
unrelated user qualification in the actual
questionnaire area or product.
Syntactic knowledge general knowledge
regarding the area.
Semantic knowledge how well the user is aware
the semantics of the problem caused by the
product.
For instance personal background should cover all
attributes related to the user such as age and gender;
syntactic knowledge should address how familiar the user
is with the actual product and semantic knowledge in
most of the cases new programs are built in order to
substitute the old one the semantic knowledge defines
how well the user can handle the issue related to this
activity (Xenos & Christodoulakis, 1995, 1997).
By knowing the user’s background the questionnaires can
be developed based on different kinds of users and
expected outputs could be predetermined. Capturing
knowledge level of the user is relevant both for the quality
of collected data and the improvement of the product
(Masayna, et al., 2007). If the survey is well-designed as
well as being addressed to the right people, the overall
then both the predesigned questionnaire and the outcome
data of the survey will be of higher quality.
Preventing errors surveys are really sensitive to errors
because humans do not like filling questionnaires,
especially when the questions have to do with their
abilities. Therefore they never take this task seriously
(Xenos & Christodoulakis, 1995).
According to Xenos & Christodoulakis (1997) in their
surveys they have measured a significant number of errors
that will occur when the user responses the questions
incorrectly such reasons are:
The user did not answer the questionnaire
himself/herself, but give it to someone else who
was inadequate to respond.
The user answered the questionnaire very
carelessly and marked randomly when he/she
was confused.
The user started to answer the questionnaire
with enthusiasm and lost interest somewhere in
the middle of the questionnaire.
Such errors can be prevented by following the simple
rules presented in this research paper, but cannot be
eliminated. Due, it is difficult to design questionnaires
that will detect the errors, the literature introduces the
techniques that can help to show the presents of these
errors and cannot ensure the absence. Techniques
presented in the following section, are used in order to
detect such errors (Xenos & Christodoulakis, 1995, 1997).
The techniques the main aim of this study is to present a
rigorous approach to perceived document quality
measurements that will help a quality manager to include
such measurements in the company’s quality assessment
program. According to Xenos & Christodoulakis (1995,
1997) this is done by using structured surveys to produce
measurement results with a minimum degree of
subjectivity, easy to analyze, respecting user qualification
and as error-free as possible.
The techniques proposed in order to measure the users’
perception of document quality are (Xenos &
Christodoulakis, 1995, 1997):
, 2013
8 | P a g e
1- Qualification weighed user opinion (QVCO) this
one is less expensive and reliable.
2- Qualification weighed user opinion with
safeguard (QVCO~s) this one is both more
reliable and expensive, than the first one.
3- Qualification weighed user opinion with double
safeguard (QVCO~ds) this one is the most
expensive and reliable.
These techniques are used by applying a formula and each
specific technique has own formula (Xenos &
Christodoulakis, 1997).
QVCO this one measures the score of user
opinion, qualification and the number of
interviewed.
QVCO~s in this one safeguard is used in order to
handle errors, where safeguard is defined as a
question embedded in the questionnaire and
checks if the user is responding correctly.
QVCO~ds this one will use double safeguard it
checks both errors in user opinions and
qualification.
In order to measure user qualification the safeguard
questions inside the questionnaire should include
information covering the aspects of qualification (Xenos
& Christodoulakis, 1995, 1997). Control questions or
repeated questions can be used as safeguard in order to
check if the user is the one addressed to the questionnaire,
where control question can be answered by one specific
response. Also, repeated questions offering different
response placed not close to each other can be used for
checking errors.
Paying attention to previous mentioned aspects in order to
measure the quality of a document by conducting surveys
regarding user satisfaction can provide significant
information about the status, condition and attitudes of
users after they have received the document. Further, if
the survey technique is applied accordingly, the data
obtained on user experiences and satisfaction is more
likely to be credible (Hatry, 1999).
2.3 Data quality
It is difficult to give a universal definition of what quality
means and it depends on several aspects. Therefore, in
order to obtain an accurate measure of the quality of data,
one has to choose which attributes to consider and how
much each one contributes to the quality as a whole
(Bobrowski, et al., 1998). Wang, et al. (1995) emphasize
that it is hard to manage data quality (DQ) without
understanding the attributes of the data, which defines its
quality. They identify the following attributes as the most
important:
Accuracy is defined as “The recorded value is in
conformity with the actual value.”
Completeness is defined as “All values for a
certain variables are recorded.”
Consistency is defined as “The representation of
the data value is the same in all cases.”
From this perspective it is clear that data quality is a
multidimensional and hierarchical concept, where
accuracy is the most obvious dimension when it comes to
DQ (Wang, et al., 1995). One could argue that these
attributes are also valid for measuring quality of CPI
document. Inaccurate or incomplete data can have
significant impacts on the success of business activities of
the enterprise, but there is a cost-quality tradeoff in
implementing data quality programs. For instance, when
the cost is extremely high zero-defect data is not possible
to sustain.
According to Jones (1991) the data coming from
questionnaires can be divided in two categories: soft data
measurement and hard data measurement. Soft data are
related to areas in which human opinion must be
evaluated and absolute precision cannot be achieved. For
the hard data elements high accuracy is both possible and
desirable.
Data quality has been a significant issue for the business
of the companies where organizations are aware about the
importance of the data and the cost to sustain in order to
deliver a good data quality (Masayna, et al., 2007).
Moreover, data need to be accessible, useful,
comprehensible and believable to the user the goal is to
facilitate the collection and the processing of data.
, 2013
9 | P a g e
So, DQ should satisfy a given set of quality requirements.
For instance, the improvement of data requires a
significant amount of resources and time where poor data
implies higher cost and time consuming. Furthermore, the
company has to go through data several times in order to
make improvements, spend more time by repeating the
process and make changes if it is necessary (Wang, et al.,
1995). Therefore, a measurement process is required in
order to objectively track actual performance against
planned objectives, to help assess overall business and
technical performance against market-driven
requirements.
2.3.1 Linking DQ to KPIs
It is important to understand the link between DQ and
KPIs because they are interdependent (Masayna, et al.,
2007). Further, there is a need to examine way of
improving DQ so that KPIs better address the goals
established for them. Figure 2 demonstrates the link
between DQ and KPIs where DQ is linked to
organizational KPIs which can enable better decision-
making with regards to organizational investment in DQ
efforts.
According to Masayna, et al. (2007) the model visualized
below is helpful, because it focuses on improving DQ so
KPIs can effectively support management objectives. The
model is related to research reports regarding the current
state of DQ initiatives in Australian organizations.
External influences Users
The link between DQ and organizational KPIs
KPI activities
Figure 2: Linking DQ to KPIs source Masayna, et al.
(2007).
However, in order to fully appreciate the relationship
between DQ and KPIs they should be light of user
activities. Today the knowledge regarding the link
between DQ and companies KPIs is still unclear. In light,
of this an enterprise should be able to identify the data
quality elements which are relevant to support the KPIs
(Arayici, et al., 2009). The next section defines KPIs in
more detail.
2.3.2 Measuring key performance indicators
Masayna, et al. (2007) defines key performance indicators
(KPIs) as measures that determine how well business
processes are performing in terms of their potential to
enable a specific target to be achieved. Ideally, KPIs
should be created in relation to a measurable business
objective. They can focus on critical parts of
organizational activities that need to be improved.
From this perspective, KPIs need to be well defined and
linked to particular outcomes. In addition, KPIs reflect the
idea that some aspects of organizational performance are
more important than others (Vial & Prior, 2003).
The construction industry has so far been the most avid
user of KPIs to improve business performance in terms of
delivering better end products (Arayici, et al., 2009).
While there are many categories of KPIs, this research
will only focus on qualitative ones including: structured
perceptions or structured feedback where the
measurement focuses on user satisfaction of the product.
Before starting the creation of the KPI following
questions need to be addressed (Masayna, et al., 2007):
Does the KPI motivate the right behavior?
Is the KPI measurable?
Is the measurement cost effective?
Is the target value attainable?
Are the factors affecting the KPI controllable?
Is the KPI meaningful?
Performance measurement (PM) enables businesses to
meet demands more effectively where a KPI is created for
the business objective, then it is quantified and measured.
The main challenges and limitations of applying KPI
procedures into enterprise business activities generally
include support from the organization and top
management commitment.
Internal influences Employees
DQ activities
, 2013
10 | P a g e
2.4 Performance measurement
Lichiello & Turnock (1997) defines performance
measurement as the regular collection and reporting of
data to track work produced and results achieved. There
are some basic components of performance measurement.
These components are keys to developing an effective,
user centered and trusted performance measurement
process and include (Lichiello & Turnock, 1997):
Incorporating stakeholder input.
Promoting top leadership support.
Creating a mission, long-term goals and
objectives.
Formulating short-term goals.
Devising a simple, manageable approach.
Providing technical assistance.
Performance measurement should be a multidirectional
process, running top-down, bottom-up and horizontally
within and across the organization where continuous
stakeholder involvement and continuous communication
forms the basis for improvement (Lichiello &Turnock,
1997). Therefore, only defining a set of KPIs and
collecting data is not enough. Performance measurement
initiatives need to be supported by a performance
assessment and a strong commitment from leaders in
order to move toward PM (Masayna, et al., 2007).
Antolic (2008) states that a successful software enterprise
implements measurement in order to provide objective
information necessary for the decisions that positively
impact the business. So, a PM culture helps the project
manager to perform a better job, implement more realistic
plans and accurately monitor progress against those plans.
Nazemi & Tarokh (2006) emphasize that an enterprise
should establish a process for both analyzing and
reporting performance data as well as a process for using
performance information to drive improvements forward.
Furthermore, Nazemi & Tarokh (2006) state that a
successful performance measurement should be based on
the following principles:
1- Measure only what is important.
2- Focus on user’s needs.
3- Keep integrated measurement approach in mind.
4- Involve employees in the implementation of the
PM process.
3 RESEARCH APPROACH
This section describes the research approach and process
adopted to fulfill the goal of this study.
3.1 Research setting
Sigma-Kudos, established in 2007, is an international
company with a focus on developing technical
documentation. It has offices in Sweden, Finland,
Hungary, Ukraine and China. Sigma-Kudos currently
employs over 400 specialists within the field of technical
and product documentation and related services such as
embedded design and information management. The
research was conducted in the Gothenburg office, which
greatly facilitated the empirical data collection.
3.1.1 Method
In order to get deeper insights into a phenomenon a
qualitative research approach is the most suitable to apply
(Creswell, 2009). In view of this, the present research was
designed as a qualitative inquiry, carried out in two
phases:
The first phase was based on data gathering by
conducting a literature review.
The second phase included conducting
interviews.
Thus, the resulting framework is a synthesis from two
different sources of data making the study robust.
3.2 Data collection
This part describes the two phases and how it was
conducted.
3.2.1 Literature review
Initially, articles, books and journals were reviewed in
order to get a better understanding of the literature in the
area of measuring the perceived quality of technical
documents. The identified key words for this study
included: benchmarking, checklists, data quality, key
performance indicators, perceived quality, performance
measurement, surveys and user satisfaction. These key
words were used to identify the main area of research, as
, 2013
11 | P a g e
well as bringing clarity to the existing knowledge within
performance measurement of technical documentation
(Sorensen, 2005).
3.2.2 Interviews
Semi-structured interviews were carried out to collect
various perspectives of the CPI and to get to the core of
perceived problems. The interview guide was designed to
capture the perspectives of the specialists employed by
Sigma-Kudos. The outcomes of the interviews were
written down as notes which then formed the basis for the
analysis. In order to make the interview process as
effective as possible the following steps were taken into
consideration (Creswell, 2009):
Ensuring that the participants felt comfortable.
Assuring that the answers were treated
completely anonymously.
Refraining from using leading questions.
According to Creswell (2009) these principles contribute
to enhancing the quality of interview data.
3.2.3 Participants
The interviewees consisted of two technical writers who
have relationships with the user and a CPI system
manager. The technical writers are continuously
interacting with the user of the CPI meaning that they
often have a good understanding of their needs and
opinions. The CPI system manager usually has a lot of
experience and knowledge about CPI. These stakeholders
have knowledge regard CPI in general.
3.3 Limitation
The interviews were performed with participants who
have direct relationship with the users rather than the
users themselves. Even if these stakeholders do
understand and know users’ concerns, their perspectives
cannot fully reflect the user’s point of view. While this
could potentially affect the validity of the interview data
in this study, the interviews with the technical writers are
still considered important in terms of indicating issues
and problems in relation to improving the quality of
technical documentation.
4 FINDINGS
This section presents the resulting tentative framework for
measuring perceived quality of CPI document.
4.1 Interview outcomes
From the interview data it is clear that there is a need for a
more systematic way of measuring perceived quality in
documents. As expressed by one of the respondents:
“We have some KPIs. These are used to check the quality
of documents but we have never used surveys to measure
these KPIs.”
Another respondent emphasized that:
“We meet the user continuously and give us relevant
feedback regarding the status and quality of CPI.”
Further, it is clear that the respondents think that the
organization should decide what business strategies to
adopt to improve the quality of costumer product
information.
The main issues regarding document quality as raised by
the respondents are:
Hard to access “users want to access
information quickly because sometimes in their
daily work they do not have time enough to
access the document.”
Hard to understand “users do not have the same
level of knowledge or qualification this implies
that they need to access the right information
based on their needs.”
Incomplete “users thought that there is lack of
information in the document’s content and this
affect the use of the document itself.”
This kind of information can be gathered during the
conduction of structured-surveys and the data could be
useful for measurement of perceived quality in CPI
document.
, 2013
12 | P a g e
4.2 Tentative framework
Based on the literature review and empirical study it has
been possible to develop a tentative framework, which
visualizes the main components of performance
measurement process (see Figure 3). The framework is
the main contribution of this study and all components
embraced to it are pivotal for the conduction of
measurement process.
A description of the main components is as follows:
KPI. Need to be well defined and linked to
business objectives (Vial & Prior, 2003). They
measure the activity goals, which are the actions
an organization has to take in order to achieve a
successful process performance (Masayna, et al.
2007).
Definition. Rate KPI 1 to 10 scales and conduct
statistical analysis where 1 implies the user is
totally dissatisfied and 10 means totally satisfied
(Xenos & Christodoulakis, 1997).
Objective. Establish the goal of the measurement
for instance to check whether the document is
well accurate (Masayna, et al. 2007).
Type. Need to be qualitative measurement based
on user satisfaction of document quality
(Masayna, et al. 2007).The alternative is a
quantitative measurement the company need to
decide which one to apply for each specific case.
Effort. Establish priority, how important it is this
specific measurement compared to others by
assigning high, medium or low priority
(Masayna, et al. 2007).
Approach. Carry out surveys and questionnaires
designed for the user in order to determine how
satisfied the user is, understand his/her behavior
and gather data for the measurement by applying
the 1 to 10 scale above (Xenos &
Christodoulakis, 1997 and Masayna, et al.,
2007).
Analysis frequency. Decide when a
measurement activity has to be performed for
example daily, weekly, monthly, quarterly or
yearly(Masayna, et al. 2007). Usually enterprises
perform analysis every year or every three
months it depends on organization needs.
KPI Name Accuracy
Purpose To determine the level of user satisfaction with the accuracy of the CPI document.
Definition How satisfied the user was with the accuracy of the CPI document using a 1 to 10 scale, where: 10 = totally satisfied. 8 = mostly satisfied. 5/6 = neither satisfied nor dissatisfied. 3 = mostly dissatisfied. 1 = totally dissatisfied.
Objective To check whether the CPI document is developed accurately.
Type Qualitative
Effort High
Assessment Approach
1- Carry out a structured survey to determine how satisfied the user was with the accuracy of the CPI document, using the 1-10 scale above.
2- The user satisfaction- Accuracy is the user’s rating out of 10.
Analysis frequency
Quarter
Figure 3: Tentative framework for measurement of KPIs
(Masayna, et al. 2007).
More detailed information regarding the application of the
framework and the findings will be discussed later into
discussion section. These will be discussed in light of the
literature review and interviews with the stakeholders, in
order to identify needs for further studies.
, 2013
13 | P a g e
5 DISCUSSION
This research is concerned with how to measure perceived
quality of CPI documents developed by Sigma Kudos. It
builds upon a recent study, which focuses on identifying
KPIs for measuring document quality. The resulting
framework as presented in this study therefore connects
the KPIs with an approach of how to measure document
quality. There is a gap between what the literature review
yields and how the organization applies its quality
assurance process. For instance, structured surveys
together with KPIs are currently not used to measure
perceived quality of CPI documents. In order to adapt an
effective measurement approach the KPIs need to be rated
1 to 10 scale and metrics could be based on structured
surveys. So, metrics can easily be automated and
empirical data is continuously collected.
This study emphasizes the importance of using surveys to
collect user data rather than checklists. According to
Xenos & Christodoulakis (1995, 1997) structured-surveys
allows an organization to collect empirical data necessary
for performance measurement and with less degree of
errors.
The main contribution of this study is the tentative
framework, which is based on a literature review and
empirical data. Underpinning this framework is the design
of suitable questionnaires for the measurement of CPI
document quality. The framework, yet to be tested, is
intended to support performance measurement of
technical documentation and thus become a potential tool
for continuous quality measurement. Further, it needs to
adapt so it can fit to different circumstances and contexts
for the best practice.
Sigma Kudos could implement the framework into their
existing quality assurance activities, though some
adaption and adjustment would be needed to make it
work. The framework can be used for the collection of
empirical data regarding the status of the CPI document.
Data could be collected by caring out structured surveys
and this data can be used for the measurement of
perceived quality of the document. Due, the approach
requires continuous interaction with the users of CPI, in
order to measure perceived quality. This indicates the
profound importance of deployment strategy in managing
users PQ, especially when a user’s expectations are high.
5.1 Application of the framework
Measurement is an iterative process where KPIs are
refined in order to capture organization business
objectives (Antolic, 2008). It is impossible to make
decisions and improvements without having data coming
from the measurement that aids the organization to take
some actions. The tentative framework in this study has
been developed in order to support the collection of
empirical data coming from user feedback regarding
perceived quality of technical documentation. While the
framework is designed for CPI document, it can be
applied to other items and sectors.
Since, Sigma Kudos develops different kind of technical
documents and products the framework could be adapted
to each specific item needs and adjusted according to
those needs. For instance, the structure and the skeleton of
the framework can easily be applied to assessment of
quality assurance process, but the design and development
of KPIs as well as questionnaires should be related to
each specific organization business objective needs. Any
company who wants to measure performance should keep
in mind the following:
First, decide what to measure by applying the
principles listed by (Nazemi & Tarokh, 2006):
1. Measure only what is important.
2. Focus on user’s needs.
3. Keep integrated measurement approach in
mind.
4. Involve employees in the design and
implementation of the measurement process.
Second, KPIs and data should be respectively
well defined (Masayna, et al., 2007).
Third, allow the KPIs to be rated on a 1 and 10
scale. According to Xenos & Christodoulakis
(1995, 1997) statistical analysis has a vital role
in quality improvement process.
Fourth, decide the category and the effort. For
example category could be qualitative and effort
could be how you prioritize low or high effort
(Masayna, et al., 2007).
, 2013
14 | P a g e
Fifth, include control questions and safeguards in
the questionnaires (Xenos & Christodoulakis,
1995, 1997).
Finally, conduct structured surveys regarding
user satisfaction of document quality (Masayna,
et al., 2007).
5.2 Surveys vs checklists
5.2.1 Surveys
Surveys are effective in term of gathering data. In the
context of quality management, KPIs are measured
through the administration of surveys to continuously
measure quality. The rationale behind surveys is that they
are designed to capture the users’ opinions.
Advantages:
User oriented. User fills in the surveys.
Flexible. Can be used for different purposes.
Effective. Allows for gathering of large amounts
of data in very short time.
Disadvantages:
Validity. There is no guarantee that what is
intended to be measured is actually measured.
5.2.2 Checklists
According to Punter, (1997) the checklist approach is a
good technique for measuring quality, but normally
checklists are not used for measuring KPIs. Further,
checklist is a technique to manage items during an
evaluation.
Punter, (1997) suggests that three subjects should be
addressed to provide objective and reproducible
evaluations:
Determination of the indicators. Suitable
indicators and measures which determine a
quality characteristic are chosen.
Procedure. Checklist requires instructions for an
evaluator in order to provide reproducible
measurements.
Judgment. After having determined the value of
indicators the degree of satisfaction about the
characteristic of the product has to be
established.
Advantages:
Easy to control. Evaluator fills in the checklist
appropriately.
Easy to use. It requires minimal effort.
Disadvantages:
Less credible. Compared to surveys.
5.3 Benchmarking
Before starting the collection of data it is good practice to
establish how the data will be used. Arayici, et al. (2009)
argues that a benchmarking approach should be identified
in order to compare different KPIs results and achieve
improvements. Benchmarking allows the comparison
among different data results and this can be done both
with internal outcomes as well as external the target is to
reach progress.
Thus, benchmarking can be embedded into the
measurement process, as a complementary measurement
approach and it is not mandatory not to adopt. Therefore,
performance measurement can be conducted with or
without applying benchmarking (Antolic, 2008).
According to Hatry (1999) the traditional benchmarking
method is based on comparing current performance level
to that of previous years and allows any organization that
wants to measure performance to make targeted
improvement. Furthermore, benchmarking through the
use of KPIs helps companies to improve performance,
motivate employees by giving measureable goals to
achieve and see how the organization measures up to
others in the industry.
, 2013
15 | P a g e
5.4 Evaluating the framework
Nazemi & Tarokh (2006) state that a successful
performance measurement should focus on customer’s
needs and it should be a feedback loop between customer
and developer where data or information regarding the
product is shared. Also, taking some action is required in
order to make improvements. The quality manager should
decide how to apply the framework and which technique
to use in each specific situation.
The main advantages of the approach are that it fits into
almost every quality assurance framework while also
offering enhanced collaboration with the users. The main
drawback is that it is not tested with documents yet. In
addition, there are costs incurred to deploying the
techniques in terms of human factors such as subjectivity
judgment involved with surveys.
Both advantages respective disadvantages of the
framework are as following:
Advantages:
It is easy to comprehend as a framework.
It is flexible and can be adapted to different
circumstances/contexts.
It supports the gathering of credible qualitative
data.
Disadvantages:
It has not yet been tested in reality.
The real benefits can only be shown after the
framework has been tested.
5.5 Validation of the measurement approach
Although it has not been possible accommodate a full
industry validation of the framework, it has been
evaluated by one of Sigma Kudos managers. From his
point of view the framework is applicable to measuring
perceived quality of technical documentation. Further, he
embraces the idea of using surveys in order to collect data
regarding the current status of CPI document.
There is recognition within Sigma Kudos that to enhance
the producer-customer relationship, this requires a method
in order to measure the customer perception of document
quality. It is suggested that to evaluate the potentiality of
the approach the company could perform measurement
activities using both surveys and checklists. They could
then compare the outcome of surveys with checklists and
see how the quality of CPI changes over time as well as to
identify potential issues that can then be investigated in
detail.
5.5.1 Limitation
The framework presented in this thesis is tentative and
therefore it needs to be validated through testing.
Specifically, it needs to be implemented in an
organization’s quality assurance activities and endorsed
by management as part of evaluating its effectiveness in
practice.
5.5.2 Issues and challenges
During the measurement process all elements and factors
mentioned in this research should be considered in order
to increase the quality of the measurement. Embracing
and using together various aspects present in this paper is
the key for a successful application of the approach this
will may arise some challenges. For example, each
specific context is unique and applying the framework to
different situations may require particular resource.
Performance measurement needs to be adopted by any
kind of business oriented enterprise that wants to measure
critical factors related to business activities. However,
supporting and applying it a proper way can be a
challenge. There are many factors and elements to take
into consideration to make it work properly and obtain the
desired effects. From this perspective, Sigma kudos could
therefore attempt to apply the framework to other types of
documents that requires user interaction.
6 SUGGESTED FUTURE RESEARCH
Initially the measurement framework as presented in this
study needs to be tested as to evaluate its real potential in
terms of enhancing the quality of CPI documents. In other
words, Sigma Kudos can start the verification activities
by testing and evaluating the potentiality of the
framework. For instance by testing the framework it
should be possible to discover all issues that can be
, 2013
16 | P a g e
related to the application of the framework. This would
also benefit the improvement of the KPIs relating to
document quality. The next step is to apply the framework
to other document types. This is certainly a worthy
research avenue to follow.
As for the future of measurement of document quality,
more research is needed to validate the findings of this
study and investigate how the PQ of documents in general
can be measured. Thus, future research should focus on
discovering what the best practice for measurement of
document quality is.
7 CONCLUSION
The aim of this study has been to develop a tentative
framework for how to measure the perceived quality of
technical documentation. As there has been little research
conducted in this area the study fills an important
knowledge gap. Employees working in Sigma Kudos
were interviewed in order to capture their views and
needs with regards to perceived quality of CPI
documents. The data from the interviews along with the
results from the literature review formed the basis for
developing a tentative framework to measure CPI
document quality. The aim of the framework is to find a
solution for quality measurement activities that focuses on
customer requirements, in controlling measurement
results and increasing confidence to both the company
and the users. Both the measurement process and the KPIs
should continuously be evaluated and improved according
to the users’ needs. The proposed research contributes to
both theoretical and practical knowledge to the field of
measuring PQ.
The main implication for industry is centered on the
benefits for Sigma Kudos and to support the
implementation of a systematic way of measuring the
quality of their CPI documents more effectively. The
tentative framework is developed for this purpose. It is
intended to aid a quality assurance manager to
systematically measure the perceived quality of the CPI
and validate the potential benefits of it. This way the
study has implications beyond the case organization.
In the future more demands on improving document
quality will be made, hence there is a need for companies
to develop and implement performance measurement
system that are fit for purpose and relevant. A framework
for measuring quality is the first step in achieving this.
8 REFERENCES
Antolic, Z., (2008). An Example of Using Key
Performance Indicators for Software Development
Process Efficiency Evaluation. Technical Report, R&D
Center, Ericsson Nikola Tesla d.d..
Aaker, D. A., (1991). Managing brand equity:
Capitalizing on the value of a brand name. Journal of
Business Research, 1994, vol. 29, issue 3, pages 247-248.
Arayici, Y., Coates, P., Koskela, K., Kagioglou, M.,
Usher, C. and O’Reilly, K. (2009b). BIM implementation
for an architectural practice, Proceedings of the Managing
Construction for Tomorrow International Conference,
October, Istanbul.
Bobrowski, M., Marré, M. and Yankelevich, D. A.,
(1998). Software Engineering View of Data Quality.
Proceedings of Second International Software Quality in
Europe.
Creswell, J.W., (2007). The Mixed Methods Reader, 1st
edition, Sage Publications: London.
Creswell, J. W., (2009). Research Design: Qualitative,
Quantitative, and Mixed Methods Approaches, SAGE 3rd
edition.
Fogelström, N., D., Gorschek, T., Svahnberg, M. &
Olsson, P., (2009). The Impact of Agile Principles on
Market-Driven Software Product Development, Journal of
software maintenance and evolution-research and practice
Vol. 22.
Glasow, P., (2005). Fundamentals of survey research
methodology. Retrieved November 8, 2012, from
http://www.mitre.org/work/tech_papers/tech_papers_05/0
5_0638/05_0638.pdf
Hatry, H.P., (1999). Performance Measurement: Getting
Results, Urban Institute Press, Washington USA.
Jones, C., (1991). Applied Software Measurement:
Assuring Productivity and Quality’. McGarry, J., (2003).
, 2013
17 | P a g e
Measurement Key Concepts and Practices, Practical
Software & System Measurements, USA.
Lahlou, S., Van der maijden, R., Messu, M., Poquet, G.
and Prakke, F., (1992). A Guideline for Survey −
Techniques in Evaluation of Research, Blussels,
ESSC−EEC-EAEC.
Lichiello P., Turnock B., (1997). Guidebook for
Performance Measurement. Turning Point: National
Program Office at the University of Washington.
Masayna, V., Koronios, A., Gao, J. and Gendron, M.,
(2007). Data Quality and KPI’s: A link to be Established,
paper presented at the Second World Congress on
Engineering Asset Management and the Fourth
International Conference on Condition Monitoring.
Nazemi, E. & Tarokh, M. J., (2006). Performance
Measurement in industrial organizations, case study:
Zarbal Complex. Vol. 2, No. 3, 54-69.
Punter, T., (1997). Using checklists to evaluate software
product quality, proceedings of the 8th
European Software
Control and Metrics Conference (ESCOM ’97), The
ESCOM Conference, Reading, UK, pp. 143-150.
Sommerville, Ian., (2010). Software Engineering, 9/E.
Addison-Wesley.
Sorensen, (2005). This is not an article: just some
thoughts on how to write one. Revised version of working
paper with same title. Department of Information
Systems, the London School of Economics and Political
Science.
Stevens, SS., (1946). On the Theory of Scales of
Measurement. Science. 103(2684):677–680.
Vial, D. & Prior, M., (2003). Use of Key Performance
Indicators in the Planning and Management of Public
Open Space. Proceedings of PLA Conference, Perth.
Wang, R.Y., Storey, V.C., and Firth, C.P., (1995). A
framework for analysis of data quality research. IEEE
Trans. on Knowl. Data Eng. 7, 4 pp. 623–640.
Wang, R.Y., Reddy, M. P., and Kon, H.B., (1995).
Toward quality data: An attribute-based approach.
Decision Support System. pp. 349–372.
Wingkvist, A., Ericsson, M., Lincke, R. and Löwe, W.,
(2010). A Metrics-Based Approach to Technical
Documentation Quality, in Proceedings of the 7th
International Conference on the Quality of Information
and Communications Technology (QUATIC’2010).
Xenos, M. & Christodoulakis, D., (1995). Software
Quality: The User’s Point of View, pp. 266-272 of
Software Quality and Productivity, Chapman & Hall,
isbn: 0-412-62960-7.
Xenos, M. & Christodoulakis, D., (1997). Measuring
perceived software quality, Pre-print version of the paper
published in Information and Software Technology
Journal, Butterworth Publications, Vol. 39, Issue 6, pp.
417-424.
APPENDIX
Interview guides
The appendix introduces interview questions which used
during the interviews described in this study.
Collection of demographic data
Name?
Age?
Gender?
Position?
Data collection
How you define a KPI?
Why you define a KPI?
How do you use KPI?
What do you measure?
How do you use the outcome of the
measurement?
What is the current state of CPI according to
users?
How the PQ of CPI can be increased according
to users?
When you perform a measurement process?
Who performs the measurement?
, 2013
18 | P a g e
, 2013
19 | P a g e
