View
34
Download
1
Category
Preview:
Citation preview
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
1
Multidimensional assessment ofperformance of organizational units: Integrating BSC and AHP methodologies
in a Brazilian telecom company
Alexandre Bentes, PUC-Rio, Brazil
Jorge Carneiro, PUC-Rio, Brazil
Herbert Kimura, Mackenzie, Brazil
Jorge Ferreira da Silva, PUC-Rio, Brazil
IAG Business School,
Pontifical Catholic University of Rio de Janeiro
March 24 – 26, 2010
Barcelona, Spain
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
2
�Appropriate assessment of organizational performance is essential for firms
�The multidimensional nature of organizational performance poses challenges to researchers and practitioners
�The performance of functional areas within an organization is an instance of the concept of organizational performance in general
�Several studies have
� used BSC to assess the performance of firms and of functional units within organizations
� used AHP to rank decision alternatives
�But few have used both � This is what is new and contributive in our study
Introduction
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
3
�Verify whether an integrated approach – that combines the multidimensional perspective of the Balanced Scorecard (BSC) with the framework for multi-criteria ranking of decision alternatives provided by the Analytic Hierarchy Process (AHP) – can be useful for the assessment of the (relative) performance of functional units within an organization.
�Extend the external validity of both the BSC and the AHP frameworks by applying those frameworks to another industry (telecommunications) and another setting (Brazilian environment and managers) not often researched
Objectives of the study
Multiple perspectives of organizational performance
Multi-criteria ranking of decision alternatives
Multidimensional assessment and ranking of organizational performance
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
4
�In this study the overall performance of three functional areas of the financial department of a Brazilian Telecommunications company was rank-ordered by taking into consideration:
� the relative degree of importance of (four) distinct perspectives of organizational performance,
� the relative degree of importance of performance indicators within each perspective, and
� the (relative) performance of the (three) functional areas in each performance indicator
Objectives of the study (cont.)
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
5
�Organizational performance and the Balanced Scorecard (BSC)
�Comparison and decision-making based on multiple criteria
�The Analytic Hierarchy Process (AHP)
Theoretical Framework
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
6
Organizational performance and the Balanced Scorecard (BSC)
�Organizational performance is a complex and multi-faceted phenomenon (Cameron, 1986; Chakravarthy, 1986; Venkatraman and Ramanujam, 1986)
�Kaplan and Norton (1996) proposed that organizational performance be simultaneously assessed from distinct, but complementary, perspectives, which were originally detailed as:
� Financial
� Customer
� Internal Business Processes
� Innovation and Learning
Theoretical Framework
Financial
Vision and
Strategy
Innovation and
Learning
Internal Business
ProcessesCustomer
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
7
Organizational performance and the Balanced Scorecard (BSC) (cont.)
�While BSC provides a balance (Kaplan and Norton, 1992)
� between short-term and long-term objectives,
� between financial and non-financial measures,
� between lagging and leading indicators, and
� between internal and external performance perspectives”
�BSC approaches, nonetheless, also present some challenges:
� Overload of information
� Necessity to reach some synthesized judgment
Theoretical Framework
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
8
Comparison and decision-making based on multiple criteria
�Some decision-making problems involve the choice among alternatives which must take into account multiple criteria which may present some degree of inconsistency with one another
� The values (performance) achieved in some criteria may suggest a distinct choice (decision) from the values in some other criteria
� Improvement of results under a given criterion may be obtainable only at the expense of another
� This may be particularly true in the assessment of organizational performance and the choice of the best performing unit (or the rank-ordering of the units under evaluation)
Theoretical Framework
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
9
The Analytic Hierarchy Process (AHP)
�A multi-criteria decision-making tool developed by Saaty (1980)
�Decomposition, comparative judgment, and synthesis
�Decomposition of a complex problem into a multilevel hierarchic structure of objectives, criteria and sub-criteria, and alternatives
�Series of pair-wise comparisons of judgments (of child items below a parent node, i.e., the relative contributions of the child items to the parent node)
Theoretical Framework
Objective
Criteria
Sub-criteria
Alternatives
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
10
The Analytic Hierarchy Process (AHP) (cont.)
�Synthesis in an eigenvector (priority vector)
� Judgments express the relative impact / importance of the elements in the hierarchy and are translated into numbers, whichindicate the relative importance (weight, priority) of an element over another in the determination of their respective higher hierarchical level
�Final weights of any given lowest element are calculated from a weighted (by degree of importance) average over all levels of the hierarchy and synthesize the multiple judgments (often not mutually consistent)
�Rank-ordering of alternatives
�Overall importance represents the relative (percent) contribution of a given indicator with respect to the global goal
Theoretical Framework
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
11
The Analytic Hierarchy Process (AHP) (cont.)
�Given the particularities of the present study, AHP has some advantages over other decision-making tools:
� Synthesis of multiple viewpoints (criteria) into a single unified result
� A consistency index can be calculated to assess the transitivityin judgments given by the paired comparisons
� Involvement of decision-makers who have to explicitly reveal their preferences
� Communication among decision-makers tends to lead to better decisions (reached through a process of conflict resolution until
some consensus or, at least, agreement is reached)
Theoretical Framework
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
12
�Units of analysis: three functional areas of the financial department of a Brazilian telecom company – Fraud unit, Collection unit, and Revenue Assurance unit
�Data collection procedures
�Data collected in 2009
� Interviewees and participants
� three senior mangers (one for each unit; all with long experience in the company),
� three supervisors (one for each unit),
� nine senior analysts (three for each unit), and
� the financial operations director.
� � The diversity in hierarchical levels was deemed important in order to reach a more consensual and “department-wide” opinion about performance that would somehow better represent the variety of views in the financial department.
Methods and data
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
13
�Iterative and interactive process of data collection
� For the selection of dimensions and measures of organizational (functional) performance
− four dimensions and nine performance indicators chosen
� For the definition of the weights (degree of importance) of eachperformance dimension and of each indicator (within each dimension)
− Paired comparisons: How much more (less) important is this dimension / indicator as compared to the other one?
�Five rounds of discussion until some agreement was reached
�Role played by the investigator
�Convergent validity check
Methods and data
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
14
Identify the best
performing unitobjective
criteriaFinancial
perspective
Ind A Ind B Ind C
Fraud unit
Customer
perspective
Innovation and
learning perspective
Internal business
process perspective
Ind D Ind E Ind F Ind G Ind H Ind I
Collection unit
Revenue
assurance unit
Fraud unit
Collection unit
Revenue
assurance unit
Fraud unit
Collection unit
Revenue
assurance unit
Fraud unit
Collection unit
Revenue
assurance unit
sub-criteria
alternatives
Methods and data
�The AHP tree
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
15
Training and capability-building activitiesI
Improvement in employees’ capabilitiesHInnovation and
Learning
Number of new projects fully delivered in
the yearG
Number of improvement projects
implemented in the yearF
Internal Business
Processes
Satisfaction level of external clientsE
Satisfaction level of internal clientsDCustomer
Decrease in operating costsC
ROI of implemented projectsB
Recovered value or avoided lossA
Financial
DefinitionIndicatorBSC Perspective
Methods and data
�Indicators selected for the present study
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
16
Financial Ind A Ind B Ind C
Ind A 1 4 7
Ind B 1/4 1 3
Ind C 1/7 1/3 1
0.71
0.21
0.08
Customer Ind D Ind E
Ind D 1 1/7
Ind E 7 1
0.12
0.88
Processes Ind F Ind G
Ind F 1 3
Ind G 1/3 1
0.75
0.25
Innovation
and LearningInd H Ind I
Ind H 1 5
Ind I 1/5 1
0.83
0.17
CR = 0.03
CR = 0.00
CR = 0.00
CR = 0.00
Application of the Analytic Hierarchy Process
�Relative importance,
�Normalized weights, and
�Consistency ratios
at the performance indicators (sub-criteria) level
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
17
BSC
perspectivesFinance Customer Processes
Innovation
and Learning
Finance 1 5 4 5
Customer 1/5 1 3 3
Processes 1/4 1/3 1 2
Innovation
and Learning1/5 1/3 1/2 1
0.59
0.21
0.12
0.08
CR 0.09
Application of the Analytic Hierarchy Process
�Relative importance,
�Normalized weights, and
�Consistency ratios
at the BSC perspectives (criteria) level
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
18
0.71
0.21
0.08
x 0.59
0.42
0.12
0.05
Indicators A,B and C
=0.12
0.88
x 0.21 0.03
0.18
Indicators D and E
=
local
weights
global
weights
local
weights
global
weights
0.75
0.25
x 0.12 0.09
0.03
Indicators F and G
=
local
weights
global
weights
0.83
0.17
x 0.08 0.07
0.01
Indicators H and I
=
local
weights
global
weights
Application of the Analytic Hierarchy Process
�From local weights to global weights of each indicator
weight of
parent
node
weight of
parent
node
weight of
parent
node
weight of
parent
node
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
19
Ind. A Fraud CollectionRevenue
assurance
Fraud 1 1/7 1/2
Collection 7 1 5
Revenue
assurance2 1/5 1
0.09
0.74
0.17
Ind. B Fraud CollectionRevenue
assurance
Fraud1 4 3
Collection1/4 1 1/3
Revenue
assurance
1/3 3 1
0.61
0.12
0.27
Ind. C Fraud CollectionRevenue
assurance
Fraud1 1/4 1/2
Collection4 1 4
Revenue
assurance
2 1/4 1
0.13
0.66
0.21
CR = 0.01
CR = 0.06
CR = 0.05
Application of the Analytic Hierarchy Process
�Relative degree of success in (performance indicators of) the financial perspective by each functional unit
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
20
Ind D Fraud CollectionRevenue
assurance
Fraud1 1/2 3
Collection5 1 7
Revenue
assurance
1/3 1/7 1
0.19
0.73
0.08
Ind E Fraud CollectionRevenue
assurance
Fraud1 1/4 2
Collection4 1 3
Revenue
assurance
1/2 1/3 1
0.22
0.63
0.15
CR = 0.06 CR = 0.09
Application of the Analytic Hierarchy Process
�Relative degree of success in (performance indicators of) the customer perspective by each functional unit
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
21
Ind. F Fraud CollectionRevenue
assurance
Fraud1 1/2 1/5
Collection2 1 1/4
Revenue
assurance
5 4 1
0.12
0.20
0.68
Ind. G Fraud CollectionRevenue
assurance
Fraud1 1/2 1/6
Collection2 1 1/3
Revenue
assurance
6 3 1
0.11
0.22
0.67
CR = 0.02 CR = 0.00
Application of the Analytic Hierarchy Process
�Relative degree of success in (performance indicators of) the internal business process perspective by each functional unit
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
22
Ind. H Fraud CollectionRevenue
assurance
Fraud1 1/5 1/3
Collection5 1 3
Revenue
assurance
3 1/3 1
0.10
0.64
0.26
Ind. I Fraud CollectionRevenue
assurance
Fraud1 1/4 1
Collection4 1 2
Revenue
assurance
1 1/2 1
0.18
0.58
0.23
CR = 0.03 CR = 0.05
Application of the Analytic Hierarchy Process
�Relative degree of success in (performance indicators of) the innovation and learning perspective by each functional unit
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
23
Fraud 0.09 0.61 0.13
Collection 0.74 0.12 0.66
Revenue
assurance0.17 0.27 0.21
Ind. A 0.42
Ind. B 0.12
Ind. C 0.05
x
Fraud 0.13
Collection 0.35
Revenue
assurance0.11
vectorial product
Ind. A Ind. B Ind. C
0.59
=
Fraud0.19 0.22
Collection0.73 0.63
Revenue
assurance
0.08 0.15
Ind. D 0.03
Ind. E 0.18
Fraud0.05
Collection0.14
Revenue
assurance
0.02
Ind. D Ind. E
0.21
x =
vectorial product
Finance
Customer
Application of the Analytic Hierarchy Process�Partial contribution of each functional unit to the overall
performance objective
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
24
Fraud0.12 0.11
Collection0.20 0.22
Revenue
assurance
0.68 0.67
Ind. F 0.09
Ind. G 0.03
Fraud0.01
Collection0.03
Revenue
assurance
0.08
Ind. F Ind. G
0.12
x
vectorial product
=
Fraud0.10 0.18
Collection0.64 0.58
Revenue
assurance
0.26 0.23
Ind. H 0.07
Ind. I 0.01
Fraud0.01
Collection0.05
Revenue
assurance
0.02
Ind. H Ind. I
0.08
Processes
Innovation
and Learning
x =
vectorial product
Application of the Analytic Hierarchy Process�Partial contribution of each functional unit to the overall performance objective (cont.)
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
25
0.240.240.020.080.020.11Revenue assurance
unit
1.001.000.080.120.210.59Total
Finance CustomersInternal Business
Processes
Innovation and
LearningTotalTotal
Fraud unit 0.13 0.05 0.01 0.01 0.190.19
Collection unit 0.35 0.14 0.03 0.05 0.570.57
Application of the Analytic Hierarchy Process
�Final results for assessment of the best performing functional unit
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
26
�Results indicate that the Collection unit performed better than the other units in three perspectives – finance, customer, and innovation and learning – and had the highest contribution (57%) to final objective.
�The Revenue Assurance unit performed better in the internal business processes perspective, but its contribution (23%) to the final decision regarding the (overall) best performing unit is not too different from the contribution of the Fraud unit (20%).
Discussion
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
27
�BSC explicitly incorporates several perspectives (besides the usual financial viewpoint) to organizational performance assessment
�Evaluating performance requires some agreement on the relative importance of performance perspectives and performance indicators in the way to reach some aggregated (synthesized) measure or ranking
�AHP handles multiple dimensions and measures with distinct degrees of importance and translates the overall result into a unified metric, explicitly accounting for incongruence among objectives and mutually inconsistent evaluations
�AHP circumvents the pitfalls of having managers use a simpler, ad hoc, weighing approach to make sense of the multiplicity of performance measures from a balance scorecard
�AHP provides more than an ordinal ranking; it also informs aboutthe magnitude of the difference between alternatives, that is, how much more (less) a functional area is performing better (worse) than another
Conclusions
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
28
�AHP provides insights to managers as to which functional areas are
more likely to contribute to overall success and which deserve more
attention
� but all in relative basis;
� note that if all areas improve, some will still be (relative) underperformers…)
� � so, both a relative and an absolute assessment should be conducted
�Results of the AHP + BSC can assist managers in deciding about:
� bonus distribution, incentive schemes
� identification (not covered in this study) of the possible reasons for poor (or good) performance
Conclusions
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
29
Academic and managerial implications:
�Such an integrated approach (between BSC and AHP) was shown to have convergent validity with overall assessments by a seniordirector while providing much more fine-grained information for future managerial actions.
�The interactive and iterative process employed in this study hasthe additional advantage of leading managers to apprehend the diverse perspectives of performance assessment and to understand the possible trade-offs thereof.
Conclusion
Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
30
Questions and commentsare very welcome!
Alexandre Bentes (alexvb@globo.com)
Jorge Carneiro (jorgemtc@iag.puc-rio.br)
Herbert Kimura (herbert.kimura@gmail.com)
Jorge Ferreira da Silva (shopshop@iag.puc-rio.br)
Recommended