Towards a General Managerial Framework for Performance Measurement: A Comprehensive Highway Maintenance Application

Journal of Productivity Analysis, 8, 127–149 (1997)c© 1997 Kluwer Academic Publishers, Boston. Manufactured in The Netherlands.

Towards a General Managerial Framework forPerformance Measurement: A ComprehensiveHighway Maintenance Application

PAUL ROUSE [email protected] of Accounting and Finance, The University of Auckland, Private Bag, Auckland 92019 New Zealand

MARTIN PUTTERILLDepartment of Accounting and Finance, The University of Auckland, Private Bag, Auckland 92019 New Zealand

DAVID RYANDepartment of Engineering Science, The University of Auckland, Private Bag, Auckland 92019 New Zealand

Abstract

This paper describes the application of Data Envelopment Analysis (DEA) to a highwaymaintenance setting, using measures of inputs, outputs and outcomes reported by NewZealand local authorities. A general framework of performance measurement is developedand illustrated through application to the highway setting. The framework encompasses aperformance pyramid embodying multiple-perspectives of the organisation with a structureof measures linking critical success factors to process drivers, methods of data analysisand influencing factors such as professional culture. Distinctions between measures ofoutcome, output and input enable finer partitioning of analyses into managerial notions ofefficiency, effectiveness and economy. The impact of environmental factors on efficiencyis explored through two approaches suggested in the literature.

Keywords: Data Envelopment Analysis, highway maintenance, performance measurement, environmentalfactors.

1. Introduction

This paper describes an application of Data Envelopment Analysis (DEA) to the perfor-mance measurement of highway maintenance activities of New Zealand territorial localauthorities (TLAs). Studies of DEA applications to highway maintenance in Canada havebeen reported by Cook et al. (1990) and (1994) who used selected measures of outcomesas outputs, and expenditure and climatic factors as inputs. Deller and Nelson (1991) usednetwork size as an output and material, labour and capital as inputs.

This paper extends these studies through consideration of additional output and inputmeasures and the incorporation of the analysis into a general performance measurementframework. Furthermore, an interpretative structure is provided which enables notions ofeffectiveness, efficiency and economy to be described and measured. The importance ofenvironmental factors is stressed in the paper and incorporated into the analysis in twoseparate approaches.

128 ROUSE, PUTTERILL AND RYAN

The paper proceeds as follows. The next section describes the New Zealand local govern-ment structure and changes over the past decade in contracting and reporting requirements,which have led to a need for better methods of data analysis. Section 3 discusses the valida-tion of measures and results and describes a general managerial framework for performancemeasurement with an illustration of its application to a highway maintenance setting. Inaddition, the influence of a “professional culture” is proposed witha priori expectations ofits effect on structural efficiency. Section 4 describes the data set and reports the resultsusing DEA for several trials, moving from a base set of inputs and outputs to a modelsupplemented by measures of outcomes and environmental difficulty. Further analysesare obtained through a finer categorisation of measures into those pertaining to efficiency,effectiveness and economy. Section 5 extends the analyses through the treatment of environ-mental measures as exogenous to the DEA model and the concluding section summarisesthe results and provides suggestions for future research.

2. New Zealand Local Government Background

There are seventy three TLAs in New Zealand (NZ) responsible for local governance andthe provision of a range of services including the maintenance of local highways. Highwaymaintenance is a major item of expenditure for local authorities exceeding NZ$300 millionin the 1993/94 year. Approximately half of this cost is met directly by TLAs with theremainder funded by a central government organisation, Transit New Zealand, a body alsoresponsible for national highways. Local authority highway maintenance activities andexpenditures are monitored on behalf of central government by Transit NZ by means ofperiodic reviews and audits of selected TLAs.

The past decade in NZ has seen major changes in central and local public sector organisa-tions. One such change has been the separation of purchase from the provision of serviceswhich has led to a large number of tasks being contracted out by competitive tender to pri-vate contractors or specialist state owned enterprises. Highway maintenance tasks whichwere previously performed by TLAs themselves, have either been gathered together intoseparate “quasi-private” local authority trading entities or contracted out altogether. Themarket has thus become the arbiter of efficiency with contracts awarded predominantly ontender price with performance requirements specified in contract form. Competitivenessin the market for highway maintenance services may vary according to region, dependingupon contractor numbers and the intensity of competition.

Long established procedures required TLAs to report measures of highway outputs andexpenditure. Increased accountability provisions were embodied in the Public FinanceAct 1989, which establishes criteria for monitoring performance based on the distinctionsof input, output and outcome, the theoretical properties of which were articulated by Ra-manathan (1985). Since 1989, these additional reporting requirements for outcomes havesupplemented traditional measures. A comprehensive data base for extensive areas ofthe highway network has been established which is now available to report performanceparameters including roughness and other defects.

As in many organisations, TLAs tend to have an abundance of data and measures, but apaucity of systematic, consistent and objective methods of data analysis. Improved methods

A GENERAL MANAGERIAL FRAMEWORK FOR PERFORMANCE MEASUREMENT 129

of data analysis are needed for budgetary and resource allocation decisions as well as forfacilitating the monitoring process of Transit NZ. In addition, attempts to achieve continuousimprovement of necessity involve benchmarking and the identification of comparable peers.Finally, a method of data analysis is needed that is capable of accommodating multi-faceteddimensions of complex organisation activities.

DEA, with its frontier based notion of efficiency and identification of peer units andtargets for non-frontier units, is well suited to these tasks. Although organisations frequentlyemploy single measures of efficiency, these depict a limited perspective of their activities(e.g. cost per kilometre). The combination of multiple inputs and outputs into a single,composite measure of efficiency enables managers to quickly evaluate a unit’s performance.The problems of appropriate choice of weights cannot be ignored as managers may disagreewith the choice of weights which do not present them in the best possible light. This problemis addressed by DEA but difficulties remain with allowing total flexibility in assigningweights (Cook et al., 1994, p. 197).1 Additional advantages of DEA include the capabilityof incorporating non-economic factors as inputs or outputs, and the absence of necessity tospecify a full functional relationship between inputs and outputs. The latter advantage isparticularly important in highway maintenance given the complexity and number of possibleinfluencing variables.

3. Validating the Measures and Results from DEA

As single measures of performance capture only a limited perspective of an organisa-tion’s activities, several measures are generally used in combination to gauge organisationalperformance. This combined “view” is obtained either through some implicit subjectiveweighting of different measures to get the “whole picture”, or through explicit weights.DEA provides a set of weights which optimise a unit’s performance subject to the weightsnot leading to any other unit violating the bounds of the frontier. However, a major method-ological problem is the validity of the measures or inputs and outputs used in the analysis.Even a combination of several measures may not ensure that all relevant aspects of themeasurement object (e.g. productivity) are captured. There is therefore a need to ensurethat the measures used in a DEA model reflect the relevant perspectives of the organi-sation. Although different suggestions have been made in the literature regarding DEAmethodology (Golany and Roll 1989), the impact of DEA measures in the context of or-ganisational control systems (Epstein and Henderson 1989), and problems with variableselection (Smith 1993; Sexton et al. 1986), little has been written on relating the use ofDEA to a general framework of performance measurement as part of a validation process.A suggested framework for this purpose is described in section 3.1 below.

The validity of DEA results is often supported by testimony of their usefulness by man-agers within the organisation or by a description of changes made in response to these resultse.g. Sherman and Ladino (1995). “Professional culture” is a further dimension that can as-sist in evaluating efficiency results. This notion, which will be amplified in section 3.2,suggests that where professionalism is high, structural efficiency should also be high. Thus,in the absence of other compelling reasons, DEA results may be suspect from studies oforganisations characterised by high professional involvement that show low structural effi-


ciency, given that professional standards and peer influence should ensure high standardsof performance. Admission criteria and monitoring processes to ensure that standards aremet are common characteristics of professional culture environments.

In the case of highway maintenance, managers are typically professional engineers whoare aware of standards of highways in areas managed by their peers and would be ex-pected to aim for similar standards. Audit and review of their areas by other professionalengineers from Transit NZ provide additional pressure. Therefore, structural efficiency isexpected to be high, and variation in structural efficiency is expected to be low for highwaymaintenance.2 Results that show otherwise may be a warning of inappropriate selection oromission of measures.

3.1. A Performance Measurement Framework

Several frameworks embodying multiple perspectives have been suggested in the literature.Total Factor Productivity measures linked to profitability are described in Verma (1992)and Hansen, Mowen and Hammer (1992). A wider and currently popular framework isprovided by the balanced scorecard approach of Kaplan and Norton (1992) encompassingfour perspectives: customer, internal business, financial and innovation and learning. Goalsand measures are developed for each perspective embodying the most critical aspects ofan organisation’s performance. A different approach but with similar notions is providedby the performance pyramid suggested by Cross and Lynch (1989) in which measures ofcustomer satisfaction, flexibility and productivity link the strategic vision of the organisationto underlying operational measures categorised under quality, delivery, process time andcost. A hierarchical system of measures linking strategic direction reflected by criticalsuccess factors (CSF) to underlying cost or process drivers has also been suggested byBeischel and Smith (1991).

There are several general principles that emerge from these suggested performance mea-surement frameworks. First, in the current environment, different perspectives must beconsidered in contrast to a traditional single focus on financial performance. Second, mea-sures must reflect critical success factors which embody the strategic directions chosenby the organisation. Third, underlying processes and cost drivers must be identified andjustified in order to explain changes in performance measures and to illuminate areas whereappropriate actions need to be taken.

Informed by the balanced scorecard approach of Kaplan and Norton (1992) and theperformance pyramid of Cross and Lynch (1989), the schematic model depicted in Figure1a has been developed where each face of the pyramid reflects a different perspective. Thereare thus four faces of the pyramid, each face respectively representing: (i) customer, (ii)internal business, (iii) financial and (iv) innovation and learning perspectives with illustrativecritical success factors (CSF). For each face of the pyramid, Figure 1b shows a hierarchy ofcritical success factors embodying strategies. In this way, organisation vision and measureslinking critical success factors to underlying process drivers, are drawn together as suggestedby Beischel and Smith (1991). The linked structure of measures, critical success factorsand process drivers is combined with the performance pyramid and balanced scorecard inFigure 1c. Not only are measures and process drivers linked on each side of the pyramid,


Figure 1a.The integration of the Balanced Scorecard with the Performance Pyramid.

but linkages also exist to other sides of the pyramid as shown by the dotted lines. Thisenables managers to understand the impact of process drivers on more than one key resultarea e.g. productivity and quality (Maani, Putterill and Sluti 1994).

The pyramid represents a comprehensive, fully integrated performance measurement sys-tem that captures multiple perspectives, ensures that measures reflect strategic directionsand provides explanation and choice of actions through identification of underlying drivers.The relationship between DEA and the performance pyramid is shown in Figure 2. TheDEA analysis provides information for productivity and performance measurement, bench-marking and comparisons of actual versus target measures which can be fed back into theperformance measurement system.

From a DEA perspective this framework has several points of recommendation. First,it enables the results of the DEA analysis to be readily interpreted in terms of underlyingprocess drivers. Second, it clearly shows a logical separation of data analysis from theperformance measurement system and reveals that both systems are necessary for worth-while performance evaluation. Third, the measures used in the DEA model can be locatedwithin the linked hierarchical structure to ensure that they capture the main elements oforganisational performance, thereby providing strong justification for the validity of themeasures used. Fourth, the identification and location of measures within the hierarchicalstructure and performance pyramid has important consequences for information system


Figure 1b.Levels of the linked structure for each side of the Performance Pyramid.

Figure 1c.Fully integrated performance measurement system with the Balanced Scorecard, Performance Pyramidand linked measures.

design. Fifth, the interrelationships described by the linked structure provide opportunitiesfor measurement of association and causality between measures for possible incorporationinto decision support models (e.g. simulation and causal modeling).

An illustration of the internal business side of the performance pyramid in a highwaycontext is shown in Figure 33. The critical success factor to illustrate the quality of man-agement of the highway asset is measured by two outcomes: surface/pavement conditionand ride quality. Major outputs affecting these outcomes are resealing, rehabilitation andgeneral maintenance activities. Due to recent changes in the NZ public sector, most if not


Figure 2. The relationship between DEA, the Balanced Scorecard and the Performance Pyramid.

all work is performed by independent contractors which means that use is made of only asingle input, cost, which is driven by various process drivers. The latter have been groupedtogether under categories depicting the competitive environment, maintenance environmentand maintenance policy. The maintenance environment describes the factors bearing di-rectly upon the highway itself whereas maintenance policy describes those influences andfactors which are considered by the highway engineer in determining appropriate treatmentpolicy. Policy is also affected through linkages to other faces of the pyramid for items suchas budgetary constraints (financial perspective) and level of service (customer perspective).

The important prerequisite for any serious attempt to validate performance measures usedin data analysis is detailed understanding of the organisation and systems/processes. It isworth reiterating that an understanding of structure alone may not be sufficient; organisa-tional processes also need to be considered during the validation process.

3.2. Professional Culture and Continuous Improvement

In the public sector, major changes of policy require organisations to respond strategically,giving rise to upheavals which can be traumatic. Yet, in between these significant episodes,there is an expectation that many small steps will be taken to enhance performance throughcontinuous improvement. “Learning to do things better” (Senge 1992) is a key organisa-tional quality which encompasses introspection i.e. re-examination of existing activities,and benchmarking (Pryor 1989). The latter involves organisations deliberately scanningtheir sector of the market to identify best performances, either in cost, price, service, in-novation or some other desired measure. Once located, a critical analysis is undertaken toisolate the reasons for competitor superiority, a process which generally leads to changesin practice which improve performance.

In a similar way, individuals can also pursue the goal of continuous improvement byinternal reappraisal of processes or through critiques of others’ performance if they areknown to be coping competently with similar conditions and challenges. This learningfrom others is a potent force, seen among sports people, entrepreneurs and professionals toname but a few.


Figure 3. Highway network asset management (illustration).

For the purpose of this paper, the centre of interest will be the professional, typified byan engineer responsible for the network of highways in a TLA. As a public servant, thisperson is held accountable for cost management and service performance. Generally, theseobligations are defined and reviewed as part of a recurring annual cycle. Yet, membershipof a professional engineering institute raises other expectations such as staying abreast ofnew developments and fine tuning organisational endeavours.

In contrast to the formal accountability requirements which are essentially backwardlooking, the “professional” posture has the potential to be proactive, very much in the styleof continuous improvement. What matters for individuals is their level of commitmentwhich in part reflects the extent to which enterprise work effort is being expended (Putterilland Rohrer 1995).

Where commitment is high, the highway engineer can be expected to seek diligently forways to do things better and be willing to turn to “peers” for guidance and example. “Peer”in this context implies informal contact with individuals in similar positions in jurisdictions


Figure 4. The three “E’s” and their relationship to measures.

“with an approximately similar range and scale of tasks for whom it might be expected thatoperating costs should not differ significantly” (Putterill 1985, 163).

Through informal networks and the consequent information exchanges between peers, apowerful process can be made to work. Preconditions for success are the level of individualcommitment, a capacity to identify “peer” settings, and the existence of a framework tohelp channel improvement efforts. One framework which might be appropriate is based ona “value for money” objective. Developed with highway maintenance in mind, it links thecycle of highway department tasks to the value achievement parameters of effectiveness,efficiency, economy and accountability (Putterill 1987).

3.3. Value Achievement Parameters

Study of Figure 3 reveals measures falling within the categories of outcome, output andinput. This trichotomy lends itself to the evaluation of effectiveness, efficiency and economy(the three “E’s”) as shown in Figure 4. Outcomes refer to the state of the network conditionas measured by the level of roughness and surface/pavement defects prevailing at a pointin time.4

In contrast, outputs refer to the activities performed during the year which, in highwaymaintenance, are general maintenance, resealing and rehabilitation. Effectiveness, there-fore, can be expressed in terms of the relationship between outputs and outcomes whichreflects how well the network condition has been preserved or altered by activities per-formed.

The inputs consumed to provide outputs enable efficiency to be gauged. Although econ-omy is often related to the relationship between cost and inputs or solely cost, another inter-pretation can be the relationship between cost and outcomes. In NZ where contracting-outhas become widespread i.e. market reliance, the latter interpretation of economy seemsmore relevant, especially from an accountability perspective. References to the three “E’s”in the remainder of this paper should be interpreted in this latter context.


3.4. Environmental Factors

The Ramanathan (1985) framework does not explicitly address the relationship of environ-mental factors to performance though it is possible to include these under the input category.Given that environmental factors can be major “cost and process” drivers, there are goodreasons to give these factors special attention when evaluating performance. Research intothe impact of geological factors on national highway maintenance costs (Rouse and Put-terill 1995) shows geological factors to be significant drivers of the cost of maintenanceactivity.

Environmental factors in that setting provide an interesting and important explanationfor performance, and offer a good example of cost drivers in the context of Activity-basedManagement. Cost drivers in this context relate closely to the fixed factors or constraintsunder which an organisation operates. As such, the resulting organisation structure andprocesses can be seen as an expression of the interaction between management strategy andpolicies with the organisation’s environment.

These environmental cost drivers are an important and essential part of the performancemeasurement structure described above, and very clearly worth considering for inclusionas variables in a DEA model of highway maintenance performance.

4. Data Set and the DEA Model Used for the Application

The previous sections have described the highway organisational environment and a linkedhierarchy of measures depicted in Figure 3. A selection of these measures is next describedfor evaluating the performance of NZ TLAs using DEA.

4.1. Initial Analysis with Successive Inputs/Outputs

Inputs and outputs and additional factors considered for the analysis are as follows:

Inputs:Total expenditure on reseals, rehabilitation and general maintenance (contractor costs);

Outputs:Kilometres of highway resealed;Kilometres of highway rehabilitated;General maintenance as measured by an index of highway surface defectsLevel of service as measured by annual vehicle kilometres;Roughness measures combined for urban and rural highways;

Categorical Variable:An assessment of environmental difficulty faced by each TLA;

Data pertaining to expenditure, reseal and rehabilitation kilometres are available for all 73TLAs for the 1993–94 year. A survey aimed at obtaining measures of vehicle kilometres,roughness and the index of surface defects was directed to 73 TLAs. To date, data from


52 TLAs have been incorporated into the DEA models. The assessment of environmentaldifficulty for each TLA on a scale of decreasing difficulty from 1 to 8 was provided byTransit NZ staff in its Review and Audit department.

Outputs are divided into quantity and quality measures. Reseal and rehabilitation outputsare quantity measures of work performed. Level of service is a quantity measure and ideallywould reflect the utilisation of capacity provided by the highway network, a measure notcurrently reported. It is therefore necessary to fall back on the use of actual vehicle kilome-tres as a surrogate measure. Roughness measures and the measure of general maintenancereflect quality of service. A categorical variable measuring environmental difficulty is em-ployed to take into account differences in geological and climatic environments faced byTLAs.

Special mention must be made of the general maintenance measure. Whereas quantitymeasures are available for resealing and rehabilitation, general maintenance covers a varietyof outputs such as pothole repairs, crack sealing, digouts. The only information availablefrom TLAs is expenditure on this activity for sealed highways. (Measures for unsealedhighways are not currently recorded by TLAs in their formal database). In an attempt toovercome this deficiency, an index of surface condition defects is employed which measuresthe dollar amount per metre of general maintenance expenditure required to rectify observedsurface defects.5 It is therefore more a measure of quality than quantity.

The DEA models used for the evaluation are the standard CCR (Charnes, Cooper andRhodes 1978) and BCC (Banker, Charnes and Cooper 1984) input-oriented models reflect-ing constant and variable returns to scale respectively. Panel 1 of Table 1 contains summarystatistics of the data for the 52 TLAs. Commensurate with the wide variation in the size ofNZ TLAs, there is considerable variation among TLAs for inputs and outputs with generallypositive skewness. Rank correlations between measures are shown in Panel 2. Quantitymeasures are all positively correlated with fairly high correlations between the input, totalexpenditure, and the other outputs. It is interesting that quality outputs are all negativelycorrelated with quantity measures. This confirms the need firstly to include quality mea-sures in any evaluation of performance in a highway context, and secondly suggests thatconsideration be given to other explanatory factors such as urban/rural highway mix, scaleeffects and environmental differences.

The analysis initially uses a base set of inputs and outputs, with further inputs and outputsadded during successive trials. The base set of outputs comprises reseal kilometres, reha-bilitation kilometres and the “surface condition index” with one input, total expenditure.Trials 1 and 2 in Table 2 show the results for this base set under constant and variablereturns to scale (CRS and VRS). There is a significant difference between the trials in boththe mean and median results, indicating the presence of considerable scale effects. This isto be expected given the differences in size of the TLAs.

Trials 3 and 4 in Table 2 include data on vehicle kilometres as additional outputs to thebase set. Major improvement in efficiencies occur in cities with large volumes of traffic.

Trials 5 and 6 in Table 2 use the base set with the addition of environmental difficulty,which is incorporated as a categorical variable. This has a major impact on efficiency andprovides support to the importance of this factor in any analysis of highway performance.However, attention must be drawn to the method by which DEA handles categorical vari-


Tab

le1

.Mea

sure

sus

edin

anal

ysis

.

Pan

el1:

Sum

mar

yS

tatis

tics

Res

eal

Reh

abVe

hkm

sTo

texp

Sur

find

Rou

ghto

tE

nviro

nG

Mex

pU

rbal

enR

ural

enTo

tale

n(k

ms)

(km

s)(1

0’s

mill

ion)

($m

illio

n)(in

dex)

(inde

x)(r

anki

ng)

($m

illio

n)(k

ms)

(km

s)(k

ms)

Mea

n60

.93

8.98

26.2

04.

5711

.32

8.79

4.83

2.58

240

856

1,09

6M

edia

n52

.10

6.80

14.2

53.

618.

266.

885.

002.

1815

665

387

5S

tand

ard

Dev

iatio

n37

.95

7.28

37.7

13.

1511

.04

7.76

1.91

1.67

262

838

806

Kur

tosi

s0.

020.

3912

.57

1.64

15.8

48.

89(0

.56)

(0.1

2)7.

768.

389.

43S

kew

ness

0.89

1.10

3.31

1.29

3.49

2.67

(0.4

8)0.

832.

662.

322.

44R

ange

160.

1027

.70

207.

8415

.37

68.0

142

.16

7.00

6.87

1,26

04,

745

4,89

7M

inim

um1.

90—

1.28

0.17

2.05

1.54

1.00

0.11

202

41M

axim

um16

2.00

27.7

020

9.12

15.5

470

.06

43.7

08.

006.

981,

280

4,74

74,

939

Pan

el2:

Spe

arm

anC

orre

latio

nC

oeffi

cien

ts

Res

eal

Reh

abVe

hkm

sTo

texp

Sur

find

Rou

ghto

tE

nviro

nG

Mex

pU

rbal

enR

ural

enTo

tale

nQ

uant

ityM

easu

res

Res

eal

1.00

0R

ehab

0.52

21.

000

Vehk

ms

0.46

80.

293

1.00

0To

texp

0.78

90.

742

0.50

01.

000

Qua

lity

Mea

sure

sS

urfin

d−0

.465−0

.207

−0.3

08−0

.327

1.00

0R

ough

tot

−0.0

31−0

.115

−0.3

51−0

.113

0.30

11.

000

Oth

erM

easu

res

Env

iron

0.01

8−0

.237

0.13

9−0

.212−0

.204

−0.0

791.

000

GM

exp

0.74

10.

635

0.34

70.

935−

0.39

1−0

.045

−0.1

971.

000

Urb

anle

n0.

429

0.21

30.

837

0.46

5−0.

249−0

.494

0.15

30.

328

1.00

0R

ural

en0.

398

0.25

5−0

.248

0.38

3−0

.243

0.42

9−0

.107

0.55

5−0

.326

1.00

0To

tale

n0.

623

0.41

40.

083

0.66

3−0.

351

0.23

5−0

.026

0.78

80.

021

0.88

61.

000


Table 2.Summary results from successive trials of different input/output combinations.

1 2 3 4 5 6 7 8

CRS VRS CRS VRS CRS VRS CRS VRS

Mean 53.7% 61.5% 60.8% 69.7% 66.0% 73.0% 77.1% 83.9%

Median 46.2% 53.7% 57.3% 67.6% 62.5% 68.1% 76.3% 96.6%

Standard Deviation 0.203 0.236 0.205 0.230 0.225 0.223 0.208 0.193

Sample Variance 0.041 0.056 0.042 0.053 0.051 0.050 0.043 0.037

Kurtosis 0.163 −0.943 −0.419 −1.150 −1.297 −1.501 −1.255 −0.961

Skewness 0.974 0.597 0.494 0.028 0.295 0.008−0.296 −0.747

Range 76.9% 75.8% 76.8% 75.8% 65.8% 62.8% 65.2% 54.8%

Minimum 23.1% 24.2% 23.2% 24.2% 34.2% 37.2% 34.8% 45.2%

Maximum 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

Number 100% efficient 3 11 6 14 8 17 18 25

Outputs:Reseal kms * * * * * * * *

Rehabilitation kms * * * * * * * *

Surface condition Index * * * * * * * *

Vehicle kms * * * *

Roughness—urban and rural combined * *

Inputs:Total Expenditure * * * * * * * *

Environmental difficulty * * * *

ables. Commencing with the most unfavourable measure of environmental difficulty, onlyTLAs falling within this category are evaluated, with successive category levels includingTLAs from previous levels. For example, in category 1, there are only six TLAs, fourof which attain full efficiency using constant returns to scale. Although it appears fair toonly evaluate specific units against those facing similar or more challenging environmentalconditions, the dramatic reduction in the size of the evaluation group for lower levels willtend to increase their efficiency scores. An alternative approach to the direct inclusion ofenvironmental variables in the DEA model is discussed later in the paper.

Trials 7 and 8 in Table 2 include vehicle kilometres, combined roughness measures forurban and rural highways and environmental difficulty in addition to the base set of inputs


and outputs. As expected with the addition of further outputs and the categorical variablereflecting environmental difficulty, efficiency scores improve significantly with almost halfattaining the frontier.6 Variation in efficiency has reduced as shown by the decline in thestandard deviation and range. These results could be considered flatteringly high, were itnot for the prevalence of the “professional culture”. This background would support therejection of trials 1 through 5 in favour of trials 6 to 8.

This initial analysis has several limitations. First, the absence of bounds on the weightsresults in many TLAs disregarding measures that do not favour them. In consequence,TLAs that do not attain reasonably high levels of efficiency, even under the most favourableconditions as in trials 7 and 8, are simply not performing up to the standards set by the otherTLAs. Given that the measures used capture most of the activities and outcomes as reflectedin Figure 3, and taking environmental differences into account, much more vigorous reasonsfor poor performance will need to be sought in a wider context. Possible reasons may include(i) individual circumstances for the 1993–94 year (e.g. a deliberate programme of highwayimprovement—one TLA had an extensive programme of reseals and rehabilitation for theyear resulting in the highest total expenditure of all TLAs); (ii) inaccurate data; (iii) majoremergency work which has not been captured by the measure of environmental difficulty(e.g. earthquakes or unusual flooding).

The second limitation arises due to the treatment of categorical variables in the DEA modelwhich may overstate some TLA efficiencies and understate others. Third, the measures usedare a mixture of outputs and outcomes which are combining both efficiency and effectivenessinto a single measure.

4.2. Analysis Using the Three “E’s”

A more informative view may be obtained by separating out measures of efficiency, effec-tiveness and economy, terms which are described below under these headings following theinterpretations described in section 3.3. The DEA scores are listed in Tables 3 and 4 usingvariable returns to scale and the categorical variable, environmental difficulty.

Efficiency:Outputs: Reseal and rehabilitation kilometres, general maintenance expenditure;Input: Total expenditure

Effectiveness:Outcomes: Combined roughness for urban and rural highways, surface condition

index and level of service;Outputs: Reseal and rehabilitation kilometres, general maintenance

expenditure.

Economy:Outcomes: Combined roughness for urban and rural highways, surface condition

index and level of service;Input: Total expenditure.


Table 3.Three “Es” evaluation.

EFFIC’Y EFFECT ECONOMY EFFIC’Y EFFECT ECONOMYTLA01 100.0% 100.0% 100.0% TLA35 100.0% 89.9% 78.4%TLA02 100.0% 100.0% 100.0% TLA36 95.5% 30.4% 36.9%TLA04 74.6% 100.0% 100.0% TLA37 64.9% 51.1% 51.3%TLA05 100.0% 15.8% 19.1% TLA39 100.0% 100.0% 34.8%TLA06 100.0% 100.0% 100.0% TLA40 99.1% 40.7% 36.6%TLA07 55.2% 55.9% 46.0% TLA41 100.0% 100.0% 100.0%TLA09 100.0% 100.0% 100.0% TLA42 100.0% 100.0% 100.0%TLA10 86.1% 100.0% 100.0% TLA43 81.9% 66.9% 62.1%TLA11 49.5% 77.8% 61.6% TLA45 76.8% 91.7% 74.5%TLA12 100.0% 100.0% 100.0% TLA46 100.0% 100.0% 100.0%TLA13 100.0% 100.0% 100.0% TLA47 100.0% 100.0% 100.0%TLA14 100.0% 59.5% 41.8% TLA48 100.0% 100.0% 82.2%TLA15 65.9% 44.6% 39.7% TLA49 65.7% 36.5% 28.8%TLA16 88.9% 51.7% 33.8% TLA51 96.9% 10.3% 13.0%TLA17 92.7% 45.1% 42.5% TLA56 71.3% 89.3% 80.9%TLA19 75.6% 100.0% 100.0% TLA57 96.6% 31.9% 26.6%TLA20 100.0% 100.0% 100.0% TLA60 87.5% 20.1% 20.9%TLA21 100.0% 100.0% 100.0% TLA61 100.0% 100.0% 47.8%TLA23 78.2% 38.3% 34.5% TLA64 83.8% 56.9% 51.7%TLA24 62.2% 100.0% 100.0% TLA67 87.9% 25.9% 16.9%TLA26 88.7% 100.0% 100.0% TLA68 100.0% 15.6% 15.8%TLA28 91.4% 31.4% 34.1% TLA69 78.7% 100.0% 100.0%TLA29 100.0% 40.5% 40.7% TLA71 90.1% 39.8% 24.0%TLA30 85.8% 100.0% 100.0% TLA72 63.1% 85.3% 59.3%TLA32 100.0% 43.6% 31.7% TLA73 100.0% 19.5% 18.4%TLA33 100.0% 26.7% 22.1%TLA34 95.9% 60.5% 58.5%

Table 4.Summary statistics for three way evaluation.

EFFIC’Y EFFECT ECONOMYMean 89.0% 69.1% 62.8%Median 96.3% 81.6% 58.9%Standard Deviation 0.1403 0.3197 0.3244Sample Variance 0.0197 0.1022 0.1053Kurtosis 0.3666 −1.5140 −1.6638Skewness −1.1798 −0.3631 0.0008Range 50.5% 89.7% 87.0%Minimum 49.5% 10.3% 13.0%Maximum 100.0% 100.0% 100.0%Number of 100% 23 22 19


A major problem is the lack of information on specific outputs under the general mainte-nance category. In the analyses reported in Table 2, the surface condition index was used asa surrogate for general maintenance outputs. This index is more in the nature of an outcome.For the efficiency category, it is necessary to fall back on the use of general maintenanceexpenditure as the only available indicator of this output. This has the obvious problemof inadequately reflecting outputs actually provided but is the only option given currentreporting systems of TLAs.

It is immediately apparent from Tables 3 and 4 that whereas efficiency is high with overhalf the TLAs achieving scores of almost 100% and the lowest score just under 50%, effec-tiveness and economy are substantially lower with approximately one-third and just undera half of the TLAs reporting scores of less than 50% for each measure respectively. Graphsof the relationships between efficiency and effectiveness, and efficiency and economy inFigures 5 and 6 reveal low variability on efficiency but high variability on effectiveness andeconomy.

There are several possible reasons for the poor performance on the effectiveness and econ-omy measures. First, as the measures of roughness and surface condition only refer to sealedhighways at present, they may not be an appropriate reflection of these outcomes for TLAswith large sections of unsealed highway.7. Second, at the time of the survey of the TLAs,some were using an older version of the database management software which provides dif-ferent results than newer versions used by other TLAs. Third, costings used by each TLA re-flect local market conditions which affect the surface index calculation and total expenditure8

Fourth, it is possible that large areas of NZ highways are over-maintained in relation to theiruse value. Further research is needed to determine explanations for these results.

5. Incorporating Environmental Factors into DEA

As discussed in section 4.1, the use of categorical variables can over-estimate efficiencies forTLAs with low values for this variable. An alternative approach is to separate measures ofthe operating environment from measures directly under the control of TLAs, and to analysethe impact of the environmental factors on performance measured using the controllablevariables.9

The approach in this paper follows Fried et al. (1995) in which the slacks from an initialDEA run with only controllable variables are regressed against the environmental factors.The predicted slacks from the regression model are used to adjust inputs and outputs whichare subsequently incorporated into a second DEA run.

This approach is favoured as it directly addresses the differences in inputs and outputsamong TLAs resulting from varying environmental circumstances. The measures usedwere:

Outputs: Reseal and rehabilitation kilometres, general maintenance expenditure;Input: Total expenditure

Environmental variables: Environmental difficulty, vehicle kilometres, the ratio ofurban to rural kilometres of highway.


∗Unclear TLAs can be identified by reference to Table 3.

Figure 5. Efficiency and Effectiveness.∗

The outputs and input are the same as the efficiency model in section 4.2 except envi-ronmental difficulty is now treated separately as an environmental variable. Reference toTable 1 shows that vehicle kilometres have a high rank correlation with urban length. Theratio of urban length to rural length has therefore been used to represent the urban/rural mixand avoid possible collinearity problems in the regression analyses.10

The DEA results from the first run using variable returns to scale for the above measuresappear in column 1 of Table 6. As the slacks from this basic model were insignificantfor outputs, only slacks for the single input, total expenditure, were regressed against theenvironmental variables. The results for the regression with four outliers removed arereported in Table 5. The regression results are strong with significant positive coefficients


∗Unclear TLAs can be identified by reference to Table 3.

Figure 6. Efficiency and Economy.∗

for vehicle kilometres and urban/rural mix which is in line with expectations. The lackof significance for environmental difficulty is disappointing as our previous studies haveshown this to be a highly significant factor at the level of individual roads, and is probablydue to using too coarse a measure for this factor.

Predicted input slack for each DMU was used to adjust total expenditure and then rerunwith the same outputs as the basic case. TLA21 had negative total expenditure after thisadjustment which rendered it infeasible. The DEA results are reported in column 3 of Table 6with column 2 reporting comparative efficiencies from section 4.2 where environmentaldifficulty is treated as a categorical variable.

Generally, there is little change in average results between the basic and slack models


Table 5.Regression results.

R-square 0.699 N 48Adj R-Sq 0.679 F-Ratio 34.082 Prob 0.000

Coefficient Std Error T Prob (2tail)Constant 0.24 0.173 1.384 0.173Environ Diffuculty −0.001 0.034 −0.034 0.973Vehkm 0.006 0.002 2.846 0.007Urb/rural mix 0.175 0.031 5.572 0.000

(bearing in mind that TLA21 has been dropped from the slack model), but the variationhas increased. The categorical model reports higher results than either of the other twodue to the reasons outlined in section 4.1.11 Changes in individual TLAs are however morepronounced and for explanatory purposes, three TLAs, 4, 42 and 64 are examined in moredetail.

TLA4 is one of the largest cities in NZ and consequently has large vehicle kilometresand a high urban/rural mix. TLAs 42 and 64 are rural areas with low vehicle kilometres(TLA42 is in the lowest decile) but face more difficult environmental conditions than TLA4(ED measures of 3 and 5 respectively versus 6). Consequently, both rural TLAs benefitfrom the treatment of environmental difficulty (ED) as a categorical input within DEA.TLA4 has no benefit from the categorical model but gains when the predicted input slacksfrom the regression models are used to adjust the input. TLA4 has original slack of 2.6million dollars which is increased to $2.9 million due to its high vehicle kilometres andurban/rural mix. TLAs 42 and 64 have original slacks of 300 and 700 thousand dollarsrespectively which are reduced to 260 and 290 thousand respectively due to their lowvehicle kilometres and urban/rural mix. These adjustments make it possible for TLA4 toimprove its efficiency score markedly whereas TLAs 42 and 64 show a large reduction inefficiency scores.

Although a comparison between these two models, categorical and slacks, is not strictlyvalid because of the additional variables included in the slack model, it does highlight aneed for further research on comparative evaluations of alternative approaches.

6. Summary

This paper has developed a general managerial framework for performance measurementwith a strong accent on the need for strategic relevance of measures, the association ofmeasures with cost or process drivers in order to explain performance and illuminate areasrequiring action, and the important, complementary role for data analysis methods. The no-tion of a performance pyramid reflecting multiple perspectives on different faces is proposedto provide a more holistic view of the organisation and its environment. The performancepyramid by itself, however, is not sufficient. Systematic, consistent and objective methodsof data analysis are needed to ensure an appropriate balance of perspectives and measuresis maintained. DEA, with its location of optimal weights to suit individual circumstancesand capacity to restrict weight flexibility, is well suited to this task.


Table 6. Efficiency scores under basic case, environmental difficulty as a categorical variable and slackadjustments for environmental factors.

Basic Cat var Slacks Basic Cat var SlacksEnviron Adj DEA Environ Adj DEA

(1) (2) (3) (1) (2) (3)

TLA01 100.0% 100.00% 100.0% TLA34 89.5% 95.90% 76.1%TLA02 100.0% 100.00% 100.0% TLA35 100.0% 100.00% 100.0%TLA04 74.6% 74.60% 98.7% TLA36 89.7% 95.50% 90.4%TLA05 100.0% 100.00% 100.0% TLA37 61.0% 64.90% 60.3%TLA06 100.0% 100.00% 100.0% TLA39 100.0% 100.00% 100.0%TLA07 51.1% 55.20% 72.7% TLA40 84.6% 99.10% 83.2%TLA09 100.0% 100.00% 100.0% TLA41 94.4% 100.00% 94.1%TLA10 70.3% 86.10% 84.9% TLA42 75.0% 100.00% 34.7%TLA11 43.1% 49.50% 53.9% TLA43 74.8% 81.90% 67.4%TLA12 70.7% 100.00% 63.6% TLA45 72.3% 76.80% 64.9%TLA13 100.0% 100.00% 100.0% TLA46 84.9% 100.00% 100.0%TLA14 100.0% 100.00% 100.0% TLA47 95.8% 100.00% 70.2%TLA15 65.1% 65.90% 53.3% TLA48 100.0% 100.00% 87.7%TLA16 87.4% 88.90% 83.0% TLA49 65.7% 65.70% 89.3%TLA17 79.7% 92.70% 78.8% TLA51 96.9% 96.90% 96.4%TLA19 67.3% 75.60% 65.9% TLA56 71.3% 71.30% 90.7%TLA20 74.5% 100.00% 70.6% TLA57 96.6% 96.60% 87.7%TLA21 100.0% 100.00% Infeasible TLA60 76.9% 87.50% 73.0%TLA23 74.2% 78.20% 73.9% TLA61 100.0% 100.00% 100.0%TLA24 45.6% 62.20% 62.6% TLA64 71.1% 83.80% 48.7%TLA26 82.7% 88.70% 79.2% TLA67 87.9% 87.90% 86.3%TLA28 81.9% 91.40% 79.3% TLA68 100.0% 100.00% 100%TLA29 90.7% 100.00% 92.2% TLA69 59.9% 78.70% 41.9%TLA30 69.7% 85.80% 100.0% TLA71 87.0% 90.10% 68.9%TLA32 100.0% 100.00% 100.0% TLA72 58.3% 63.10% 43.6%TLA33 100.0% 100.00% 100.0% TLA73 100.0% 100.00% 100.0%

Summary Statistics:Basic Cat var Slacks

Environ Adj DEA(1) (2) (3)

Mean 83.1% 89.0% 81.7%Median 86.0% 96.3% 86.3%Standard Deviation 16.1% 14.0% 18.4%Kurtosis −0.470 0.367 −0.310Skewness −0.637 −1.180 −0.780Range 56.9% 50.5% 65.3%Minimum 43.1% 49.5% 34.7%Number of 100% 16 23 16Count 52 52 51

From a methodological perspective, this framework provides a robust base for DEAapplications as a means of validating the measures used and locating explanations for DEAresults. An additional consideration in the review of results is the “professional culture”dimension. In applications where a “professional culture” is known to prevail, structuralefficiency is expected to be high and variation in efficiency is expected to be low. Although


the DEA results for the highway maintenance application in this paper provide limitedsupport for the existence of this force, the usefulness of the notion lies more in determiningwhen the results are not meaningful due to inadequate or inappropriate measures.

The DEA application to highway maintenance of local authorities initially incorporatedmeasures of quantity and quality within a single model. Using the distinction betweenoutcomes and outputs, additional insights have been provided by partitioning the measuresinto a trichotomy reflecting efficiency, effectiveness and economy. Although the strongemphasis placed on the three “E’s” in the public sector has been criticised (Carter 1991),much of the criticism revolved around the difficulty of measurement of outcomes. In high-way maintenance, reasonable measures of outcomes pertaining to ride quality and surfacecondition are available and consequently, measures of the trichotomy can be provided formanagerial use.

Finally, an alternative approach has been described and explored for the treatment ofenvironmental factors. As these factors not only provide major explanations for performancevariability but also are major drivers of activities and resource usage, their identification,measurement and evaluation are of critical importance. Different approaches to measuringthe impact of environmental factors on efficiency provide very different results as shown insection 5. This is an important area and more research is needed to determine a preferredapproach.

Future research will focus on validating the results reported using the methods of valida-tion described, evaluating the appropriateness and effects of imposing bounds on variableweights, and investigating the treatment of environmental variables.

Acknowledgments

We would like to acknowledge the assistance of Transit New Zealand in the provision ofdata and technical advice in this study. We would also like to thank Knox Lovell for hishelpful comments and suggestions.

Notes

1. The problems of inappropriate weight selection can be addressed through a variety of methods such as coneratios, assurance regions and bounds on virtual weights. References to these are provided in Doyle and Green(1995).

2. The argument does not apply so well to notions of effectiveness and economy. In these cases, many morefactors beyond the control of the roading engineer may affect outcomes. For example, budgetary constraintsmay limit the nature and extent of maintenance outputs even though existing maintenance requirements arehigh; outputs may cost more than comparable TLAs or quality of work may be lower due to a lack of marketcompetitiveness for contractors in particular locations.

3. The “pyramid” concept is currently being amplified. Details may be obtained from the authors.

4. Among NZ TLAs, surveys of these measures are conducted annually by applying rating procedures over a oneto two month period.


5. The surface condition index is based on the amounts of general maintenance expenditure required to remedyall recorded defects observed in the roading network for a TLA, for a specified Benefit/Cost ratio (in this case5). The rating period when observations are carried out is usually over 1–2 months duration, and is performedevery 1–2 years. The index is a composite measure of surface condition comprising specific defects (e.g.rutting, flushing, scabbing), weighted by the cost of rectification.

6. For the VRS model (trial 8) 25 TLAs are designated as 100% efficient, 9 between 80–99%, 11 between 60–79%and 7 between 40–60%.

7. It is not possible to make any a priori statement of whether outcome measures for unsealed roads wouldimprove or worsen the results.

8. If these differences are substantial, then this is of interest to Transit NZ and to affected TLAs.

9. Approaches to the treatment of environmental factors in DEA include: direct inclusion as non-discretionary orcategorical variables (Banker and Morey 1986a and 1986b); regression of DEA efficiencies using controllablevariables on non-controllable variables (McCarty and Yaisawarng 1993, Lovell, Walters and Wood 1994).

10. The Pearson correlation between urban/ rural mix and vehicle kilometres is 0.40 (Spearman 0.58).

11. This is confirmed by a more exhaustive comparison of alternative approaches to the treatment of environmentalvariables in a forthcoming working paper by the authors.

References

Andersen, P. and N. C. A. Petersen. (1993). “A Procedure for Ranking Efficient Units in Data EnvelopmentAnalysis.” Management Science39, 1261–1264.

Banker, R. D., A. Charnes, and W. W. Cooper. (1984). “Some Models for Estimating Technical and ScaleInefficiencies in Data Envelopment Analysis.”Management Science30(9), 1078–1092.

Banker, R. D. and R. C. Morey. (1986a). “Efficiency Analysis for Exogenously Fixed Inputs and Outputs.”Operations Research34(4), 513–521.

Banker, R. D. and R. C. Morey. (1986b). “The Use of Categorical Variables in Data Envelopment Analysis.”Management Science32(12), 1613–1627.

Beischel, M. E. and K. R. Smith. (1991). “Linking the Shop Floor to the Top Floor.”Management AccountingOctober, 25–29.

Carter, N. (1991). “Learning to Measure Performance: The Use of Indicators in Organizations.”Public Admin-istration69, 85–101.

Charnes, A., W. W. Cooper, and E. Rhodes. (1978). “Measuring the Efficiency of Decision Making Units.”European Journal of Operations Research2(6), 429–444.

Cook, W., A. Kazakov, and Y. Roll. (1990). “A DEA Model for Measuring the Relative Efficiency of HighwayMaintenance Patrols.”INFOR. 28(2), 113–124.

Cook, W., A. Kazakov, and Y. Roll. (1994). “On the Measurement and Monitoring of Relative Efficiency ofHighway Maintenance Patrols.” In Abraham Charnes, William Cooper, Arie Y. Lewin, and Lawrence M. Seiford(eds.),Data Envelopment Analysis: Theory, Methodology and Applications. Kluwer Academic Publishers.

Cross, K. and R. Lynch. (1989). “Accounting for Competitive Performance.”Journal of Cost ManagementSpring, 20–28.

Deller, S. C. and C. H. Nelson. (1991). “Measuring the Economic Efficiency of Producing Rural Road Services.”American Journal of Agricultural Economics73(1), 194–201.

Doyle, J. R. and R. H. Green. (1995). “Cross-Evaluation in DEA: Improving Discrimination Among DMUs.”INFOR33(3), 205–222.

Epstein, M. K. and J. C. Henderson. (1989). “Data Envelopment Analysis for Managerial Control and Diagnosis.”Decision Sciences20(1), 90–119.

Fried, H. O., S. S. Schmidt, and S. Yaisawarng. (1995). “Incorporating the Operating Environment into a Measureof Technical Efficiency.” Working paper, Department of Economics, Union College.

Golany, B. and Y. Roll. (1989). “An Application Procedure for DEA.”OMEGA17(3), 237–250.Hansen, D. R., M. M. Mowen, and L. H. Hammer. (1992). “Profit-Linked Productivity Measurement.”Journal

of Management Accounting Research4, 79–98.Kaplan, R. S. and D. P. Norton. (1992). “The Balanced Scorecard—Measures that Drive Performance.”Harvard

Business ReviewJanuary–February, 71–79.


Lovell, C. A. K., L. C. Walters, and L. L. Wood. (1994). “Stratified Models of Education Production UsingModified DEA and Regression Analysis.” In Abraham Charnes, William Cooper, Arie Y. Lewin, and LawrenceM. Seiford (eds.),Data Envelopment Analysis: Theory, Methodology and Applications. Kluwer AcademicPublishers.

Maani, K. E., M. S. Putterill, and D. G. Sluti. (1994). “Empirical Analysis of Quality Improvement in Manufac-turing.” International Journal of Quality & Reliability Management11(7), 19–37.

McCarty, T. A. and S. Yaisawarng. (1993). “Technical Efficiency in New Jersey School Districts.” In HaroldO. Fried, C. A. Knox Lovell, and Shelton S. Schmidt (eds.),The Measurement of Productive Efficiency: Tech-niques and Applications. Oxford University Press.

Pryor, L. S. (1989). “Benchmarking—A Self Improvement Strategy.”Journal of Business Strategy10(6), 28–32.Public Finance Act (1989). New Zealand Government, Wellington.Putterill, M. S. (1985). “Developing and Intervention Strategy to Promote Enhanced Public Service Performance

by Agent Institutions.”Information and Management9(3), 161–187.Puttterill, M. S. (1987). “Information Systems for Road Maintenance Management—A Value for Money Ap-

proach.”Research in Governmental and Nonprofit Accounting3(Part A), 131–145.Putterill, M. S. and T. C. Rohrer. (1995). “A Causal Model of Employee Commitment in a Manufacturing Setting.”

International Journal of Manpower16(5/6), 56–69.Ramanathan, K. V. (1985). “A Proposed Framework for Designing Management Control Systems for Not-for-

Profit Organisations.”Financial Accountability and Management1(1), 75–92.Rouse, P. and M. Putterill. (1995). “Strategic Cost Management and Cost Drivers in Roading.” Working paper,

Department of Accounting and Finance, The University of Auckland.Senge, P. M. (1992).The Fifth Discipline. Sydney: Random House.Sexton, T. R., R. H. Silkman, and A. J. Hogan. (1986). “Data Envelopment Analysis: Critique and Extensions.”

In Richard H. Silkman (ed.),Measuring Efficiency: An Assessment of Data Envelopment Analysis. Jossey-BassInc.

Sherman, H. D. and G. Ladino. (1995). “Managing Bank Productivity Using Data Envelopment Analysis (DEA).”INTERFACES25(2), 60–73.

Smith, P. (1993). “Misspecification Bias in Data Envelopment Analysis.” Discussion Paper No. 93/15, Departmentof Economics and Related Studies, University of York.

Verma, K. (1992). “Total-Factor Productivity Measurement, Pertinent or Pass´e?” In W. J. Bruns (ed.),PerformanceMeasurement, Evaluation and Incentives. H.B.S. Series in Accounting and Control. Harvard Business School.

Documents

Towards a General Managerial Framework for Performance Measurement: A Comprehensive Highway Maintenance Application