Efficiency Measurement of Road Efficiency Measurement of Road Maintenance using Data Envelopment Maintenance using Data Envelopment

Analysis Analysis PRESENTED BY: KONSTANTINOS TRIANTIS

AUTHORS:MEHMET EGEMEN OZBEKJESUS M. DE LA GARZAKONSTANTINOS TRIANTIS

DATE:JUNE 28TH, 2007

TENTH EUROPEAN WORKSHOP ON EFFICIENCY AND PRODUCTIVITY ANALYSIS

Today…Today…

MOTIVATION FOR THE RESEARCH

PROBLEM STATEMENT

RESEARCH PURPOSE, OBJECTIVES, AND HYPOTHESES

DATA AND MODELING ISSUES

IMPLEMENTATION EXAMPLE

CONTRIBUTION OF THE RESEARCH

For the last two decades, the For the last two decades, the road maintenanceroad maintenance concept has been gaining tremendous concept has been gaining tremendous attention.attention.

Surveys by National Council on Public Works Improvement and American Society of Civil Engineers indicate a substantial deterioration in the road system from 1988 to 2005.

The Federal Highway Administration endorsed “asset management” to be the future approach of road maintenance for all state departments of transportation.

PERFORMANCE MEASUREMENT

Capital investment to build.

Maintenance to upkeep what is built.

Performance-based:

The risk of deficient design is imposed on the contractor.

The contractor often seeks innovative construction methods to make profit under fixed fee payment.

Asset management:

Comprehensive and well-structured approach to the long-term management of assets to provide effective

and efficient services. Calls for the utilization of

management, engineering, and economic principles in making decisions as to how resources should be allocated.

Virginia Department of Transportation (VDOT) has Virginia Department of Transportation (VDOT) has been one of the been one of the firstfirst state state DOTsDOTs to implement a to implement a pilot performance-based asset management pilot performance-based asset management contract.contract.

Earlier research has developed a Earlier research has developed a five-five-component frameworkcomponent framework to compare to compare effectivenesseffectiveness of the traditional maintenance of the traditional maintenance approaches against the performance-based approaches against the performance-based asset management approaches and asset management approaches and implemented it to evaluate the pilot project implemented it to evaluate the pilot project

(Piñero 2003, p. 47)

Quality of Service(Agency/UsersSatisfaction)

Timeliness ofResponse

Safety

Overall Performance

Cost

Level of Service

Current Road Maintenance Performance Measurement Systems

Not knowing how “efficient” state DOTs are in being “effective” can lead to excessive and unrealistic maintenance budget expectations.

For the cases in which comparative analyses are made, disregarding such external and uncontrollable factors and using pure effectiveness results may lead to unfair comparisons.

Solely focus on “effectiveness” measures, e.g., level-of-service.Disregard the “efficiency” concept,e.g., the amount of resources utilized to achieve such level-of-service.

Do not investigate the effect of the environmental factors, e.g., climate and location.Do not investigate the effect of the operational factors, e.g., traffic, load, design-construction adequacy.

The purpose of this research is, by using the Data Envelopment Analysis (DEA) approach, to develop and implement a generic, replicable, and comprehensive framework that can: Measure the overall efficiency of road maintenance operations. Consider the effects of external and uncontrollable factors on such efficiency.

The specific objectives of this research are, through the use of real data, to identify: 1) The relative efficiency of different units in performing road maintenance services.2) The reasons of the efficiency differences between units.3) The effects of the external and uncontrollable factors on the efficiency of units.4) The benchmarks (peers) and best practices that pertain to the inefficient units.

The hypotheses of this research are as follows:1) State DOTs are more efficient when they utilize the performance-based approach instead of the traditional method-based approach for the maintenance of the roads. 2) A significant portion of the observed efficiency differences between different units can be attributed to the effects of the environmental factors (e.g., climate, location, etc.) and operational factors (e.g., traffic, load, design-construction adequacy, etc.) faced by such units.

This presentation focuses on the This presentation focuses on the data and modeling issuesdata and modeling issues relevant to the relevant to the first first seven tasks seven tasks of the frameworkof the frameworkDefine the comprehensive list of Input-Output

Variables and Uncontrollable Factors

Address the issue of uncontrollable factors

Refine the comprehensive list of Input-Output Variables and Uncontrollable Factors

Perform data mining

Clean the data

Allocate the data to the DMUs

Decide on the size of the DMU and the time unit of analysis

Perform data conversion and data rearrangement

Choose the type of DEA models to be run

Run the DEA models and obtain the efficiency score, targets, and peer(s) for each DMU; and the overall efficient frontiers

Identify (i) the reasons of efficiency differences between the units and (ii) effects of uncontrollable factors on the efficiency of the units

Prepare the data to be used in the DEA models

Validate the results, identify best practices, and perform policy analysis

To define the input-output variables and uncontrollable To define the input-output variables and uncontrollable factors, factors, thethe process process that is under investigation should be that is under investigation should be studied. studied.

Preserving theCapital Investment

Maintenance ofHighways

Maintaining FreeFlow of Traffic

Preventive Maintenance -Planned, Cyclical

Restorative Maintenance -Minor Repairs

Rehabilitative Maintenance - Major Repairs and Replacements, Reconstruction

Types of maintenance performed to meet Level-of-Service requirements

Restorative Maintenance -Minor Repairs

Incident Response

Snow and Ice Removal

Types of maintenance performed to meet Timeliness of Response requirements

Preventive Maintenance -Planned, Cyclical

Works in the favor

OutputLevel-of-Service

(LOS)Works against

Preventive and Restorative Maintenance Deterioration

To define the input-output variables and uncontrollable To define the input-output variables and uncontrollable factors, factors, thethe process process that is under investigation should be that is under investigation should be studied. studied.

Consumes inputs Consumes outputs

Affected by uncontrollable factors

Inputs:Labor

MaterialEquipment

This research deals with This research deals with allall of the of the asset itemsasset items present within the present within the right-of-way fences. However, this presentation will focus only on the right-of-way fences. However, this presentation will focus only on the bridgesbridges..

Defining Input-Output Variables for the Maintenance of Defining Input-Output Variables for the Maintenance of BridgesBridgesInputs: Resources used by DMUs Outputs*: Benefits made as a

result of the process undertaken by the DMUs

*: Undesirable outputs are not considered in this research

Labor

Material

Equipment

Bridges’ Maintenance

Bridges Maintained

to meet LOS Requirements

Inputs OutputProcess

Input

Labor Hours

Quantity of Material

Equipment Hours

Bridges’ Maintenance

National Bridge Inventory Rating

Input Metrics to Establish Technical Efficiency

Output MetricProcess

Cost

Input Metric to Establish Cost Efficiency

METRICS

Once everything is considered, the list of input-output variables and Once everything is considered, the list of input-output variables and uncontrollable factors becomes uncontrollable factors becomes exhaustiveexhaustive..

The DMU is chosen to be the The DMU is chosen to be the county of Virginiacounty of Virginia and the time unit of and the time unit of analysis is chosen to be a analysis is chosen to be a fiscal yearfiscal year-- Both decisions are Both decisions are dictated by dictated by the cost data.the cost data.

1

2

3

4

5

8

7

6

The resulting number of DMUs The resulting number of DMUs dictateddictated the approach to be used to the approach to be used to deal with uncontrollable factors. deal with uncontrollable factors.

One Stage ApproachesUncontrollable Factors treated as Controllable Variables in the DEA Model

Uncontrollable Factors treated as Uncontrollable Variables in the DEA Model

Uncontrollable Factors used to Develop Categories of DMUs to be included in the DEA Models

Continuous Uncontrollable Factors used to Restrict the Peer Reference Set

Multi Stage ApproachesUncontrollable Factors used to Perform Regression Analysis over the Obtained Efficiency Scores

Parameters Obtained by the Regression Analysis used to Build an Overall Environmental Harshness Index

Uncontrollable Factors used to Perform Bootstrapped Regression Analysis over the Obtained Efficiency Scores

There is a number of approaches to There is a number of approaches to refine the comprehensive lists of refine the comprehensive lists of controllable variablescontrollable variables and and uncontrollable factorsuncontrollable factors..

Develop (or adjust) the list of DMUs to be compared

Develop the comprehensive list of variables

Refine the variable list using the Judgmental Process

Achieved a well refined list that can be used in the

DEA models?

Refine the variable list using the Quantitative Methods

Refine the variable list using the DEA Based Analyses

Achieved a well refined list that can be used in the

DEA models?

Achieved a well refined list that can be used in the

DEA models?

Formalize final DEA model using the list of variables obtained at this point

YES

YES

YES

NO

NO

NO

Ø Expert opinionsØ Analytic Hierarchy

Process (AHP)

Ø Aggregation MethodØ Rescaling MethodØ Principal Component Analysis (PCA)Ø Identification of Correlations

Ø Performing trial DEA runs with different combinations of variables

To To refine the comprehensive list of variablesrefine the comprehensive list of variables, , aggregation methodaggregation method was used. was used.

OutputsUndesirable outputs are out of the scope of this researchCritical output variables are Deck, Superstructure, Substructure, and Slope/Channel ProtectionNo cost data for Slope/Channel Protection is provided by VDOTDeck, Superstructure, and Substructure are combined into one variable by using the weighting structure developed and implemented by Virginia Tech as agreed by VDOT

InputsUsing the research by Dadson (2001), 16 input variables (Variable 2- Variable 4 and Variable 6-Variable 18) can be combined into a single input variable that represents all of those to a great extent

Dadson identified the Dadson identified the effects of theeffects of the VA Environmental Regions VA Environmental Regions on the on the deterioration of the bridges.deterioration of the bridges.

TW: TidewaterEP: Eastern PiedmontWP: Western PiedmontN: NorthernCM: Central MountainSM: Southwestern Mountain

As a result of the refinements, As a result of the refinements, total number of variablestotal number of variables (controllable (controllable and uncontrollable) reduced and uncontrollable) reduced from 26 to 4from 26 to 4..

Before showing the Before showing the implementation exampleimplementation example of the developed of the developed framework for bridges, it is essential to discuss the framework for bridges, it is essential to discuss the datadata..

Level-of-Service DataProvided by Federal Highway Administration.Federally-mandated program.Used to establish investment requirements, to develop data summaries at the national level for reports to Congress, and to respond to inquiries from entities.

Cost Data

Provided by VDOT as obtained through Financial Management System II.Per fiscal year, per county.Includes only preventive maintenance and restorative maintenance expenditures. No rehabilitative maintenance expenditure is included.No district or central office overhead is included.

Data miningData mining and and data cleaningdata cleaning were performed. were performed.

The data was needed to be The data was needed to be convertedconverted into the format suitable to into the format suitable to represent the variables. Moreover some represent the variables. Moreover some rearrangementsrearrangements had to be had to be made in the data to make it meet the made in the data to make it meet the structuring requirementsstructuring requirements of the of the DEA model.DEA model.

A number of A number of final refinementsfinal refinements accompanied the accompanied the DEA model selectionDEA model selection and and runrun..

BCC Model is selected to be utilized as it is believed that the bridge maintenance process is more likely to experience variable returns to scale.

Change in Overall Bridge Condition

Total Area Served

Regional Effect Variable

Cost

Modified Change in Overall Bridge Condition

DMU TAS

DMU REGIONAL EFFECT

Cost adjusted for overhead and inflation

Frontier AnalystFrontier Analyst was used to run the chosen DEA model and to was used to run the chosen DEA model and to obtain the results.obtain the results.

Conclusion 1: All counties are efficient relative to each other. Conclusion 2: The model, as structured and run above, is not able to discriminate between the DMUs and thus overlooks the relative efficiency differences (inherent within the DMUs) that exist in reality.

To test the validity of the alternative To test the validity of the alternative conclusions, conclusions, further analysesfurther analyses were were performed.performed.

Reducing the Number of Variables using DEA Based Analyses

Increasing the Number of DMUs

Increasing the number of DMUsIncreasing the number of DMUs changed changed the the efficiency scores to a great extent.efficiency scores to a great extent.

Fauquier

Albemarle

Augusta

Alleghany

Roanoke

Henrico

Spotsylvania

Rockbridge

Given the fact that the value of the cost variable for the DMUs Given the fact that the value of the cost variable for the DMUs belonging to Roanoke County is extremely low, further analysis (this belonging to Roanoke County is extremely low, further analysis (this time time excluding the DMUs belonging to the Roanoke Countyexcluding the DMUs belonging to the Roanoke County) is needed ) is needed to verify that Spotsylvania County is indeed extremely inefficient to verify that Spotsylvania County is indeed extremely inefficient relative to other counties in the data set. relative to other counties in the data set.

Removing the Roanoke County from the analysis Removing the Roanoke County from the analysis affected affected onlyonly the efficiency score of the the efficiency score of the Spotsylvania County.Spotsylvania County.

The results of the models need to be The results of the models need to be validatedvalidated and and verifiedverified by the decision makers at VDOT before by the decision makers at VDOT before actually taking any of the actions suggested by the actually taking any of the actions suggested by the model.model.

DEA does not directly pinpoint the underlying causes of inefficiencies of DMUs.

The findings of a DEA study are intended to be used as guides for managerial actions and policymaking as calculated targets for inputs and outputs indicate potential performance and efficiency increases for inefficient DMUs.

Contribution to the Contribution to the Highway MaintenanceHighway Maintenance literature: literature: A framework that is able to differentiate effective and

efficient maintenance strategies from effective and inefficient ones.

A A frameworkframework that is able to take into account: that is able to take into account: multiplemultiple views (e.g., the driver’s versus the maintenance views (e.g., the driver’s versus the maintenance

provider’s)provider’s) important important societal goalssocietal goals (e.g., safety) (e.g., safety) uncontrollableuncontrollable considerations (e.g., climate, location, etc.) considerations (e.g., climate, location, etc.)

Contribution to the Contribution to the DEA / Measurement ScienceDEA / Measurement Science literature: literature: Application of DEA within the Application of DEA within the engineeringengineering discipline discipline Illustration ofIllustration of

issues associated with issues associated with definingdefining andand refining the listrefining the list ofof input-output variables input-output variables andand uncontrollable factors uncontrollable factors

data availabilitydata availability issues issues modeling issues associated with modeling issues associated with definingdefining the appropriate the appropriate

decision making unitdecision making unit the potential the potential impactimpact that efficiency analysis can have on that efficiency analysis can have on

road maintenance improvementroad maintenance improvement

Proposed research’s Proposed research’s contribution to BOKcontribution to BOK is is twofold:twofold: