Evaluating public transport efficiency with neural network models

Pergamon

PII: S0968-090X(97)00017-X

Transpn Res.-C, Vol. 5, No. 5, pp. 301-312, 1997 © 1997 Elsevier Science Ltd

All rights reserved. Printed in Great Britain 0968-090X/97 $17.00 + 0.00

E V A L U A T I N G P U B L I C T R A N S P O R T E F F I C I E N C Y W I T H N E U R A L

N E T W O R K M O D E L S

/kLVARO COSTA Faculdade de Engenharia da Universidade do Porto, Rua dos Bragas, 4099 Porto Codex, Portugal

and

R A P H A E L N. M A R K E L L O S * Department of Economics, Loughborough University, Loughborough, U.K., LEI 1 3TU

(Received 4 October 1996; in revised form 29 July 1997)

Abstract--This paper is concerned with measuring performance of public transport services based on the concept of productive efficiency. A new nonparametric approach is proposed based on multi-layer perceptron neural networks (MLPs). The advantages and limitations of this approach are discussed and compared with those of mathematical programming and econometric techniques. The MLP is used, along with data envelopment analysis (DEA) and corrected least squares (COLS), to set out comparative annual efficiency measures for the London Underground, for the period 1970 to 1994. It is argued that the MLP approach is superior to traditionally applied techniques since it is both nonparametric and stochastic and offers greater flexibility. Finally, it is demonstrated that the proposed MLP efficiency analysis has important practical implications for decision making. © 1997 Elsevier Science Ltd. All rights reserved

Keywords: efficiency analysis, multilayer perceptron neural networks, London Underground.

1. INTRODUCTION

The productivity of an economic unit is typically defined as the ratio of its output to its input and is a function of many factors such as technology, environment, efficiency, etc. One of the most widely used measures of productivity performance is efficiency which compares realised and optimal levels of outputs and inputs. I f this comparison is made in terms of production possibilities, then 'technical' efficiency is measured while a comparison in terms of a behavioural goal (e.g. cost, revenue, profit) measures 'economic' efficiency.

The selection of a specific efficiency measure depends much on the nature of the organisation under study: For example, while for most economic units profit maximisation is a natural objective, this is not necessarily the case for public transport services. Because public transport opera- tion is generally managed as a public service, in spite of the presence of a traded output, profitability cannot be used as the sole measure of success (Fielding et al., 1985). Furthermore, public providers are subject to objectives and constraints (e.g. equity and fiscal balance consid- erations) different from those of private providers. In deciding on the use of a specific efficiency measure one must also consider the peculiarities the problem, the underlying research objectives and the availability of relevant data. In this paper we concentrate on 'technical' efficiency and refer to it simply as efficiency. The techniques proposed can just as easily be used to measure other types of efficiency in public or private organisations.

Measuring public t ransport efficiency is very important since it allows us to evaluate and com- pare the success and potential of individual operators. In view of the recently realised or proposed extensive changes towards increased deregulation, reorganisation and privatisation of public transport, efficiency gains are perhaps the most important yardstick of evaluation. Efficiency analysis is also important in terms of identifying and measuring sources of successful performance and

*Author for correspondence: e-mail: [email protected]

TRtO 5-s-c 301

302 .~lvaro Costa and Raphael N. Markellos

therefore can be used in policy planning and allocation of resources. For example, if for some production unit a comparatively low 'technical' efficiency is attributed to the workforce, then this information can be used in decisions related to training, reorganisation of production, human resources management, etc. Finally, efficiency analysis can be used in verifying and quantifying theoretical propositions and hypotheses. For example, a researcher may want to examine the effect of privatisation on the 'technical' and 'cost' efficiency of transport operators (a more detailed treatment of efficiency analysis objectives can be found in Lovell, 1993).

The basic techniques that have been used for measuring efficiency can be categorised into econometric and mathematical programming approaches (for a description of efficiency measurement techniques and applications see Fried et al., 1993). The former include regression-based techniques and are stochastic in that they attempt to distinguish noise from inefficiency. Noise is used to represent events that occur randomly (these may influence efficiency but their occurrence cannot be controlled, e.g. weather) and variables that are not explicitly included in the efficiency measurement model. Most of the mathematical programming approaches are nonstochastic but unlike the former do not make strict assumptions about the functional form of production and the statistical properties of the data. Lovell (1993, p. 19) expresses some of the concerns in the efficiency analysis literature by arguing that: 'Obviously it would be desirable to make the programming approach stochastic, and to make the econometric approach more flexible in its parametric structure. Limited progress has been achieved in both areas'. The present paper offers an alterna- tive to econometric and mathematical programming approaches, it proposes the use of multi-layer perceptron neural networks (MLPs) in measuring productive efficiency (see Dougherty, 1995 for other applications of neural networks in transportation). MLPs are nonparametric models whose estimation is based on stochastic nonlinear mathematical programming techniques. The proposed approach makes no assumptions about the statistical properties of the data and the functional form of the underlying efficiency model (i.e. the function of cost, production, etc.). Additionally, it is not necessarily deterministic and allows the use of stochastic components in the analysis of efficiency. Finally, unlike mathematical programming techniques it is not a completely 'black-box' approach and offers possibilities for post-processing and interpreting the estimated efficiency model.

The material is organised as following: in the next section the econometric, mathematical programming and neural network methodologies of efficiency measurement are described and compared. Section 3 demonstrates an application of the proposed MLP approach in estimating annual 'technical' efficiency measures for the London Underground from 1970 to 1994. The results are also compared with those obtained by an econometric and a mathematical programming technique. Section 4 interprets the results and discusses their practical significance for decision makers. The final section concludes the paper and offers routes for future research.

2. EFFICIENCY MEASUREMENT METHODOLOGY

Assuming that a transport operator uses inputs xeT"Z~+ to produce outputs yeR+ then a generic form of the production can be specified as:

Yi = f(xi; r) + ei -F ui (1)

where fl is the vector of technology parameters that must be estimated and i = 1 ..... I indexes production units (over time for one unit and/or over different organisations). The term ei is an l iD disturbance designated to account for statistical noise, i.e. measurement errors, climatic effects, etc. u; represents technical inefficiency and is constrained to negative or zero values that are assumed to be distributed independently of e;. The deterministic frontier is given by f(xi; ~) and the stochastic frontier by f(xi;/~) + e;. Although the above representation refers to the single output case it can easily be extended for multiple outputs. Standard inputs used in analysing public transit production functions include number of workers and vehicles and network length. Outputs can be measured by variables such as production levels (e.g. in kms covered by vehicles, kgrs of freight) and number of consumers (a more detailed description of input and output variables can be found in Gathon, 1989).

Evaluating public transport efficiency 303

2.1. Econometric modelling For many years, most empirical research in efficiency measurement has concentrated on least

squares-based statistical methodologies. These parametric approaches make strict theoretical assumptions such as: existence of Cobb-Douglas technology, constant scale elasticity and unitary substitution elasticity (a description of these concepts can be found in Ostaszewski, 1993). Fur- thermore, all the statistical suppositions related with least square techniques must be accepted. If it is assumed that in (1) technical inefficiency u,. is zero then the problem simplifies to estimating the parameters of a stochastic production function via ordinary least squares (OLS), while if it is assumed that ei is zero the problem is reduced to estimating the parameters of a deterministic frontier with no noise. Three basic techniques can be applied in measuring the parameters of a deterministic frontier: corrected ordinary least squares (COLS), modified or displaced ordinary least squares (MOLS) and maximum likelihood (MLE) (see Fried et al., 1993). COLS estimates the parameters/3 in (1) by OLS and then corrects the downward bias in the estimated intercept by shifting it upwards to the point that all corrected residuals are nonpositive and at least one is zero. MOLS assumes a specific functional form for technical inefficiency and corrects the OLS intercept by shifting it by minus the estimated mean of ui. The OLS residuals are then modified in the opposite direction and used to calculate efficiency. MLE also assumes a specific functional form for technical inefficiency, and simultaneously estimates the technology parameters and the parameters of the distribution of ui. For modelling a stochastic frontier only MOLS and MLE can be applied. In this case, the error of the regression is 'decomposed' into statistical noise and technical inefficiency by applying a technique proposed by Jondrow et al. (1982). This decomposition requires making an assumption about the distribution of ui (e.g. half normality). Either the mean or the mode of this distribution gives a point estimate of the one-sided inefficiency component which can then be inserted in (1) to estimate technical efficiency (see Lovell, 1993 for a brief description of error decomposition techniques). The efficiency measurement methodology in the case of a single or of multiple outputs, m = 1 and m > 1, respectively, are very similar although very few empirical studies estimate a multi-output production frontier.

2.2. Mathematical programming The mathematical programming approaches in production frontier analysis include a wide

range of nonparametric techniques which are largely nonstochastic, such as data envelopment analysis (DEA), free disposal hull (FDH), goal programming, etc. (see Fried et al., 1993). In this paper we consider only DEA since it is the most commonly used mathematical programming method in efficiency analysis. A brief description of DEA is given below; a more thorough description can be found in Weyman-Jones (1991).

The standard DEA makes some assumptions about constant returns to scale, the strong dispo- sability of inputs and outputs, and convexity of the set of feasible input/output combinations (a description of these concepts can be found in Ostaszewski, 1993). Although all of these assumptions can be relaxed, in practice this is rarely done for the last two (Lovell, 1993). DEA assumes a deterministic frontier, although it is possible to modify it for the stochastic case (a version of stochastic DEA has been proposed by Land, Lovell and Thore and is described in Lovell, 1993). DEA provides relative measures of efficiency and is increasingly being used in evaluating the performance of public service industries (for an overview see Ganley and Cubbin, 1992). The efficiency measures are distances to an empirical production frontier and the values are calculated on the basis of standard Pareto efficiency. No assumption has to be made about the production function because the frontier is the observed best practice of the raw data set available. The inefficiency terms in (1) are calculated by solving the following 'envelopment' problem via linear programming:

max(w, y) (2)

subject to

n n

W" Yi < ~-a ks. Xsi and ~ ks. xs = 1 with ks, w > 0, s = l s = l

s = 1 , . . . ,n , i = 1 , . . . , I

304 ,~lvaro Costa and Raphael N. Markellos

where w,k s are weights, Xsi is the matrix of inputs, y; is the vector of outputs and n is the number of inputs. Intuitively, this measures the efficiency of a production unit i by finding a set of weights to apply to its inputs and outputs. The above representation is only one of many existing formu- lations of the 'envelopment' problem, it gives an input-minimisation version of DEA and assumes agents wishing to minimise the inputs required for any given level of output. The problem can easily converted to the output maximisation or multiple output case.

2.3. Multi-layer perceptron neural networks Artificial neural networks (ANNs) form a class of nonparametric models able to acquire

knowledge under noise and uncertainty, to perform generalisation and abstraction and to create their own knowledge by self organisation (a general treatment of ANNs can be found in Fu, 1994). The relationships in these models are implicitly generated and are ideally sufficiently general to interpolate accurately in high-dimensional input/output spaces. Although ANNs have been applied in many areas of transportation (for a review see Dougherty, 1995) their potential in estimating production functions and measuring performance has not yet been discussed.

The MLP is perhaps the most popular and widely applied of the many existing ANN types. Hornik et al. (1990) have shown that subject to mild regularity conditions these models can approximate any function and its derivatives to any degree of accuracy.

The MLP basic properties can be summarised in the triplet: multi-layer, feed-forward and supervised neural network. Its processing elements, known as 'neurons', are organised in at least three layers: the input layer, the output layer and the hidden layer(s) in between. These neurons are all fully connected between adjacent layers. MLPs are feed-forward networks, i.e. all connections point in one direction, from the input towards the output layer. Finally, they are supervised networks since all patterns of both inputs and outputs must be provided. The development of a neural network model requires the specification of a 'network topology' and a 'training strategy'.

The MLP topology is defined by the number of input and output neurons, the number of hidden layers and the number of neurons in each hidden layer. The training strategy refers to the procedure and technique used in estimating the optimal network parameters (known as 'weights'). This optimisation is done by repeatedly adjusting an initial set of random weights so as to minimise some cost function. The most popular and successful method of estimation is a stochastic approximation based technique called 'backpropagation', where the error propagates backwards from the output layer to the hidden layer(s), until it reaches the input layer.

The basic concerns when constructing a MLP model is to determine the optimal network topology and duration of training. This is because an over-parameterised and/or overtrained model may overfit the data and loose its ability to generalise over unknown data. While the optimal network strategy and duration of training can be determined in several different ways, in this study we discuss two of the basic approaches: the 'cross-validation' method and the 'gradient- norm' criterion method.

Cross-validation requires the use of two different data samples: a training set and a cross-validation set. The training set is used to estimate the neural network weights by backpropagation. As the network is optimised in the training set its performance is monitored in the cross-validation set. This performance is measured in terms of some error function, typically the root mean square error (RMSE). The optimisation of each model is halted when the performance of the model, in terms of the error function, in the cross-validation sample stabilises or starts to deteriorate.

The 'gradient-norm' criterion method determines the optimal level of training by examining the magnitude and stability of the gradient-norm controlling the network optimisation process. If this parameter is very small and has stabilised then the benefits of further optimisation are regarded as small and training is terminated. This approach has the advantage of not resorting to data outside the training sample, something very important when the available data is limited. On the other hand the a priori determination of the optimal magnitude and derivative of the gradient-norm can be a very tedious task. Usually, the best results are obtained by combining both approaches. When the network is to be used for predictions then the out-of-sample performance of a model must be evaluated on a test set that has not been used for training or cross-validation.

The optimal network topology is selected by trial-and-error on grounds of parsimony, in order to avoid models with more neurons and hidden layers than required. An explicit formulation of the principle of parsimony is given by the so-called Information criteria, that penalise


over-parameterised model configurations. In this study we use the Schwartz information criterion, also known as SIC (Schwartz, 1978):

SIC(M) = 1. log(MSE) + M . log(l) (3)

where M is the number of model parameters and I is the sample size. Rather than calculating the mean square error (MSE) of the optimal MLP directly, more precise estimates can be obtained through the 'Bootstrapping' method (see Efron and Tibshirani, 1993).

Many efficiency measurement problems in transportation are characterised by the limited availability of data in terms of sample size. Furthermore, the nature of the analysis does not allow us to exclude part of the data in estimating the production frontier. These two characteristics essentially prohibit the direct application of cross-validation techniques. One approach for over- coming these problems is to isolate one vector of data, use it for cross-validation and then repeat the procedure until all data vectors take their place in the single cross-validation vector. An alter- native far less complicated approach is to use a synthetic sample for training and the original sample for cross-validation. While several methods exist for the creation of synthetic data (e.g. the Waterloo simulations suggested by Hipel and McLeod, 1993) in this paper we adopt a simple method that involves adding a predetermined amount of statistical noise, usually distributed as N(0, tr2), to the data. This addition is repeated several times until an arbitrary large sample of synthetic data is available for training.

Using the MLP function representation, the production (1) can be re-specified as:

q

Yi = ai + E ydeP(fldXi) + ei + ui (4) d=l

where o(z) is typically a bounded, monotonic function, the so-called 'squashing function', a;, Ya and/~d are the MLP free parameters to be estimated and q is the number of nonlinear terms. The number of model parameters is given by M = n(q + 1). The MLP frontier and efficiency measures can be calculated in one of the following ways:

• After the optimal form of the network is estimated and a deterministic or stochastic frontier is assumed, efficiency can be calculated as when using the COLS or MOLS methods described above. If training on synthetic data is selected, then an approach very similar to that of stochastic DEA can be followed. This requires making the assumption that the inputs and outputs are (normally distributed) random variables with specific expected values and variance/covariance matrices. These assumptions are then utilised in the creation of the synthetic data and the MLP is estimated via the cross-validation or gradient-norm method.

• Alternatively, the frontier can be modelled by using an 'oversized' network to overfit the original data. Since such an over-parameterised MLP can eventually fit the training sample with zero error, training is halted at the point when the signal-to-noise ratio reaches a level that satisfies some a priori assumption about the level of statistical noise ei in the data. A signal-to-noise ratio can be calculated as (Deboeck and Cader, 1994):

s : n = l O . l n "I j=l (5)

where 5i are estimates from the model and a signal-to-noise ratio of 10 implies that the signal is ten times more evident than noise in the data. It must also be assumed that at that point the model has overfitted the technical inefficiency and not the noise. If the model is left to completely overfit (envelop) the sample then a solution very similar to that of DEA should emerge.

Since the first approach is related more to the econometric methodology and the second to the DEA, we will refer to them hereafter as econometric-MLP (ECN-MLP) and envelopment-MLP (ENV-MLP) approaches, respectively.

306 A, lvaro Costa and Raphael N. Markellos

2.4. Selecting an efficiency measurement technique This section considers the main factors governing the choice of a particular efficiency measure-

ment approach for a given problem. A rough comparison of approaches based on specific factors is given in Table 1. It is evident that no single approach appears to be overall superior and that the selection must depend on the specific characteristics of the problem underhand, a discussion of this follows below.

The major advantages of DEA and MLPs over econometric techniques is that they make mini- mal assumptions about the data and the form of production. While DEA and MLPs are mathematical programming techniques their estimation is very different. The backpropagation method used in the estimation of MLPs is a stochastic non-linear programming technique and the resulting model has strong statistical analogies (e.g. Smith, 1993 and Ripley, 1993), it can be broadly considered as a nonlinear regression model. The MLP approach is more flexible than DEA and Econometric techniques in that it can be applied to a wide selection of problems: it makes no theoretical and statistical assumptions and it can be used in modelling both deterministic and stochastic frontiers, econometric and DEA techniques, unlike MLPs, have the advantage of a well developed theoretical basis and they have been widely applied in efficiency analysis. The major disadvantage of DEA and MLP models is that the results cannot be assessed on the basis of statistical significance. Unlike DEA, MLPs offer the possibility to interpret the results and the estimated production function (e.g. interpretation of the technology parameters, the form of the production function, the contribution of each production input, etc.). Furthermore, MLPs do not necessarily overfit the data studied and therefor they can be used for analysing data and decisions outside the estimation sample. Econometric techniques are advantageous in terms of interpretability only if constant returns to scale production (i.e. a linear production function) can be assumed. Finally, one should also consider that MLPs are at present relatively costly in terms of specialised software and time (pre-processing data, estimation and post-processing) required for the analysis.

Figure 1 gives a graphical depiction of the general forms of production frontiers constructed by different methods. All econometric techniques, such as COLS, result linear frontiers and assume production with constant returns to scale. Although DEA can accommodate variable returns to scale, by local linear approximations, it cannot provide information on the technology parameters and slope of the production function. The MLP methodology is the only purely non-linear technique and can represent any form of the production function, even in a congested are, i.e. a production where a decrease in inputs increases output, this can be graphically represented by a downward slope production curve. Furthermore, the MLP method can be used to 'explore' the possibilities and behaviour of a production curve and to asses the conditional technology parameters, i.e. the technology parameters that are conditional to certain production possibilities.

In general, a straightforward econometric approach, such as the COLS, should always be applied initially as a portmanteau statistic. If the assumptions required by econometric techniques are likely to be invalid, then a nonparametric technique should preferred. When a problem involves measuring efficiency over time (time series or panel data), econometric techniques usually assume that the variables of production are nonstationary and that they move along some deterministic linear time trend. In order to transform to stationarity, they adjust the logarithmic levels of the variables for a linear trend. This is a serious drawback for econometric techniques, since their apparent stochastic nature contradicts their assumptions regarding the existence of deterministic trends. In the case of MLP and DEA, the problem of nonstationarity is less important, and therefore these approaches are more appropriate for analysing efficiency over time. In problems

Table 1. Comparison of efficiency measurement approaches

Comparative factor Econometric DEA MLP

Statistical and functional form assumptions Strong Modest None Flexibility Low Modest High Theoretical basis Strong Strong Weak Statistical significance Yes None None Interpretability of results Modest Low High Projcction/generalisation High None Modest Cost Low Low High


Output

l • f , s "

/'""" ~I ~ p Input Fig. 1. Production frontiers using econometric, DEA and MLP models.

where information is needed on the form of the production function and the potential success of various production scenarios, then the only available option is to use an MLP model. When the analysis must be projected to make out-of-sample decisions, then an econometric or the ECN- MLP approach should be preferred. For in sample normative comparisons a mathematical programming or the ENV-MLP approach should be chosen. This is because these approaches, especially DEA/VRS, are sensitive to details of the data under analysis and are more likely than other approaches to overfit the data. Although this can be useful for explaining and interpreting the data, it limits the generality of the results and our ability to make numeric projections of the parameters, in the future.

Although, the MLP approach offers many possibilities, it is very complicated and there is no theoretical guarantee that it will result an optimum solution, in the sense of BLUES/maximum likelihood estimators or mathematical programming optima. We believe that rather than searching for a strictly optimum production frontier the objective should be to obtain a model that is 'likely' in the Bayesian sense. If a rather complex approach (such as MLPs) has a greater posterior prob- ability than a simpler one, despite a prior beliefs favouring simpler approaches, then it should be selected. This approach, also favours the combination of different approaches in efficiency measurement problems.

3. EMPIRICAL RESULTS

Annual historical data for the publicly operated London Underground, from 1970 to 1994, were used for performing efficiency analysis. This time series was selected since it is characterised by two of the main properties that make the use of econometric efficiency measurement techniques pro- blematic: nonstationarity and multicolinearity.

The number of trains and workers were selected as inputs of production and the total train km distance per year covered by the London Underground fleet was considered as the single output. Although this formulation of the production process serves the purposes of the present analysis, it is by no means intended to be complete. As argued by Costa (1996), the success of public transport production should not only consider the raw output produced but also the utilisation or 'effectiveness' of product offered. Since no significant improvement was made to the London Under- ground production technology during the period under study, the impact of embodied technical progress is not considered in this application, all improvements are classified as efficiency changes.

A deterministic frontier was estimated via COLS* on the trend-adjusted logarithms of the data. Although COLS has been heavily criticised in the literature we apply it on grounds of simplicity and that it usually yields similar results to other econometric efficiency measurement techniques. The MLPs* were optimised by using a modification of the standard back-propagation algorithm

*MLP and Econometric analyses were done using the EXPO/NeuralNet T M (described in Markellos et aL, 1996) and EXPO/ Econometricff M software, respectively, on a desktop personal computer with a 486DX-2/66 type microprocessor.

308 Alvaro Costa and Raphael N. Markellos

that accelerates the convergence of the model optimisation procedure. A synthetic sample of 5000 input/output vectors was created by adding a zero mean/unit variance normally distributed noise to the outputs. All data were normalised in the range -1 to 1. Small zero-mean random initial MLP weights and a sigmoidal 'squashing' function were selected. During the process of deter- mining the best network topology and duration of training, models were re-trained 20 times using different random weight initialisations. This was done in order to decrease the danger that the network weights got trapped in a local minimum in the error space. MLP efficiency analysis was performed by using the two proposed approaches, i.e. econometric-MLP (ECN-MLP) and envelopment-MLP (ENV-MLP). Following the first approach, several models with different topologies were trained on the synthetic data and the original data were used for cross-validation. The training of each model was halted when the RMSE in the cross-validation sample started to increase. The model that minimised the SIC was selected as the best model. This selection was then validated by repeating the training process and using the 'gradient-norm' criterion method. On the basis of the second MLP efficiency analysis approach, an over-parameterised network with 50 neurons in the hidden layer was optimised on the original data until the signal-to-noise ratio reached the value 9/1, i.e. a level of 10% noise in the data. In both MLP approaches the efficiency measures were created, as in the COLS method, by correcting the downward bias in the estimated curve by shifting it upwards to the point that at least one of the corrected residuals was zero. DEA* efficiency analysis was performed on the raw data by assuming first constant (DEA/CRS) and then variable (DEA/VRS) returns to scale.

The problem of local minima during the MLP estimation process was quite severe with 90% of the random weight initialisations leading to sub-optimal solutions. The cross-validation and gradient-norm training strategies indicated the same SIC minimising MLP with one hidden layer and five hidden units. The optimal duration of training for this network topology was 60 optimisation cycles. The oversized network reached the 9/1 signal to noise ratio after 20,000 optimisation cycles. The results of applying the DEA and MLP approaches are depicted in Fig. 2. The results of the COLS technique are omitted since it identified all years as being very close to the frontier. A simple inspection of the graph indicates that all methods produced comparable results, identifying nearly identical local minima and maxima and an overall upward trend in the efficiency scores. This trend is much more apparent in DEA calculated for constant returns to scale (DEA/CRS). Both MLP and DEA approaches are able to identify important changes and events for the period under study, such as the changes in the organisational structure in the second half of the sample. An interpretation of the trend as the result of technical progress cannot be accepted, since no significant improvement was made to the London Underground technology during the period under study. Although the ECN-MLP efficiency scores where estimated in a similar way to the COLS method, no year has an efficiency of 100%. This is because the function is shifted according to the

1 0 0 ~ o , ," ° . . * * " " - " " ° - t ~ " " - . °~

" "~'~ ~"~"~'" °J Legend

- - 0 ¢ ~ ECN-MLP . . . . ENV-MLP

. . . . . . DEAICRS

. . . . . . . DEANRS

9 0 % , " ~ * " o * ' l S

Efficiency fl ,~/

80%

", i 70% ', : ; :

1970 1973 1976 1979 1982 1985 1988 1991 1994

Years

Fig. 2. DEA and MLP efficiency scores for the London Underground, 1970-1994.

DEA was performed using the Sciconic V/M Mathematical Programming Package on the Loughborough University Unix mainframe.


maximum error of the MLP in the synthetic data and not according to the error in the original data. In this way the ECN-MLP production frontiers are not deterministic in the sense that the COLS frontier is.

Table 2 presents the correlations (r 2) between the efficiency scores obtained by each method and the descriptive statistics of the scores. The correlations show that in general the MLP efficiency measures have a similar variance with DEA and COLS scores. The ENV-MLP frontier is closer to the DEA frontiers than the ECN-MLP. This is expected since the ENV-MLP envelops more tightly the data.

4. IMPLICATIONS FOR DECISION MAKING

Usually, the results of efficiency analysis are utilised in normative and positive analysis. Nor- mative analysis consists of looking at the history of one or several operators and identifying those that were most successful in terms of productive efficiency. These results are then interpreted in the light of other factors and events that effected the production process. Normative analysis also determines the level of output that 'should have' been produced for each historically realised combination of production inputs and these results can be used in drawing conclusions and making decisions about the optimal characteristics of future production. Positive analysis is concerned with setting output targets for given combinations of production inputs or input targets for a given level of production.

The proposed MLP methodology can be used in traditional tasks of normative and positive efficiency analysis but has much more to offer. The MLP method not only models a frontier with variable returns over scale, i.e. a non-linear frontier, as in DEA, but also gives the technology parameters of a continuous non-linear function of production inputs and outputs. Since these technology parameters are partial derivatives of a non-linear function, their overall impact, in the sense that they are calculated in linear models, can be calculated only approximately as an 'aver- age' effect via sensitivity analysis. This exact effect of each input is conditional to specific levels of production and inputs and can be evaluated by analysing different scenarios of production. Although the analysis of non-linear production functions and frontiers is more complicated than in the linear case it has the advantage of providing information on the 'local' slopes of the production function.

In Fig. 3 the estimated production function of the London Underground is presented. This graph was created by simulating the estimated ECN-MLP for 1000 different combinations of inputs (number of workers and vehicles) and recording the output (millions of train kms per year covered by fleet). Depending on the method used, the production frontier is estimated by a parallel shift of this surface. It is clear that the MLP has identified a non-linear process not only between each input and output but also between the inputs themselves. This effect is known as 'context' non-linearity and means that, for example, the effect of fleet size on production is not constant but depends non-linearly on the number of available workers. In economic terms this effect implies that inputs not only have variable returns over scale, but also variable returns across scales. Another important feature of the graph is that for most regions there is a negative or near zero local slope between inputs and outputs. Although at first this may appear as a spurious result, since one would expect a positive relationship between inputs and production, it is consistent with the events of the period under study. For the last 25 years it has been argued that the London Underground has been functioning in a congested area and that the historically realised outputs

Table 2. Correlation (r 2) and descriptive statistics of efficiency scores

ENV-MLP ECN-MLP DEA/CRS DEA/VRS COLS

ECN-MLP 0.551 DEA/CRS 0.438 0.346 DEA/VRS 0.196 0 0.736 COLS 0.424 0.681 0.287 0

Mean 0.922 0.881 0.789 0.906 0.998 SD 0.009 0.029 0.037 0.035 0.001 Max 0.935 0.909 0.864 1 0.999 Min 0.892 0.795 0.713 0.858 0.996

310 ,h, lvaro Costa and Raphael N. Markellos

Production train kms (mils)

5

4520

Workers 25820

Vehicles

Fig. 3. MLP production function for the London Underground, 1970-1994.

Output range

1~51 35-54

=n 4g.5.-51.75

Q 47.25-49.5

0 45-47.25

could have been succeeded with less inputs, i.e. vehicles and staff. This became apparent after the reorganisation of the operator, from 1984 onwards, a region that is depicted in the higher regions of the surface in Fig. 3. Since the estimated model is non-linear it could provide a single frontier that describes and explains both congested and increasing returns to scale areas of production.

Prior to using the MLP estimated function for efficiency measurement it can be studied and projected to asses expedient production configurations and trends. By assuming an output maximisation objective, the simulated data depicted in the previous figure were used to determine such expedient levels of production inputs. The comparison of these levels with respective present levels can be used to determine target levels for future inputs. The results of this analysis are shown in Table 3. It must be noted that this combination is unique and that the effects of each input are non-separable. This means that changing only one of the inputs towards its target level does not guarantee a positive effect.

A more realistic approach to the optimisation of the production process involves examining several scenarios of production. Usually, inputs of production are highly inelastic, i.e. changes in workforce are limited by unions, agreements, cost of compensations, etc., and therefore one should examine the optimal value of some input(s) for a given level of other inputs. Tables 4 and 5 present various scenarios of production for given workforce and fleet levels, respectively.

The production frontier can be used to calculate 'frontier' levels of production that were possible for given inputs. In the case of the MLP, these frontier levels are not given with reference to some optimal point or region of the production function, but by considering the most efficiently combined set of inputs. This means that success of production can be separated in two parts: productivity gains that result from the variable returns over and across scales, and, efficiency gains that are due to successful design and organisation of production. Table 6 gives the normative efficiency-based outputs or targets for the London Underground for the period 1970 to 1994.

Table 3. Optimal input combination for maximum production

Expedient level 1994 levels Policy target

Fleet 3950 3923 >4000 Workhorse 16,740 16,740 < 16740

Table 4. Optimal fleet and expected production for given workforce

Workforce scenario Optimal fleet Expected output (mil. kms)

21,280 4140 49.7280 24,690 4290 47.7302 17,880 3980 52.5546

Evaluating public transport efficiency

Table 5. Optimal workforce and expected production for given fleet

311

Fleet scenario Optimal workforce Expected output

4190 16,740 52.4641 4440 16,740 51.0854 3950 16,740 53.3271

Table 6. Actual and frontier outputinmillions of kms covered by the fleet

Year Actual Frontier Year Actual Frontier

1970 48.0 53.9 1983 46.7 52.0 1971 49.0 54.4 1984 46.3 53.8 1972 49.0 54.5 1985 46.8 54.3 1973 47.0 54.6 1986 48.1 55.5 1974 43.0 54.1 1987 50.1 57.5 1975 47.0 53.3 1988 50.5 56.9 1976 48.0 52.8 1989 50.1 53.8 1977 48.0 53.5 1990 52.4 55.0 1978 47.0 53.1 1991 52.5 54.8 1979 46.0 52.8 1992 52.5 57.5 1980 47.0 53.9 1993 52.7 58.6 1981 49.0 53.9 1994 54.8 59.4 1982 46.7 53.4

5. CONCLUSIONS

In this p a p e r we p r o p o s e d a m e t h o d o l o g y for mode l l ing p roduc t ion funct ions and measur ing p r oduc t i on efficiency, based on the M L P neural ne twork model . The p r o p o s e d M L P efficiency analysis is advan tageous over t r ad i t iona l ly app l ied techniques in tha t it has a flexible pa rame t r i c s tochast ic s t ructure and makes min ima l stat is t ical and theoret ica l assumpt ions . This m e t h o d o l o g y was discussed in c o m p a r i s o n with o ther pa rame t r i c and nonpa ra me t r i c app roaches and was app l ied to annua l da t a f rom the L o n d o n U n d e r g r o u n d , for the pe r iod 1970 to 1994. We found tha t the M L P approach , depend ing on its design, p r o d u c e d similar results to the C O L S and D E A methods . I t was demons t r a t ed how the M L P a p p r o a c h is able to provide more in fo rma t ion on evalua t ing the p roduc t i on funct ion than t rad i t iona l ly app l ied methods . This in fo rma t ion was utilised in no rma t ive and posi t ive analysis to d r aw specific r e c omme nda t i ons for decision making . Our results are consis tent with o ther studies and suggest tha t the L o n d o n U n d e r g r o u n d has been funct ioning with conges ted p roduc t ion for the pe r iod under s tudy and tha t the o rgan i sa t iona l changes since 1984 have resulted significant improvements .

A n i m p o r t a n t po in t tha t has no t been addressed by this pape r and has received l imited a t t en t ion in the l i te ra ture is the measurement o f efficiency when the pa rame te r s o f p roduc t ion are d i s t r ibu ted over t ime as nons t a t i ona ry var iables . Ra the r than ignor ing the presence o f a t ime trend, or than assuming that it is determinis t ic , researchers mus t search for ways o f deal ing with s tochast ic uni t roo t s (trends). Fu tu re research will be d i rec ted in deve loping a f r amework for ana lys ing p roduc - t ion var iables con ta in ing unit roots and in p rov id ing M o n t e - C a r l o compar i sons between neural ne twork , economet r ic and ma thema t i ca l p r o g r a m m i n g efficiency measuremen t techniques.

Acknowledgements- -We wish to thank Tom Weyman-Jones, Terence Mills and two anonymous referees for their invaluable comments on earlier versions of this paper. The authors are fully responsible for any remaining errors. We also thank Jay Smith, Leading Market Technologies Inc., Cambridge, MA for providing the EXPO/NeuralNet T M and EXPO Econometrics T M software used in this study. We acknowledge financial support from the Caloustc Gulbcukiat~ Foundation, Lisbon and the Department of Economics, Loughborough University, respectively.

REFERENCES

Costa, ,/~. (1996) Public transport efficiency and effectiveness: Metro de Madrid. In I",r,,f)c,m I,,mW,,, ~ \,-:~,," :,, ~ ." , , , " , Analysis and Policies, ed. K. Button, P., Nijkam and H. Priemus. Edward I!ig:l l , ( 'hclicldt:~m. I,,t th, ,,1~11,1c

Deboeck, G. J. and Cader, M. (1994) Pre- and postprocessing of [inancial dala In l),hlm.c ,,,t Jh,- ? .~: ,~,L ~, I 1\",,,',,',,-'~, pp. 27--44. Wiley, New York.

312 ,/klvaro Costa and Raphael N. Markellos

Dougherty, M. (1995) A review of neural networks applied to transport. Transportation Research 3C, 247-260. Efron, B. and Tibshirani, R. J. (1193) An Introduction to the Bootstrap. Chapman and Hall, London. Fielding, G. J., Babitsky, T. and Brenner, M. (1993) Performance evaluation for bus transit. Transportation Research 19A,

73-82. Fried, H. O., Lovell, C. A. and Schmidt, S. S. (eds) (1993) The Measurement of Productive Efficiency. Oxford University

Press, Oxford. Fu, L. M. (1994) Neural Networks in Computer Intelligence. McGraw Hill, New York. Ganley, J. and Cubbin, J. (1992) Public Sector Efficiency Measurement: Applications of Data Envelopment Analysis. Elsevier. Gathon, H. J. (1989) Indicators of partial productivity and technical efficiency in the european urban transit sector. Annals

of Public and Cooperative Economics 60, 43-59. Hipel, K. W. and McLeod, A. I. (1993) Time Series Modelling of Water Resources and Environmental Systems. Elsevier,

Amsterdam. Hornik, K., Stinchcombe, M. and White, H. (1990) Universal approximation of an unknown mapping and its derivatives

using multi-layer feedforward networks. Neural Networks 3, 535-549. Jondrow, J., Lovell, C. A. K., Materov, I. S. and Schmidt, P. (1982) On the estimation of technical inefficiency in the sto-

chastic frontier production function model. Journal of Econometrics 19, 233-238. Lovell, C. A. (1993) Production frontiers and productive efficiency. In The Measurement of Productive Efficiency, ed. H. O.

Fried, C. A. Lovell and S. S. Schmidt, pp. 3-67. Oxford University Press, Oxford. Markellos, R. N., Mills, T. C. and Siriopoulos, C. (1996) Handbook of Neural Network Analysis with EXPO/NeuralNet TM.

LMT, Cambridge, MA. Ostaszewski, A. (1993) Mathematics in Economics, Blackwell, Oxford. Ripley, B. D. (1993) Statistical aspects of neural networks. In Networks and Chaos--Statistical and Probabilistic Aspects, ed.

O. E. Barndorff-Nielsen, J. L. Jensen, and W. S. Kendall, pp. 40-123. Chapman and Hall, London. Schwartz, G. (1978) Estimating the Dimension of a Model. Annals of Statistics 6, 461-464. Smith, M. (1993) Neural Networks for Statistical Modelling. Van Nostrand Reinold, New York. Weyman-Jones, T. (1991) Productive efficiency in a regulated industry. Energy Economics 2, 116-122.

Documents

Evaluating public transport efficiency with neural network models