Measuring criminal court efficiency using DEA

Charlie Lee

Forecasting and Model Development Unit (FMDU)

CJ Sig Event

March 2013

2

Agenda

• Background

- Background information about the project and why we are doing it.

• Overview of the DEA methodology

- Overview of the main points in the Data Envelopment Analysis methodology.

• Developing the DEA model for the Crown courts

- A walkthrough of things to consider when developing the DEA modelling.

• Next steps

3

Background

• HMCTS currently record vast quantities of management information related to the court

process.

• Regular performance reports and charts are produced and updated to keep track of

various measures and targets related to court performance.

• However, because there is so much data, it can be difficult to determine which courts are

efficient and where to focus resources on.

Project aim

To explore a new analytical method of comparing the court level performance between

each court (for the Crown Courts and magistrates’ courts) using a combined range of

indicators.

• Identification of most efficient courts for best practice

• Identification of least efficient courts for improvement.

• Identification of courts that are most similar, and why they differ in efficiency.

To provide HMCTS decision makers with useful information and give a new perspective on

efficiency performance to enable better use of time and resources

4

What is Efficiency?

In its simplest form, efficiency can be described as: .

This can be applied to any input and output, for example:

Input

Output

Papers per Author =Authors

Papers Goals Per Game = Sandwiches = Per Hour Games

GoalsHour

Sandwiches

Efficiency can be defined as a measure of how well inputs are converted into outputs:

Inputs OutputsProcess

A unit (e.g. a university, a football team, a cafe) can then measure their relative performance in

two ways:

1) Against other units operating in the same field, where the higher their score, the higher their

efficiency.

2) Against a theoretical benchmark, where the nearer to the benchmark they get, the higher their

efficiency.

5

Measuring Efficiency with Multiple Inputs and Outputs

In most cases, combining multiple single efficiency ratios can be problematic as they

can include contradictory views. For example, who is more efficient, Café A or Café B?

The most simple way around this problem is ratio analysis, which works by creating a

weighted single score.

Although simple, this method requires an agreement on the weighting of scores which can

lead to disagreements between units under investigation.

More advanced techniques such Data Envelopment Analysis can avoid this problem.

Sandwiches sold per staff

Teas sold per staff

Efficiency score

Weighting 0.2 0.8Café A 27 5 9.40Café B 11 16 15.00

Sandwiches sold per staff

Teas sold per staff

Café A 27 5Café B 11 16

6

Data Envelopment Analysis (DEA)

Data Envelopment Analysis (DEA), also known as frontier analysis, is a linear programming methodology to measure the efficiency of multiple Decision Making Units (DMUs) when the production process presents a structure of multiple inputs and outputs.

• Can handle multiple inputs and outputs

• Relative peer to peer measure

• Non Parametric approach

Results

• Gives a single efficiency score, although multiple units are likely to be 100% ‘efficient’.

• Shows target input production and/or output consumption required to achieve

efficiency.

• Shows potential role models (efficient peers) for comparison.

7

How DEA works

Theoretical CafeCafé A Café B

Inputs

(u1) * Number of Staff (x1)

Outputs

(v1) * Sandwiches sold (y1)

(v2) * Tea sold (y2)

Café X

Stage 1 - Apply weights (ui, vi) to each of the input (xi) and outputs (yi) for the first unit. DEA selects the weights that maximises the units’ efficiency, which eliminates the need to create an agreed set of weighting between units.

Stage 2 – Apply the same weights to all other units and see if they can achieve a better rating. This can be a single unit or a selection of units to make a theoretical achievable unit point.

Stage 3 – Efficiency scores are created based on the difference between the efficient unit and the unit under investigation given the weighting

Stage 4 – Repeat from stage 1 with next unit

8

Output 1

Out

put 2

Café A

Café B

Café X

Theoretical Achievable Point

Café C

0

Graphical Illustration

Given an example with two outputs and a single input, an efficiency frontier can be created by drawing straight lines between units that are 100% efficient. In the example below this is Café A, Café B and Café C. It is then assumed that all points on this line are achievable and 100% efficient.

When evaluating a units’ efficiency that does not lie on the frontier, a straight line can be drawn to the origin point. In the example, this is represented by the combination of the solid and dashed line between Café X and (0,0).

The efficiency of Café X is therefore the ratio of the distance between 0 and Café X over the distance between 0 and the theoretical achievable point.

9

DEA Formulation

J

jjkj

I

iiki

xu

yv

1

1

1

1

1

J

jjnj

I

iini

xu

yv

I

iiki yv

1

Max

Subject To

011

J

jjnj

I

iini xuyv

11

J

jjkj yu

0, ji uv 0, ji uv

Max

Subject To

Ratio Formulation Linear Formulation

The model appeared first in ratio form, but can be converted to its linear form, making it easier to solve. It works by maximising the objective by varying the weights, but ensuring that applying those same weight to every other units will not lead to a score greater than 1.

10

Input

Out

put

Orientation of DEA

Input Orientation

Attempts to minimise the inputs given the outputs

Input

Out

put

Output Orientation

Attempts to maximise the outputs given the inputs

e.g. Use fewer staff to serve the café’s regular customers who always order the same sandwiches and teas.

e.g. Keep the café’s staff numbers the same, but sell more sandwiches and teas to the regulars or attract new customers.

11

Economies of Scale

Input 1

Inp

ut

2

Input 1

Inp

ut

2

Constant Returns to Scale (CCR)

Assumes that economies of scale have no impact and everyone operates under the conditions that creating 100 units of outputs i requires exactly 100 times the effort as creating a single unit of output i.

Variable Returns to Scale (BCC)

Assumes that economies of scale have an impact on units, therefore acknowledging that smaller and larger units outputs may benefit or be affected by the units size.

This will always lead to more units becoming 100% efficient.

e.g. Decide if size affects how a café operates. A larger café may be more difficult to manage due to its size, but can buy its goods can in bulk. Small cafés may be too small to operate at optimal efficiency, but maybe better at retaining customer loyalty.

12

Developing DEA modelling for

the Crown courts

13

Overview of the Criminal Courts in England and Wales

Recorded Crime

Offence Detected

CPS

Out of Court Disposals

Magistrates’ court

Sent

for TrialMags court

proceedings

CPS

discontinue

AppealMags court Sentencing

Committed for Sentencing

Crown court

Committed

for Trial

GuiltyNot guilty/

acquitted

14

Inputs and Outputs Selection

Possible Input measures

Input measure Description

Court sitting

days used A measure of the number of days spent to deal with the workload. A

courts budget is allocated based on the estimated sitting days

needed to deal with its workload.

Court sitting

hours usedSimilar measure to sitting days, some courts may sit a different

number of hours in a day. This could be used as a proxy for the

amount of effort spent in a court.

Number of Staff An indication of the resources available to the court.

Financial Operating

Costs The actual costs involved in running the court, care must be taken to

examine in detail what is included, as some types of costs may not

be relevant for comparison.

15

Inputs and Outputs Selection

Possible Output measures

Output measure Description

Disposals A measure of throughput for the court, these are the volumes of

cases dealt with.

Early Guilty Pleas Where the defendant pleads guilty before the trial start.

Vacated Trials Where a case listed for trial is cancelled before the trial start date.

Ineffective Trials Where the trial is cancelled on the day of the trial.

Trial Waiting Time A timeliness measure, the time from lodging the hearing/trial to the

start of the hearing/trial.

Juror Utilisation Not all jurors selected for Jury are utilised, a low juror utilisation rate

means a wastage of resources and time.

Witness Waiting time A timeliness measure for witnesses called to give evidence.

16

Inputs and Outputs Selection

Issue

• The DEA method, in common with all linear programming are extreme point

techniques.

• The results are therefore sensitive to all inputs and outputs selected, as all variables

can be seen as important as each other.

• The more inputs and outputs selected, the more 100% efficient units there are.

Solution

• Careful selection of inputs and outputs.

• Work closely with customers to determine priorities and include only a few of the most

relevant inputs and outputs.

• Check the data availability and quality is feasible for the analysis.

17

Average court duration time

Burglary HandlingStolenGoods

SexualOffences

MaliciousWounding

Murder Sentencing Averagecourt duration

Investigating the case mix effect

• The different types of cases tried in court will vary in their complexity as indicated by their average court duration time.

• Due to regional variations, some courts may deal with more of a particular type of case, their case mix will vary.

e.g. Some courts may only deal with serious cases such as murders, whilst others may deal with more burglary and handling stolen goods.

• Therefore, we need a method to fairly compare courts given their case mix.

The average court duration times shown are for illustration only

18

• To deal with the case mix effect, we have created a weighting system by sub offence level based on the average court duration time for a particular sub offence compared to the average case in England and Wales.

• For example, if the average court case is given a weighting of 1 point, then a Murder is worth 13 points, as the average court duration for a murder case may take 13 times longer than an average case.

• Using this weighting system enables the courts to be fairly compared against each other when we consider their volumes of disposals as an output.

Investigating the case mix effect

The figures shown above are for illustration only

Case type (Sub offence)Case mix weighting

Burglary 0.8

Handling Stolen Goods 1.0

Sexual Offences 4.0

Malicious Wounding 2.0

Murder 13.0

Sentencing 0.5

Average court duration 1.0

19

Crown Court Investigation

To illustrate the sensitivities when selecting the parameters for DEA modelling, the following simplified models will demonstrate graphically how the results can vary.

The models are run using an output orientation on the assumption that inputs are fixed by HMCTS and given these inputs they should be producing a certain amount of outputs.

Inputs

Sitting Hours

Outputs

Disposals

Model 1: One input and one output (Constant returns to scale)

Inputs

Sitting Hours

Outputs

Disposals

Model 2: One input and one output (Variable returns to scale)

Inputs

Sitting Hours

Outputs

Disposals

Early Guilty Pleas

Model 3: One input and two outputs (Variable returns to scale)

20

Model 1: one input and one output

The points on the chart represent the 76 Crown courts (DMU’s), disposals as an output are plotted against an input of sitting hours used. This is a standard ratio analysis model to illustrate the differences in results when scale is not considered.

Sitting hours

Dis

po

sa

ls

Efficiency LineDMU 4

DMU 1

DMU 2

DMU 3

21

Model 1: one input and one output

We can see the most efficient unit is DMU 1, and those furthest away from the efficiency frontier are considered to be the least efficient.

Sitting hours

Dis

po

sa

ls

DMU 4

DMU 1

DMU 2

DMU 3

DMU 4 is far away from the efficiency line and relatively not very efficient in this model.

22

Model 2: One input and one output (Variable returns to scale)

Using the same data points, if we choose a variable returns to scale approach, the efficiency frontier changes as more units are now captured as being most efficient.

The efficiency score for the other units also change as efficiency is relative to the efficiency frontier.

Sitting hours

Dis

po

sa

ls

DMU 4

DMU 1

DMU 2

DMU 3

DMU 4 now lies on the efficiency frontier and is relatively 100% efficient compared to peers in this model.

23

Sitting hours

Dis

po

sa

ls

Model 3: One input and two outputs (Variable returns to scale)

Using the same data points as before, we include an extra output measure (early guilty pleas) into the

model. The same efficient units from the previous model are still efficient, but we now have a few more

efficient units (DMU 5, 6, 7), the next chart shows why they are included.

DMU 4

DMU 1

DMU 2

DMU 3

These courts (DMU 5, 6, 7) are not on the frontier for disposals but are considered 100% relative to peers.

DMU 5

DMU 6

DMU 7

24

Sitting hours

Ea

rly

Gu

ilty

Ple

as

Model 3: One input and two outputs (Variable returns to scale)

The new efficient units (DMU 5, 6, 7) are efficient because they lie on the efficiency frontier for early

guilty plea. This is an example of DEA using extreme points to consider both outputs simultaneously to

give an overall result.

DMU 4

DMU 2

DMU 3

DMU 1

DMU 7

DMU 6

DMU 5

DMU 4 performs relatively poor for EGP, but is considered 100% efficient relative to peers, when both outputs (disposals and EGP) are considered.

25

In summary…

• These basic models have illustrated that careful consideration must be given to select

the appropriate parameters, input and output measures to include in model development.

• Although the modelling is still developing the initial reaction from our customers have

been very positive, they have been engaged in our meetings and very interested in the

information this methodology can provide.

• The resulting modelling will hopefully help our customers gain greater insight into their

business and provide an alternative perspective on court performance, neither of which

have previously been possible with the existing MI tools and performance reports

26

Next steps…

• Continue to develop and refine the Crown court DEA modelling• Explore and evaluate the results from the selection of inputs and outputs• Drill down to the court level to find out what is driving the efficiency scores• Explore using weight restrictions for some outputs and evaluate the results

• Apply a similar methodology to the magistrates’ court.• Magistrates’ court has over 220 courts across 7 regions in England and Wales• Possibility to model at regional level• Slightly different priorities and measures

Thank you!