Professional Certificate in Policy

Implementation and Evaluation

Ian “Ren” Rennie

INTRODUCTION TO POLICY EVALUATION

• Who Evaluates? • Elites:

– Government Agencies, Think Tanks, National or International Audit institutions, Regulators

• Interest Groups: – Trade Associations, Business groups, Pressure groups,

Media

• Citizens: – Complaints, Whistle Blower laws, Public Opinion, Voting

• Of course, some agencies may be shielded from evaluations, some may have repeated evaluations thrust upon them.

Politics of public policy evaluation

• Determine policy outcomes

• Identify policy strengths

• Identify and improve weaknesses

• Justify use of resources

• Increased emphasis on accountability

• Professional responsibility to show effectiveness.

Why evaluate?

• Outcome

• Implementation

• Formative

• Summative.

Types of evaluation

• Summary process.

Evaluation

Identify scope and

stakeholders

Analysis

Information gathering

Identify sources of

information

Team Selection

Compare with: business strategy

business case

Use the information for more effective business operations

• Why are you doing the evaluation? – mandatory? – policy outcomes? – policy improvement?

• What is the scope? How large will the effort be? – large/small – broad/narrow

• How complex is the proposed evaluation? – many variables, many questions?

• What can you realistically accomplish?

Scope of evaluation

• There are a number of common challenges that may be encountered

• the evaluation team need to be aware of these

• although they may not be able to solve them.

Common challenges

• more than one organisation involved, no common standard for measuring and recording benefits and costs

• lack of documentation • inadequacy of baseline measures - measures of success can

only be made accurately by comparing the level of performance before implementation

• sensitivities – examining performance may lead to feelings of insecurity or grievance for those involved

• the organisation is too busy and it never gets it done (there should be policies to ensure that it is normal practice)

• lack of cooperation by involved agencies.

Challenges:

• There is one most important question

• What is it that you want to know about your policy?

• Once you have defined this make it measurable

• Do not move forward if you cannot answer this question!

Ensuring effective and efficient monitoring

• What is the purpose?

• What should we measure to ensure that the policy: • is producing the required products

• is being carried out to schedule and in accordance with its resource and cost plans

• remains viable against its business case

• Monitoring activities facilitate the checking and reporting on progress against the plan

• Control activities promote revisions to be made to the plan in response to issues discovered during monitoring.

Monitoring ... for control

• To benchmark is to compare performance against a standard

• To improve the effectiveness of monitoring and evaluation (M&E) in the policy cycle, benchmarking can be useful in three ways

1. can help place an outcome in context - was the achievement good, bad, or indifferent?

2. can help assess the reasonableness of targets that may be set

3. can help identify specific regions or subgroups whose exceptionally good or poor results hint at what factors drive performance.

International comparisons and benchmarking

• Today, there is much online availability of large internationally comparable data sets (such as those in the World Bank’s Data Development Platform)

• Plus widely available national survey data

• It is now considerably easier to use international data for benchmarking exercises.

Benchmark with what?

Range of institutions

LINKING POLICY IMPLEMENTATION AND POLICY EVALUATION

• Give an example of a performance indicator for your area of responsibility ...

Exercise

• Consider these assumptions:

• Indicators will provide clear answers to questions about policy efficiency and effectiveness

• Indicators will help explain how decisions and decision outcomes are linked

• Having the right indicators will result in better decisions.

Performance indicators

• Political use – indicators used to support a predetermined position

• Symbolic use – ritualistic assurance about appropriate attitudes towards decision-making (“we even developed indicators”)

• Tactical use – indicators and the process used as a delaying tactic or substitute for action (“we will do it later once we have indicators”)

• Conceptual use for enlightenment – change the understanding of an issue

• Instrumental – direct relationship between indicators and decision outcomes.

Possible motives

• Indicators play an important role in the policy cycle and can serve a useful purpose

• They can – Help assimilate and better understand stakeholder views – Add to the process of governance – Help guide and mould policy decisions

• Indicators are linked to the governance process ... • “which can act like a magic bullet causing decision-

making to become instantly objective and scientific”.

Indicators in the Policy Cycle

• Saliency

– will anyone care?

• Legitimacy

– can people trust it?

• Credibility

– can people believe it?

• Usability

– can people understand it?

Observing agency outputs

Standards for an M&E Plan

• Utility – serve practical information needs of intended

users

• Feasibility – be realistic, prudent, diplomatic and frugal

• Propriety – conducted legally, ethically, and with regard to

those involved in and affected by the evaluation

• Accuracy – reveal and convey technically accurate

information.

Factors to Consider When Setting Targets • What can realistically be achieved given the

resources and the program context?

– Baseline levels

– Past trends

– Needs and gaps in services

– Capacity and logistics

– International benchmarking.

Approaches to Setting Targets

• Establish final target then plan progress in between

• Establish intermediate targets

• Assess progress in attaining targets and readjust, if necessary.

• Past trends

• Expert opinion

• Research findings

• What has been accomplished elsewhere

• Client expectations.

Useful Information for Setting Targets

• Advocate for the need for M&E

• Understand program goals and objectives

• Identify user needs and perspectives

• Learn about existing data collection systems & their quality

• Understand indicators that are being collected and used (or not used)

• Determine capacity for collecting and using data.

Stakeholder Consultation

• Developing M&E framework

• Selecting indicators

• Setting targets

• Reviewing results

• Requires building consensus & commitment and maintaining effective relationships with intended users.

Stakeholder Participation

DEVELOP A DATA COLLECTION MATRIX FOR YOUR PROGRAM. CONSIDER THE FOLLOWING ISSUES:

• Who will be responsible for data collection and its supervision? • Who will be responsible for ensuring data quality at each stage? • How will data quality be checked at every stage? • How often will the data be collected, compiled, sent, and analyzed? • What indicators will be derived from each data source? • How will the data be sent (raw, summary)? • What tools/forms will be used, if any? • What resources (staff, office supplies, computers, transportation) will

be needed at each stage? • Who will analyse the data? How often will analysis occur? • How often will the results be compiled into reports? • To whom and how often will the results be disseminated?

Data Collection Plan

Developing & Implementing an M&E Plan: Logic Model

Inputs •Human resources •Understanding of the program •Authority and mandate •Stakeholders

Processes •Advocate •Assess strategic information needs •Assess information systems capabilities •Achieve consensus and commitment •Develop mechanism for M&E plan review •Prepare document for final approval

Outcomes

Output •M&E Plan Document

Short-term •M&E System for obtaining Strategic Information decision making Long-term •Evidence-based decisions for improving programs

Impacts •Policy delivery

• Start early

• Involve stakeholders at all stages in the process

• Assess strategic information needs for intended users

• Assess current capacity and use what is already available

• Avoid duplication of data collection and reporting

• Do not collect information that will not be used

• Review progress / results regularly and make adjustments to M&E plan, if necessary.

Summary: The Basics

DIRECT VS INDIRECT POLICY IMPACTS

Dandelion dilemma

• Case study 1 – Continuously troublesome students are excluded from

school to prevent disturbance of well behaved students

• Case study 2 – in 18th- and 19th-century northern Europe, there was

a “window tax” – a property tax whose amount depended upon the

number of windows in a house – an explicit “income tax” was politically unpopular, so

the intention was to create an income tax without explicitly taxing on the basis of income.

First order and second order impacts

• Unintended consequences arise because we live in an interconnected and complex world

• Interconnectedness and complexity of the world imply that a small change in one part of a system can potentially generate far-reaching effects in another part of the system

• Also policymakers may be unable to anticipate or understand how their policies affect the incentives of individuals within the system.

Complex systems

• public policy attempts to address social problems, often without understanding the policy’s effects on individuals’ incentives

• systematic consideration of a policy’s intended and unintended effects is imperative

• just because a policy could theoretically affect individuals’ incentives doesn’t mean that the policy actually will

• the extent to which individuals respond to incentives is an empirical question, which requires an empirical answer.

Empirical answers

POLICY EVALUATION METHODOLOGIES

• The good evaluation of a project or a policy needs on the one hand thorough application of methodology up to the highest professional standards ...

• and is on the other hand a creative thought exercise: what do we really want to know and how to find out.

Evaluation methodology

• If policies are meant to improve on outcomes, they should not be based on ideology or wishful thinking, but rather on proven effectiveness

• But obtaining convincing evidence on the effects on specific policies is not an easy task

• Relevant data on possible outcomes has to be gathered • Showing a mere correlation between a specific policy or practice

and potential outcomes is not proof that the policy or practice caused the outcome

• For policy purposes, mere correlations are irrelevant, and only causation is important

• What would really happen if a specific policy is introduced – would it really change any outcome that society cares about?

• Evidence-based policy require answers to causal questions.

Overview

• The need for evidence-based policy is increasingly recognised (e.g., Commission of the European Communities 2007)

• Providing empirical evidence suitable for guiding policy is not an easy task

• It refers to causal inferences that require special research methods

• Not always easy to communicate due to technical complexity

• Econometrics is a set of quantitative techniques that are useful for making "economic decisions“

• Of course, economic decisions are not restricted to those made by economists ...

Econometrics

• The decision of a student to stay an extra year at university to earn a double degree

• The decision of a spice seller to open a shop in this village, or another

• The decision of a bank to approve a loan application

• The decision of a power company to start-up an extra generator on a certain day

• The decision of a Government to introduce a purchase tax in order to reduce income taxes.

Examples:

• The "efficient allocation of resources" is the subject matter of economics

• The traditional role of econometricians is to "test" alternative economic theories

• To confront theories with real data and to reject those that are not compatible with reality

• It is what transformed economics from an "art" into a "science“

• However, it only gave the econometrician an indirect role as an economist's sidekick

• If economists didn't come up with new theories, econometricians didn't have anything to do!

What is old?

• Regardless of what economists do, economic problems persist in reality and decisions have to be made

• The modern role of an econometrician is therefore more direct

• Econometricians extract information from the available data in order to provide support for an economic decision

• The raw material that the econometrician uses is more often real observed data, instead of economic theory

• In any specific problem if an economic theory that has been validated by data exists, it will certainly be used

• If no such model exists, however, econometrics must still make a statistical model for the data, and provide intelligent support for the decision process.

What is new?

• In government decision-making there are always competing priorities for funds

• The purpose of a CBA is to allow competing policy priorities to be compared in a consistent way, and for their economic, social and environmental impacts to be assessed

• In all areas of policy, the function of the CBA is to assist policymakers to identify the best way to deliver the strategic objectives of governments

• And then to assist with the evaluation of the policy.

Cost Benefit Analysis (CBA)

1. needs to be future looking

2. needs to be objective

3. needs to consider implementation risks

4. to be easily understood so it can be subject to a degree of contestability.

What does a good CBA look like?

1. establishing the base case – or the ‘no change’ policy option

2. defining the policy options to be evaluated and compared against the base case and against each other

3. laying out the estimates and assumptions for external factors affecting the policy outcomes

4. defining and estimating the costs of a policy proposal 5. defining and estimating the benefits of a policy

proposal 6. drawing a conclusion.

Key steps in a CBA

• An initial question ‘what would happen if this investment did not take place?’

• allows the CBA to compare two futures: the future with the investment and that without the investment

• The base case does not mean ‘how things have been in the past’ and should include any observed long term trends

• In particular, the base case needs to account for future population and economic growth

• All forecasts contain an element of uncertainty • In order to address this uncertainty, forecasters need to

develop different forecast scenarios (usually low, mid-range and high scenarios).

1. Establishing the base case

• The purpose of any investment proposal is to achieve an outcome that improves the wellbeing of the community

• In many instances there are multiple approaches that could achieve the outcome

• Similar to the base case, the policy options in a CBA should be clearly articulated and reasonable

• It is also important to consider whether there are any policy options that may have been overlooked and not included in the CBA

• A well thought-out CBA includes all the feasible options and provides justification where options have not been covered

• Avoid “Goldilocks” analysis.

2. Defining the policy options

• Analysis will be built up from a range of estimates and assumptions • Estimates are variables where there is existing evidence, preferably

from multiple sources • Assumptions are variables where there is less robust data and the

team developing the CBA has had to use judgement • Estimates are clearly preferable to assumptions • A CBA, however, almost always includes assumptions because not

all of the information needed to perform the analysis is available • Just because it is necessary to make some assumptions to conduct a

CBA does not mean it is not worth doing as all analysis involves assumption about the future

• One strength of CBA is that it makes it clear what the key assumptions are and allows for the systematic testing of assumptions.

3. Estimates and assumptions

• What assumptions need to be made to conduct the analysis?

• consider the impacts of a proposal across all sectors of the economy by thinking about the inter-relationships between sectors (economists term this ‘general equilibrium analysis’)

• include all intangible costs and benefits where they can be reliably estimated

• include all subsequent or contingent costs/investments

• analyse over the economic life of the underlying proposal or assets, subject to a maximum of 20 years.

3. Estimates and assumptions

• Takes into account all the impacts of the policy options that produce undesirable impacts

• A useful way of looking at costs is to identify which individuals or groups in the community are worse-off as a result i.e. – Government – Business – The community

• The full range of costs in a CBA should be identified, even those that may be difficult to measure - understating or overlooking a cost that is difficult or impossible to quantify is a common fault with CBAs

• Everything is relative to the base case – that is, the costs that are additional to the base case.

4. Costs

• Helpful to ask, ‘who is made better off as a result of the investment?’ i.e. – Government: time or money saved through more efficient programs or

processes – Businesses: reduction in time or cost, particularly with respect to

administration, or improvements in efficiency, productivity and innovation – The community: increase in welfare, safety, participation and/or connectivity

• Many CBAs will also look at benefits according to the ‘triple-bottom-line’ which covers the positive economic, environmental and social outcomes of the investment

• While there is nothing wrong with classifying benefits under these headings, it is crucial to avoid double-counting

• Like the cost side of the CBA, it is important to identify all the benefits related to the policy options and measure them relative to the base case.

5. Benefits

• Finally calculate the value of each policy option taking into account all future discounted benefits and all future discounted costs

• The final overall value is usually expressed in one of three ways: – Net present value or Internal rate of return or Benefit-cost ratio

• In a CBA some options may be easier to value than others • Although CBA places an emphasis on quantifying in financial terms

the impacts of potential options, the CBA should not be biased towards an option just because it is easy to value

• This does not necessarily make it the best approach • However, where difficult-to-value impacts are included, document

carefully and transparently the manner in which that has been done.

6. Drawing a conclusion

• An important and often overlooked step, and purpose is to:

– Identify whether the expected benefits of the project have been realised

– Assess whether any aspects of the current project require remedy

– What lessons can be learned from the project for future investment projects.

Post implementation

• Quantitative observations are made using scientific tools and measurements

• The results can be measured or counted • Any other person trying to quantitatively assess the

same situation should end up with the same results • An example of a quantitative evaluation would be

"this year our government paid 4m citizens a pension totalling $4,000m at an average of $1000 per pensioner.“

Quantitative

• More subjective than quantitative evaluation, qualitative observations are defined in science as any observation made using the five senses

• Because people often reach different interpretations when using only their senses, qualitative evaluation becomes harder to reproduce with accuracy

• Two individuals evaluating the same thing may end up with different or conflicting results

• In research and business, qualitative evaluations may involve value judgments and emotional responses

• An example of a qualitative evaluation is "our government provided a better pension arrangement last year than this year."

Qualitative

• Both evaluation methods have their benefits, though one is usually more appropriate than the other in any given situation

• Quantitative evaluation can help remove human bias from a statistic, making it more of a reliable fact than any piece of information gathered qualitatively

• Thus, accurate quantitative evaluations can be relied upon as truth • Qualitative evaluations may also entail truths, but these truths are

harder to get at, and evaluators may not always agree • Still, qualitative information is invaluable precisely because it

involves human interaction and interpretation • Many fields need this interpretation in order to decide how to move

forward or to judge the value of past and future sources of data.

Benefits

• Individuals or groups are randomly assigned to either the policy intervention or non-intervention (control) group and the outcomes of interest are compared

• There are many methods of randomisation from field experiments to randomised control trials

• Random allocation design means that systematic differences between groups are unlikely and so any differences and changes in outcomes between the two groups can be confidently attributed to the policy intervention.

Control groups

Random allocation design

• Increasing recognition that rigorous evaluation of public interventions should feature in the policy decision making process

• Yet there is frequently a gap between the desire for information on the effectiveness of programs and an understanding of the potential and the limitations of evaluation tools

• What questions can evaluations answer? • What administrative structures are required to implement them? • What are the political and social factors surrounding the acceptance

of evaluations by target groups and the public? • How much do evaluations cost? • How long do they take to complete?

Cost-implications of different methodologies

• For international aid programmes it is suggested that the cost of an evaluation is between about 5 and 7 percent of the cost of the program being evaluated as a rule of thumb (W. K. Kellogg Foundation, 1998)

• Other evidence suggests that there is very little correlation between the cost of an impact evaluation and the size of the program being evaluated.

Evaluation costs

• Objectives – the policy questions to be addressed determine how complex

the study design must be and consequently how costly – simpler is usually better

• Availability of representative socioeconomic data – if national data exists, alternative methods of selecting a

comparison group can be considered which may be cheaper than the ideal random assignment design

• Timeframe – evaluators frequently do not have the luxury of months or years

for data collection – this lack of time necessitates more creative use of data and

more complicated statistical techniques, potentially raising the overall price of evaluation.

Cost factors

IMPACT ANALYSIS

• Conduct a gap analysis • Determine whether the project

goals were achieved • Determine the satisfaction of

stakeholders • Determine the project's costs and

benefits • Identify areas of further

development • Report findings and

recommendations.

Questions for post-implementation review

• An equality impact assessment (EIA) is a tool that helps public authorities make sure their policies, and the ways they carry out their functions, do what they are intended to do and for everybody

• Carrying out an EIA involves systematically assessing the likely (or actual) effects of policies on people in respect of disability, gender, including gender identity, and racial equality and, where you choose, wider equality areas

• This includes looking for opportunities to promote equality that may have previously been missed or could be better used, as well as negative or adverse impacts that can be removed or mitigated, where possible.

What is an Equality Impact Assessment

• In many countries, if any negative or adverse impacts amount to unlawful discrimination, they must be removed

• Typically there are four possible outcomes and more than one may apply to a single policy: – No major change – Adjust the policy – Continue the policy (justification) – Stop and remove the policy.

Possible Outcomes

• Statutory Equality Duties Guidance • Competition Assessment • Small Firms Impact Test • Greenhouse Gas Impact Assessment • Wider Environmental Impact Test • Health and Well-Being • Human Rights • Justice Impact Test • Rural Proofing • Sustainable Development Impact Test.

Other impact tests include ...

• http://webarchive.nationalarchives.gov.uk/20100503135839/http://www.ogc.gov.uk/documents/policy_to_delivery_A4_Guidance_P1_(2)_(2).pdf

• https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220541/green_book_complete.pdf

• https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf

• http://webarchive.nationalarchives.gov.uk/20130129110402/http://www.hm-treasury.gov.uk/d/orange_book.pdf

• http://webarchive.nationalarchives.gov.uk/20130129110402/http://www.hm-treasury.gov.uk/d/riskmanagement_assessment_framework220709.pdf

References

QUESTIONS AND CLOSE