Contribution Analysis: An introduction

Anita MorrisonScottish Government

National Performance Framework

Evaluation in Scottish Government

• A need to revisit our practice in context of performance management and outcomes-based approach – shift balance from small scale, process evaluation

• Building a more ‘strategic’ approach to evaluation planning and delivery - multiple policies / interventions to deliver outcomes

• Ensure the right kind of evidence is available – answering

questions of effectiveness, contribution, for Spending Review 09 and beyond

Key analytical questions

• What progress are we making towards the Purpose, Purpose targets and National Outcomes?

• What contribution have our policies and programmes made?

• How cost-effective are our policies and programmes?

• What are the lessons for future policy intervention /changes in resource allocation?

Contribution Analysis

Contribution analysis

• Holy grail of attribution…“Builds a reasonably credible case about the difference the program is making.” (John Mayne)

• Good contribution analysis is, in effect, theory-based evaluation• Use of logic models and results chains strengthens programme

planning and design• Clearly sets out assumptions and external factors• Also strengthens approach to research / analytical planning

Developing in a couple of ways in Scotland: contribution of organisations; contribution of policies and programmes

Alcohol – Partner Contributions to Outcomes

Inputs

Activities

Outputs

Reach

Short-term outcomes

Intermediate outcomes

High level outcomes

NHS

Brief advice

Preventive services

Hazardous and harmful

drinkers

Service uptake &

engagement

Police

Enforcement of drink driving laws

Random breath testing

Drivers

Increased detection rate

Local authorities

Enforcement of planning

controls & licensing laws

Enforcement actions

Licensed trade

Compliance with laws

BehaviourReduced alcohol consumption levelsLess drunkenness; less drink-driving

Improved mental wellbeingReduced inequalities in healthy life expectancy

Reduced inequalities in alcohol-related deaths and hospital admissions

EnvironmentsPhysical: Reduced exposure to alcohol-related hazardsEconomic: Reduced availability/affordability of alcohol

Social: Drunkenness less attractive; sensible drinking the norm

SG, UK govts, EU

Industry regulation

Taxation,displays, promotions,

advertising

Alcohol industry

Increased priceReduced incentives

Scottish Govt

Media campaigns

Sensible drinking

messages

General public - targeted

Understanding risks, attitudes

to drinking

Vol orgs

Detox, Intensive support

Addiction services

Adults with alcohol

problems

Increased sobriety &

stability

A Basic Results Chain

9steve.montague@pmn.net

7. End results 7. What is our impact on ‘ends’?

6. Practice and behavior change 6. Do we influence [behavioural] change?

5. Knowledge, attitude, skill and / or aspirations changes

5. What do people learn? Do we address their needs?

4. Reactions 4. Are clients satisfied? How do people learn about us?

3. Engagement / involvement 3. Who do we reach? Who uses / participates?

2. Activities and outputs 2. What do we offer? How do we deliver?

1. Inputs 1. How much does our program cost? ($, HR etc)

Program (Results) Chain of Events(Theory of Action) Key Questions

Source: Adapted from Claude Bennett 1979. Taken from Michael Quinn Patton, Utilization-Focused Evaluation: The New Century Text, Thousand Oaks, California, 1997, p 235.

Indirect Influence

Direct Influence

Control

WHY?

WHAT?

WHO?

HOW?

Assembling and critiquing the story

Multiple sources of evidence to accommodate; can include:

• Population based statistics• Survey data• Needs assessments• Process and outcome evaluations• Economic analysis and evaluation• Systematic reviews and other evidence synthesis• International policy and analytical work

Baseline and forward look – data /evidence gaps

How do you actually do this in practice?

• To date – few ‘live’ examples of contribution analysis applied in the UK – Health Scotland led on this

• Scottish Government early work: outcome 15, GES population target, interest on outcome 8, outcome 12

• Sessions to develop and discuss logic model (mix of analysts and policy leads)

• Baseline ‘narrative’ and performance story so far

• Analytical planning – what data and analysis will we need in the future?

Some tentative conclusions

• Outcomes focus is an opportunity to undertake longer-term, more strategic analytical work

• Needs input of policy, practice and analytical professionals

• Contribution analysis not a panacea – but a tool to help think and discuss policy and service development differently

• Does help you to plan any research and analysis required in assessing progress

Over to you….

• Are you using this, or components of this, already?

• Can this be applied to outcomes / indicators in Communities?

• Or to larger programmes you are evaluating?