Using Contribution Analysis to Address Cause-Effect ... · Using Contribution Analysis to Address Cause-Effect Questions: Theory and Concepts John Mayne Advisor on Public Sector Performance

Using Contribution Analysis to Address Cause-Effect

Questions:Theory and Concepts

John MayneAdvisor on Public Sector Performance

[email protected]

2


Outlinethe ideatheories of changecontribution analysiscomplexityconclusions

3


The challengeAttribution for outcomes always a challengeStrong evaluations (such as RCTs) not always available or possibleA credible performance story needs to address attributionComplexity significantly complicates the issueWhat can be done?

4


The ideaBased on the theory of change of the program,Buttressed by evidence validating the theory of change, Reinforced by examination of other influencing factors,Contribution analysis builds a reasonably credible case about the difference the program is making

5


Proving CausalityThe gold standard debate (RCTs et al)AEA and EES: many methods capable of demonstrating scientific rigourMethodological appropriateness for given evaluation questionsCausal analysis: auto mechanic, air crashes, forensic work, doctors

6


Contribution analysis: the theory

There is a postulated theory of changeThe activities of the program were implementedThe theory of change is supported by evidenceOther influencing factors have been assessed

7


outputs(goods and services

produced by the program)

activities(how the program carries

out its work)

intermediate outcomes(the benefits and changes resulting from the outputs)

end outcomes(the final or long-term

consequences)

Examplesnegotiating, consulting, inspecting, drafting legislation

Exampleschecks delivered, advice given, people processed, information provided, reports produced

Examplessatisfied users, jobs found, equitable treatment, illegal entries stopped, better decisions made

Examplesenvironment improved, stronger economy, safer streets, energy saved

Immediate outcomes(the first level effects of the

outputs)

Examplesactions taken by the recipients, or behaviour changes

A results chain

External Factors

8


outputs(goods and services

produced by the program)

activities(how the program carries

out its work)

intermediate outcomes(the benefits and changes resulting from the outputs)

end outcomes(the final or long-term

consequences)

Examplesnegotiating, consulting, inspecting, drafting legislation

Exampleschecks delivered, advice given, people processed, information provided, reports produced

Examplessatisfied users, jobs found, equitable treatment, illegal entries stopped, better decisions made

Examplesenvironment improved, stronger economy, safer streets, energy saved

Immediate outcomes(the first level effects of the

outputs)

Examplesactions taken by the recipients, or behaviour changes

Why will these

immediate outcomes

come about?

�Results chain links

External Factors

9


Theories of changeA results chain with embedded assumptions and risks identifiedAn explanation of why the results chain is expected to work; what has to happen

Reduction in smoking

Anti-smoking campaign

Assumptions: target is reached, message is heard, message is convincing, no other major influences at work

Risks: target not reached, poor message, peer pressure very strong

10


Theory of change analysisNeed to identify which of the links in the results chain have the weakest evidenceSome may be supported by prior researchSome may be well acceptedBut some may be a large leap of faith, or the subject of debateWith limited resources, these contested links are where effort should be focused

11


Steps in Contribution Analysis

1. Set out the attribution problem to be addressed

2. Develop the postulated theory of change3. Gather the existing evidence on the ToC4. Assemble & assess the contribution story5. Seek out additional evidence6. Revise & strengthen the contribution

story7. Develop the complex contribution story

12


1. Set out the attribution problem

Acknowledge the need to address attributionScope the attribution problem– What is really being asked– What level of confidence is needed?

Explore the “expected contribution”What are the other influencing factors?How plausible is a contribution?

13


2. Develop the ToC and Risks to It

Build the postulated results chain and ToC– Identify roles played by other influencing

factors– Identify the risks to the assumptions– Determine how contested the ToC is

14


3. Gather existing evidence

Assess the logical robustness of the ToCGather available evidence on– Results– Assumptions– Other influencing factors

15


4. Assemble and assess the contribution story

Set out the contribution storyAssess its strengths and weaknesses

16


5. Seek out additional evidence

Determine what additional evidence is neededIf needed, refine the ToCGather new evidence

17


Strengthening Techniques

Survey knowledgeable others involved Track program variations and their impacts (time, location, strength)Undertake case studiesConduct a component evaluationIdentify and synthesize relevant research or evaluationUse multiple lines of evidence

18


6. Revise and strengthen the contribution story

Build a more credible contribution storyReassess its strengths and weaknessesRevisit step 5

19


Causality in ComplexityMany factorsMany interventionsNo room for experimentingNot everything is plannedCausality a real mess– How to know? Many theory strands– What to know?

20


7. Develop the complex contribution story

Develop the contribution story for each theory strandDevelop the contribution story for any general theory of change

21


Evaluating in ComplexityGoal should be understanding not determining what works per se (Sanderson, Pawson)CA seeks to understand what is working and whyRobust predictions cannot be made but we can observe the consequences and learnToC will be revised frequentlyWe will learn incrementally; CA can help

22


Some ReferencesSanderson (2000). Evaluating in Complex Policy Systems. Evaluation 6(4):433-454Pawson (2006). Simple Principles for the Evaluation of Complex Programmes. In Public Health Evidence. OxfordSampson (2007). Developing Robust Approaches to Evaluating Social Programmes. Evaluation 13(4): 477-493Eoyang & Berkas (1998). Evaluating Performance in a Complex Adaptive System. http://www.winternet.com/~eoyang/CAS_Abstract.htm

http://www.winternet.com/~eoyang/CAS_Abstract.htm

http://www.winternet.com/~eoyang/CAS_Abstract.htm

23


Levels of contribution analysis

Minimalist contribution analysisContribution analysis of direct influenceContribution analysis of indirect influence

24


Contribution analysisBuilds evidence on

Immediate/intermediate outcomes, the behavioural changesLinks in the results chainOther influencing factors at playOther explanations for observed outcomes

Documents

Using Contribution Analysis to Address Cause-Effect ... · Using Contribution Analysis to Address Cause-Effect Questions: Theory and Concepts John Mayne Advisor on Public Sector Performance