View
2.899
Download
0
Tags:
Embed Size (px)
Citation preview
John YoungHead of Programme, RAPID, ODI
For Assessing the Impact of Policy Engagement
RIMISP-IFAD Learning Event IFAD, Rome 23rd June 2015
Methods for M&E of Policy Advocacy
Overview
• Me and my background
• Why is it important?
• What do you want to evaluate?
• Some ways of thinking (frameworks)
• Some ways of doing it (tools)
• Some conclusions
Image source: LQ-CN7499 World Bank
What to evaluate?
• Do the policies deliver better outcomes?
• Do the policies support improved education?
• Do your programmes influence education policy?
• Does your programme do it well?
Monitoring and Evaluation
Agenda Setting
DecisionMaking
Policy Implementation
Policy Formulation
Policy processes are complex
Civil Society
DonorsCabinet
Parliament
Ministries
Private Sector
The Working Paper
• Trends and issues in M&E of policy engagement.
• Frameworks for understanding policy engagement.
• Methods and tools for M&E of policy engagement.
• Case studies.
Image source: LQ-CN7499 World Bank
Current issues
• What is the purpose of M&E for policy engagement?
• How do you measure success?• Attribution vs Contribution?• How can you assess causality?
Tactics:Inputs, activities, outputs
Outcomes: Changes in behavior, relationships, policy
Impact: Changes in state, wellbeing, environment
Sphere of control
Sphere of influence Sphere of concern
Spheres of influence
The RAPID framework
The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc.
The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc
External Influences Socio-economic and cultural influences, donor policies etc
The links between policyand research communities – networks, relationships, power, competing discourses, trust, knowledge etc.
Eight “policy” outcomes
1. Attitudes of stakeholders.
2. Public opinion.
3. Capacity/engagement of other actors.
4. Change in discussions among policy actors.
5. Improvements in policy processes.
6. Change in policy content.
7. Behaviour change for implementation.
8. Systems for supporting delivery of change.
A systematic approach1. Strategy and direction –are you doing
the right thing?
2. Management –are you doing what you planned to do?
3. Outputs – are the outputs appropriate for the audience?
4. Uptake – are people aware of your work?
5. Outcomes and impacts –are you having any impact?
1. Strategy and direction
2. Management
3. Outputs
4. Uptake
5. Outcomes and impacts
Logical frameworks
Goal Indicator MOV
Purpose Indicator MOV Assumptions/Risks
Output 1 Indicator MOV Assumptions/Risks
Output 2 Indicator MOV
Output 3 Indicator MOV
Output 4 Indicator MOV
√√
Scalar models & rubrics
0. No engagement with SDA.
1. Occasional meetings, no participation
2. Some meetings, poor communication
3. Some meetings, limited inputs
4. Regular meetings, opinion considered.
5. …. and can monitor implementation
6. …. and consulted on proposed plans
7. …. and influence plans
8. …. and documented and shared
9. Commitment to action
10. Policy or practice change.
Impact: stories of change
1. Stories of change:• Situation before• Context• Situation after• What changed and why
http://www.mande.co.uk/docs/MSCGuide.pdf
2. Most significant change (Davies) • Stories of change from
different stakeholders• Systematic analysis of significance.
3. Micro-narratives (Snowden)
Stories of change (DFID 10%)
1. What is the story about?2. Is there any other important
background information?3. What was the existing behaviour?4. What did the RPC do that
contributed to the change?5. What was the behaviour change?6. What factors helped make this
change successful?7. What comes next for the key actors?8. Where can we find more information or
supporting evidence?
Conclusions
• What exactly are you trying to do?
• What do you want to know, why and who for?
• What is your theory of change?
• Are you seeing straight?
• Do the changes have anything to do with you?
• Are you changing?
• “Perfect is the enemy of good1”
1 Voltaire, La Bégueule: “Dans ses écrits, un sage Italien Dit que le mieux est l'ennemi du bien".