Integrated Assessment ApproachM&E and IE

6 Purposes of Impact Planning, Assessment, Reporting & Learning (IPARL)

• Improve projects

• Strengthen relationships

• Demonstrate impact

• Inform strategy

• Sustain credibility and legitimacy

• Educate society

Integrated Assessment ApproachM&E and IE

• Baseline Data/Evidence

• KPI Scorecards

• Evidence of Change Journals

• Comparative Constituency Feedback (CCF)

• Rapid Advocacy Learning Activity Reports

• Comparative Case Studies

KPIs Scorecards

Evidence of Change Journals

• Constituents’ perceptions of CEPA’s performance and impact

• Quality of relationships within CEPA

• Comparative analysis

• New insight on the campaign’s effects• Information on key relationships• Inform assessments and course correct• Amplify voice and participation• Legitimacy• Advocacy tool

National level Global level

• Partners• Allies• Media

• UNAIDS• PEPFAR• Global Fund• UNITAID• CHAI• Unicef

Relationships Impact

• To what extent has your association with CEPA met your expectations?

• How would you rate the quality of the communications within CEPA?

• How would you rate CEPA’s impact in the field of paediatric HIV/AIDS?

• In which ways is CEPA contributing to the elimination of paediatric HIV/AIDS in your country/globally?

Oct-Dec 2009 Design and implementation of global level GAA/CEPA baseline survey

Oct-Dec 2009 Design of country level CEPA baseline survey

Jan- March 2010 Implementation of the country level baseline survey

Q4 2010 Implementation of global and country level constituency surveys

Q1 2012 Repetition of global and country level constituency surveys

Rapid Advocacy Learning Tools

• Bellwether Methodology

• Policymaker Ratings

• Advocacy Pre-mortem

• Intense Period Debriefs

Rapid Advocacy Learning Tools

Intense Period Debriefs

• Occur shortly after policy windows or intense activity periods• Gathers in-depth and real-time information

• Convening of either a focus group or the conducting of individual interviews with advocacy stakeholders to capture data about advocates’ recent experiences.

• Captures:• The public mood and political context during the policy window;• What happened and how the campaign members responded to eve

nts, especially as related to actions that occurred behind closed door s;

• Perspective on the outcome(s) achieved or not achieved;• How strategies might be adjusted in hindsight.

Intense Period Debriefs

Sample Questions1. What events triggered this intense period? 2. How was the organization’s response determined? Who w

as responsible for that decision? How was that decision co mmunicated to partners and allies?

3. Which elements of the organization’s response worked well ? Which elements could have been improved?

4. What was the outcome of the intense period? Was the resu lt positive or negative?

5. What insights will you take away from this experience that might inform your strategies going forward?

Comparison and Triangulation “A Platinum Standard of Rigor”

For Impact Evaluation

Comparison and Triangulation of:

• Alternate theories of action (casual pathways)• Before and after, counterfactual• Contexts (location, time, scale)• Multiple types of information and data• (Competing) stakeholder perspectives

Rigorous Comparison and Triangulation Advances in Case Study Designs

Case studies still a central “method” and can be made more rigorous with relatively small investments

Case studies essential for analysis of:• context – location, time, scale, scope• descriptive inference: construct validity• causal processes: internal and external validity

(simple, complex, complicated, equifinality, etc..)• multiple results (outputs, outcomes, impacts)• unintended results/unexpected consequences• triangulate qualitative and quantitative data• useful communication and reporting tool

Rigorous Comparison and Triangulation Advances in Case Study Designs

Purposes and types of Case Studies:

• Generating theory of change – “inductive”• Elaborating TOC – “plausibility probes”• Evaluating TOC – “least likely” and “most likely” “tough

tests”, “counterfactual thought experiments”• Innovating theory of change – “deviant case studies” of

exceptional “outliers”

Rigorous Comparison and Triangulation Advances in Case Study Designs

Case Study Methods and Tools:• Process-Tracing• Analytic Induction• Structured-focused comparison• General (and Specific) Elimination• Typological Theorizing• Increasing Observations• Data-bases of Case Studies

… and so many more…