Transcript
Page 1: M&E Frameworks How frameworks guide program monitoring and evaluation

M&E Frameworks

How frameworks guide program monitoring and evaluation

Page 2: M&E Frameworks How frameworks guide program monitoring and evaluation

Learning Objectives

Describe function of Conceptual, Logical and Results frameworks

Apply Proximate Determinants Framework to M&E Planning

Use logical frameworks to guide M&E of HIV programs

Apply achievements to results frameworks

Page 3: M&E Frameworks How frameworks guide program monitoring and evaluation

Session Overview Function of frameworks Conceptual Frameworks

Proximate Determinants Logic Frameworks

PMTCT program implementation, GFATM Results Frameworks

PEPFAR GFATM M&E Reporting Framework Activity

Developing a logic framework for an HIV program

Page 4: M&E Frameworks How frameworks guide program monitoring and evaluation

Discussion

Why do we use frameworks in M&E?

Page 5: M&E Frameworks How frameworks guide program monitoring and evaluation

Functions of Frameworks for M&E

Roadmap to program planning, monitoring and evaluation

Delineate clear pathways to program goals & objectives

Define relationships between program inputs, processes, outputs, and outcomes

Describe how program factors interact with external context (environmental factors)

Lead to sound implementation and good M&E plans

Page 6: M&E Frameworks How frameworks guide program monitoring and evaluation

Types of Frameworks - Conceptual

Definition Illustrate relationships among all factors (systemic,

organizational, individual) that influence program operation & successful achievement of goals

Purpose Show how program components will operate to

influence outcomes within wider context Guide identification of indicators Guide impact analysis (causal pathways)

Page 7: M&E Frameworks How frameworks guide program monitoring and evaluation

Conceptual Frameworks

Makes explicit connections among relevant contextual (environmental) factors and your program

Helps clarify the “why” and “how” questions of program operation and design:

• Assumptions that underlie chosen activities

• Factors activities are expected to affect

• Causal linkages leading to achievement of program objectives

Page 8: M&E Frameworks How frameworks guide program monitoring and evaluation

Conceptual Framework Examples:Proximate Determinants

Premise of “proximate determinants” frameworks: Factors leading to health or demographic outcome

operate through an ordered path of determinants Determinants grouped in proximity to outcome

Underlying (socioeconomic) factors farthest away, biological closest

“Proximate” determinants take intermediate position between underlying and biological Proximate determinants are affected by underlying

factors & affect the biological determinants Proximate determinants are what most programs seek to

influence

Page 9: M&E Frameworks How frameworks guide program monitoring and evaluation

Proximate Determinants Example: HIV (STI) Incidence1

Underlying determinants Socioeconomic and socio-cultural context (personal and

community) Health programs environment: VCT, ARV, STD control, condom

promotion, IEC Proximate determinants (interventions)

Partner acquisition, mixing patterns, concurrency, abstinence Condom use, STI present, risky practices, STI treatment Treatment

Biological Determinants Rate of contact between susceptible and infected Efficiency of transmission Duration of activity

Outcomes: HIV incidence/mortality, STI incidence

1 Boerma and Weir, 2005

1Boerma and Weir, 2005

Page 10: M&E Frameworks How frameworks guide program monitoring and evaluation

Proximate Determinants Framework for HIV/STI1

1 Boerma and Weir, 2005

UnderlyingDeterminants

ProximateDeterminants

BiologicalDeterminants

HealthOutcome

Demog.Outcome

ContextSocioculturalSocioeconomicDemographic

InterventionsVCTSTI controlCondom prom.IECBlood safetySafe injections

Partner acquis.Mixing patternsConcurrencyIDU

Condom useOther STIType of sexNeedle safety

ARV TreatmentSTI Treatment

Exposure ofsusceptible toinfected

Efficacy oftransmissionper contact

Duration of infectivity

HIVinfection Mortality

Disease

1Boerma and Weir, 2005

Page 11: M&E Frameworks How frameworks guide program monitoring and evaluation

Ecological Framework:Links between violence against women and HIV1

1 WHO 2010, Addressing violence against women and HIV AIDS: What works?

More risky male partners:

More controlling & violent masculinities

More sexual risk taking

More likely HIV/STIs

Gender & relationship inequality

Rape, child sex abuse &

IPV

Psychological Distress:

Chronic AnxietyDepression

PTSDSubstance use

Reduced protective powers:

More acquiescent femininities

More frequent sexLess condom use

More risky sex:

More partnersConcurrency

Transact. Sex

Sex while intox.

HIV

Rape

Direct transmission

Indirect transmission

Page 12: M&E Frameworks How frameworks guide program monitoring and evaluation

Types of Frameworks -Logic Definition

Pictorial diagram that summarizes essential program information by describing its main elements (inputs, activities, outputs, outcomes) & how they work together to affect a desired health outcome

Purpose Make intended program outcomes explicit Show internal logic of program Reveal assumptions of how program leads to outcomes Guide indicator selection Promote communication among stakeholders Guide impact evaluation

Source: CDC Evaluation Guidance Handbook at: http://www.cdc.gov/hiv/aboutdhap/perb/guidance/chapter4.htm

Page 13: M&E Frameworks How frameworks guide program monitoring and evaluation

Logic Frameworks - Components Problem statement

The “why” of program implementation Statement what puts a population at risk for HIV

Inputs Program resources like staff, curricula, money

Activities What the program does like outreach, training, testing

Outputs Deliverables of program like people trained or tested

Outcomes Program results like change in knowledge, service use

Impact Long term change in health status like decreased HIV incidence

Source: CDC Evaluation Guidance Handbook at: http://www.cdc.gov/hiv/aboutdhap/perb/guidance/chapter4.htm

Page 14: M&E Frameworks How frameworks guide program monitoring and evaluation

Logic Framework Example: PMTCT Program1

Problem Statement HIV rates rising in pregnant women & infants in

Thailand. Transmission risk from mom to child significant during pregnancy & delivery, with additional risk pp via breastfeeding

Inputs Staff, funding from CDC GAP & gov’t, PMTCT

counseling/protocol, HIV test kits, ARV drugs, breast milk substitutes

Activities ANC services include counseling, testing & referral &

referral, distribute ARV prophylaxis, provide infant feeding counseling & support

1CDC M&E Guide, Version 1, Appendix 1

Page 15: M&E Frameworks How frameworks guide program monitoring and evaluation

Logic Framework Example: PMTCT Program, cont’d.1

Outputs (per quarter) 500 women participate in a 2 hr. ANC session 250 of these will get HIV test results All (of 500) who test positive will be referred for ARV prophylaxis, infant

feeding C&S Half of all positive women will receive prophylaxis and infant feeding

counseling and breast milks substitutes Outcomes

Increase in access to HIV testing Increase in knowledge of status Increase in knowledge about prophylaxis Increase in ARV prophylaxis use by pregnant HIV+ Increase in rapid infant weaning/use breast milk substitutes

Impact: ↓ HIV incidence in infants, ↓ morbidity/mortality among people living with HIV/AIDS

1CDC M&E Guide, Version 1, Appendix 1

Page 16: M&E Frameworks How frameworks guide program monitoring and evaluation

Developing a Logic Model1

Input from all levels of stakeholders Program staff, program participants and others to analyze

intended program outcomes

A good problem statement leads to a good logic model Analyze the problem, program resources, desired

outputs, outcomes and impact Design activities to achieve outputs and outcomes Usually expressed in diagram or table

1CDC, ETA Evaluation Briefs, Logic models, No. 2, Dec. 2005

Page 17: M&E Frameworks How frameworks guide program monitoring and evaluation

Types of Frameworks – Results1

USAID funded projects, like a logic framework Definition

A tabular/graphic description of a program’s strategy for achieving a specific strategic objective (SO) through intermediate results, with a narrative supplement.

Purpose Like a logic framework, planning & management tool for programs Specifies out exactly how, through a set of intermediate results, the

program SO will be achieved Provides a basis of consensus for all program stakeholders Conveys the development hypothesis—plausible linkages between

the intermediate results and SO

1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947

Page 18: M&E Frameworks How frameworks guide program monitoring and evaluation

Results Frameworks - Components1

Strategic Objective The ultimate end point of the Results Framework—the goal

of the program “Standard by which the operational unit (performance) is

willing to be judged” Appropriate level of ambition means the SO should be

achievable, clear and measurable Intermediate results

Achievable in the short term, lower level than SO An intermediate step leading to achieving the SO Lower level of impact than the SO; cumulative effect leads to

the SO

1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947

Page 19: M&E Frameworks How frameworks guide program monitoring and evaluation

Results Frameworks - Components1

Clarification of causal linkages Narrative that shows how intermediate results are linked to

the SO

Links can be between results which then leads to SO

Critical Assumptions Set of conditions necessary under which the development

hypothesis (strategy for achieving the SO) will hold true

Not under project/USAID control, but will affect SO/results

1USAID, Performance Monitoring and Evaluation TIPS: Building a Results Framework, PN-ACA-947

Page 20: M&E Frameworks How frameworks guide program monitoring and evaluation

Results Framework Example – PEPFAR funded ART Program

SO: Utilization of ART services

IR-1: Availability of quality services

IR-2: Demand for services

IR-1.1: Increase ART sites

IR-1.2: Supply sites with ARVs

IR-1.3: Training for providers

IR-2.1: Increase knowledge ofART

IR-2.2: Increase referral from VCT

Page 21: M&E Frameworks How frameworks guide program monitoring and evaluation

Developing a Results Framework Set an appropriate strategic objective

Identify intermediate results

Clarify causal linkages between results

Identify critical assumptions

Spend time on the narrative portion so that all stakeholders understand What is trying to be achieved

How to report verifiable achievements

Page 22: M&E Frameworks How frameworks guide program monitoring and evaluation

GFATM M&E Framework1

Key Principles Country ownership, harmonization, consistency, balance between

routine & survey data, avoid duplication of efforts Small set of indicators

Focus on multi-agency M&E Toolkit (http://www.theglobalfund.org/en/performance/monitoring_evaluation/)

12-18 indicators per grant Relies on UNAIDS MERG documents/indicators No “Global Fund” indicators (country ownership) Mix of short- & long-term indicators

Quarterly or 6-monthly for some 15-18months for others, important for next Phase

1The Global Fund Monitoring and Evaluation KEY MESSAGES,http://www.theglobalfund.org/en/files/about/structures/lfa/background/LFATrainingMaterials/ME/ME_Key_Principles.pdf

Page 23: M&E Frameworks How frameworks guide program monitoring and evaluation

GFATM M&E Framework, contd.1

Avoid duplication Use existing in-country indicators, data systems, planned

surveys

Strengthen existing systems and budget for data gaps

Indicator targets need to match grant proposals

Existing tools for performance evaluation M&E Toolkit, M&E Systems Strengthening Tool, Data Quality

Assessment Tool

Tools developed for GFATM grants, but applicable to many contexts, as we’ll see

1The Global Fund Monitoring and Evaluation KEY MESSAGES,http://www.theglobalfund.org/en/files/about/structures/lfa/background/LFATrainingMaterials/ME/ME_Key_Principles.pdf

Page 24: M&E Frameworks How frameworks guide program monitoring and evaluation

New Donor Demands: Gender

PEPFAR, UNAIDS, GFATM, WB require gender integration into programs

Addressing gender in meaningful way = measureable

All agencies have strategies to address gender equality

Many resources out there (we’ll examine some at the end of the week)

Page 25: M&E Frameworks How frameworks guide program monitoring and evaluation

Activity: Developing a Logic Framework

Assemble workshop group-work members Use provided CDC-GAP activity sheet:

“Logic Model Worksheet for Planned Implementation and Outcomes”

Take about an hour to complete sheet Resulting logic models to be presented back (5

minutes for each group) by group representative This draft logic model will guide your group work and

form the basis for indicator selection; you will have the opportunity to revise based on feedback

Page 26: M&E Frameworks How frameworks guide program monitoring and evaluation

MEASURE Evaluation is a MEASURE project funded by the

U.S. Agency for International Development and implemented by

the Carolina Population Center at the University of North Carolina

at Chapel Hill in partnership with Futures Group International,

ICF Macro, John Snow, Inc., Management Sciences for Health,

and Tulane University. Views expressed in this presentation do not

necessarily reflect the views of USAID or the U.S. Government.

MEASURE Evaluation is the USAID Global Health Bureau's

primary vehicle for supporting improvements in monitoring and

evaluation in population, health and nutrition worldwide.


Recommended