Upload
veronicalazzari
View
844
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Program Monitoring and Program Monitoring and EvaluationEvaluation
Colorado State University,USA 14 September 2002
Milan K.Dinda Ph.D
M I
L A
N -
CS
UM
I L
A N
- C
SU
Why the Need for M&E?
MIL
AN
-CS
U
Why M&E? Why M&E?
Internal uses External uses Asses progress against targets Modify program design and
operation Identify staff needs Informed long range planning Make resource allocation
decisions Identify problems or
bottlenecks Develop a common
understanding of desiredoutcomes
Report to funding agency Respond to questions or
criticisms about the program Promote the program to
potential participants Seek additional and more
targeted funding Improve outreach, public
relations and marketing effort
MIL
AN
-CS
U
Defining the Terms
MIL
AN
-CS
UM
ILA
N-C
SU
What is Monitoring?What is Monitoring?
• A routine processroutine process used to determine the extent to which a program has been effectively implemented at different levels, in time and at what cost
• Basically an internal activity
• Carried out regularly, for example, monthly,quarterly, half-yearly or annually.
MIL
AN
-CS
U
What is Evaluation?What is Evaluation?
• An episodic processepisodic process designed to determine the value or worth of a specific project or program.
• Could be both internal and external
• Links particular outputs or outcome directly to an intervention
MIL
AN
-CS
U
M and E in a NutshellM and E in a Nutshell
Monitoring Monitoring EvaluationEvaluation
Routine, daily assessment of on-going activities and progress
Looks at what is being done
Tracks changes over time in inputs, processes and outputs
Episodic assessment of achievements
Examines what has been achieved
Measures outcomes/ effects and impact cased by interventions
MIL
AN
-CS
U
Three Types of EvaluationThree Types of Evaluation
Process EvaluationProcess Evaluation
• Are activities actually implemented? (Content)
• Does the program reach its intended clientele? (Scope and Coverage)
• Are activities being implemented as planned with necessary inputs? (Timing)
MIL
AN
-CS
U
Three Types of EvaluationThree Types of Evaluation
Outcome EvaluationOutcome Evaluation
• Examines short-term effects (based on program objectives)
• With HIV/AIDS interventions, immediate effects or outcomes are often relatedto behavior changes and underlying changes in knowledge, attitudes and beliefs (e.g., use of condom)
• Usually looks at “before and after situation”
MIL
AN
-CS
UM
ILA
N-C
SU
Three Types of EvaluationThree Types of Evaluation
Impact EvaluationImpact Evaluation
• Assesses the longer-term results of a program • Takes a long time, are expensive and complex • Example: Reduction in the new infection of HIV/AIDS
MIL
AN
-CS
U
Systems Approach to M&E
MIL
AN
-CS
UM
ILA
N-C
SU
Systems Analysis to M&ESystems Analysis to M&E
Input OutputProcess Impact Outcome
MIL
AN
-CS
UM
ILA
N-C
SU
Systems Analysis to M&ESystems Analysis to M&E
Inputs OutputProcess Impact Outcome
InputsInputs
• Resources invested in a program (supplies, people,
• 5 M’s: (Wo)Man, Machines, Money, Methods, Materials
MIL
AN
-CS
UM
ILA
N-C
SU
Systems Analysis to M&ESystems Analysis to M&E
Inputs OutputProcess Impact Outcome
Process Process
• activities carried out to achieve objectives
• shows what is done and how well it is done, for example, how volunteers are recruited and trained to become peer counselors
I C
O
M
P
I C
O
M
P
Systems Analysis to M&ESystems Analysis to M&E
Inputs OutputProcess Impact Outcome
Outputs Outputs
• immediate results of activities achieved (deliverables of the program)
• number of workshops conducted; MSMs reached by BCC materials; STI, VCT care; knowledge of HIV transmission
MIL
AN
-CS
UM
ILA
N-C
SU
Systems Analysis to M&ESystems Analysis to M&E
Inputs OutputProcess Impact Outcome
Outcomes (Effects)Outcomes (Effects)
• changes observed among clients of the program
• changes in behavior or skills (e.g., increased usage of condoms among truck drivers; exchange of clean needles); reduction in STI trend; increase social support
MIL
AN
-CS
UM
ILA
N-C
SU
Systems Analysis to M&ESystems Analysis to M&E
Inputs OutputProcess Impact Outcome
ImpactImpact
• changes in the longer term that occur in the community level as a result of a given program
• usually observed in community level (e.g., reduction in HIV/AIDS related mortality; decreased prevalence; coping capacity of the community)
MIL
AN
-CS
UM
ILA
N-C
SU
Logical Framework and Systems Approach to M&E
MIL
AN
-CS
UM
ILA
N-C
SU
What is a Logical Framework?What is a Logical Framework?
Management tool for strategic planning and project management
Systematically and logically sets out a program’s objectives, outputs and activities.
MIL
AN
-CS
UM
ILA
N-C
SU
What is a Logical Framework?What is a Logical Framework?
Hierarchy of Aims
Goal (Impact):Goal (Impact): the ultimate end of the program to which the specific project will contribute (reduction of prevalence of HIV/AIDS among sex workers in Chiang Mai)
Purpose (Outcome/Effect):Purpose (Outcome/Effect): what the project is expected to achieve once it is completed within the allocated time (provision of counseling and clinic services to sex workers)
Outputs:Outputs: products or deliverables of activities (trained peer counselors; clinical services provided)
Activities (Inputs; Process):Activities (Inputs; Process): actions to be undertaken and the resources available to produce the outputs (upgrading of clinics)
MIL
AN
-CS
UM
ILA
N-C
SU
Logframe and Systems Analysis Logframe and Systems Analysis
MIL
AN
-CS
UM
ILA
N-C
SU
M & E Framework for HIV/AIDS Program
MIL
AN
-CS
UM
ILA
N-C
SU
M&E Framework for HIV/AIDS ProgramM&E Framework for HIV/AIDS Program
OrganizationalResources
ExternalResources
TechnicalCompetence
ManagementProcesses
Multi-sectoralparticipation
Care andSupport
Counseling andPreventiveServices
IEC, BCC andAdvocacy
Socio-political,economic andcultural factors
Prevalence Statusof HIV/AIDS
Behavior andAttitude
Practices
Social/Community
Support, PolicyChanges
Inputs Process Outputs Outcomes Impact
Monitoring Evaluation
MIL
AN
-CS
UM
ILA
N-C
SU
M & E Process
MIL
AN
-CS
UM
ILA
N-C
SU
ProcessProcess
• Define the Purpose of M & E
• Define the Scope of M & E
• Develop Indicators
• Determine Data Sources and Methods
• Develop an Implementation Plan
• Disseminate and Utilize Results
MIL
AN
-CS
UM
ILA
N-C
SU
Define the Purpose of M & EDefine the Purpose of M & E
Three W’sThree W’s What: What interventions to be monitored and evaluated?
Why: Is it a routine process or a short-term process for decision making? Is it to examine results at output, outcome or impact level?
Who: Who will use the results (e.g., NGO, program manager, CBD staff , community, donor)
MIL
AN
-CS
UM
ILA
N-C
SU
Define the Scope of M & EDefine the Scope of M & E
Important: Review first your program goals and Important: Review first your program goals and objectivesobjectives
• Intervention Level (Coverage): Program level, community level
Causality Links: Cause and Effect of what?
Timetable: How long will the M & E be undertaken?
MIL
AN
-CS
UM
ILA
N-C
SU
Develop Indicators Develop Indicators
• Marker of performance - to track progress in the performance of the HIV/AIDS interventions
• Operational measures - “if you can measure it you can manage it”
• Key Standards: QualityQuality (binary - yes/no measure, categorical); QuantityQuantity (rate, absolute number, average, proportion), CostCost and TimingTiming
MIL
AN
-CS
UM
ILA
N-C
SU
Develop Indicators Develop Indicators
Characteristics of Good A Good Indicator Valid: It is applicable to the interventions being measured (percentage of condom usage)
Reliable: Bias-free and consistent
Specific: Identifies specific areas to be examined (users during the last 3 months in District 1)
Sensitive: Reflects changes in the situation (Increase in the number of condom users)
Operational: It is measurable or quantifiable (number, percentage)
MIL
AN
-CS
UM
ILA
N-C
SU
Develop IndicatorsDevelop Indicators
Component IndicatorsInputs Number of personnel assigned to the program (Quantity)
Qualified personnel assigned (Quality) The total cost of program. Budget for counseling component (Cost) Timeline of the program (Timing)
Process Number of peer educators trained (Quantity) Number of peer leaders trained who passed the test (Quality) Cost per peer training (Cost) Period of training days (Timing)
Outputs Increase in the number of sex workers reached (Quantity) Percent of sex workers who understood HIV/AIDS risks (Quality) Added cost of increasing the numbers of peer educators (Cost) Frequency of follow-ups undertaken (Timing)
Outcomes Increase in condom use rate among sex workers (Quantity) Improvement in negotiation skills of sex workers (Quality) Costs associated with the increase in condom usage (Cost) Time required to achieve improvement in condom usage (Timing)
Impact Reduction in HIV/AIDS rate among sex workers (Quantity) Ability of sex workers to collectively negotiate for occupational safety
(Quality) Cost of achieving reduction in HIV/AIDS prevalence (Cost) Time required to achieve reduction in HIV/AIDS prevalence (Timing)
MIL
AN
-CS
UM
ILA
N-C
SU
Determine Data Sources/Methods Determine Data Sources/Methods
• Program-based: Direct Interventions - routine program statistics, client surveys, FGDs, facility-based surveys, in-depth interviews, provider survey, program documents and reports, direct observations
• Community-based: Indirect Interventions - community-based surveys, FGDs, qualitative surveys, population census
• Context-based - parliamentary records, public libraries, media reports
MIL
AN
-CS
UM
ILA
N-C
SU
Develop an Implementation Plan Develop an Implementation Plan
Tip: Review your Logframe (if you have)
• Who is doing what?
• When is it being done?
• How much time does it take?
• How much does it cost?
MIL
AN
-CS
UM
ILA
N-C
SU
Disseminate and Utilize Results Disseminate and Utilize Results
• Utilize in Decision Making
• Discuss with program staff
• Advocacy
• Share to donors and other interested parties
• Share to stakeholders and communities
MIL
AN
-CS
UM
ILA
N-C
SU
Principles of M & E
MIL
AN
-CS
UM
ILA
N-C
SU
Principles (BLEND) Principles (BLEND)
• Bias-free • Logical and Clear
• Educational
• Non-threatening
• Diversity of views
MIL
AN
-CS
UM
ILA
N-C
SU
Pitfalls to Avoid
MIL
AN
-CS
UM
ILA
N-C
SU
Avoid Pitfalls Avoid Pitfalls
• Too scientific • Perfecting the design
• Ignoring Stories
• Probing only Success
• Getting Done With It