Monitoring and Evaluation:Frameworks

Learning Objectives

At the end of the session, participants will be able to: • Identify and differentiate between conceptual

frameworks, results and logical frameworks, and logic models

• Design goals and objectives for specific intervention programs

• Design frameworks for specific intervention programs• Discuss how frameworks are used for M&E planning

Module Outline

• Introduction to frameworks– Conceptual Frameworks– Goals and objectives– Results and Logical Frameworks– Logic models

• Exercises developing frameworks• Interface between frameworks and M&E planning• Work on group project

Why are Frameworks Useful?

Designing frameworks assist developing:• Clearly understood program/project goals and measurable,

long-term, short-term, and intermediate objectives• Clearly defined relationships between program/project

inputs, processes, outputs, and outcomes, and between program/project activities and the external context (environmental factors)

• Sound implementation and M&E plans

Types of Frameworks

• Conceptual frameworks• Results frameworks• Logical frameworks• Logic models

Conceptual Frameworks(a.k.a., research or theoretical frameworks)

Diagram that identifies and illustrates the relationships between all relevant systemic, organizational, individual, or other salient factors that may influenceprogram/project operation and the successful achievement of program or projectgoals.

M&E Purpose:• To show where program fits into wider context• To clarify assumptions about causal relationships• To show how program components will operate to influence outcomes• To guide identification of indicators• To guide impact analysis (causal pathways)

Similar frameworks:• Proximate Determinants

Conceptual Frameworks

Individual characteristics

Technicalinputs

Program supply Institutionalcapacity

Programsustainability

Healthstatus

HealthypracticesService

utilization

Conceptual Frameworks

Source: MEASURE Evaluation 2004 (draft)

National Tuberculosis (TB) Program

TBmortality

TB infectionExternal Factors• Demographic: (age, gender)• Socio-economic: (education, occupation)• Environmental: (war, weather)

Program Factors• Political commitment• Donor involvement• National TB program

Health Systems (DOTS)• Availability• Access• Utilization•Case Detection

TB Knowledge• Stigma• Treatable• Symptoms

Co-morbidity• HIV• Malnutrition• Alcoholism•Diabetes

TB Morbidity• Prevalence• Incidence• HIV co-infection• MDR-TB

Conceptual Frameworks

Source: UNICEF, State of the World’s Children, 1998

Immediate causes

Underlying causes at

household/ family level

Basic causes at societal

level

Outcomes

Insufficientaccess to food

Potential Resources: environment, technology, people

Quantity & quality of actual resources - human, economic and organizational – and the way they

are controlled

Inadequatematernal & child

care practices

Poor water/sanitation & inadequate health

services

Child malnutrition, death and disability

Inadequate Diseasedietary intake

Causes of malnutrition in society

Activity

• Get into groups of 4 or 5• Develop conceptual framework for utilization

of family planning/maternal and child health services– Identify factors that influence health service

utilization– Arrange factors into a conceptual framework

• After 30 minutes, a member of each group will share the framework with all participants

Goals and Objectives

• Goal: a broad statement of a desired, long-term outcome of the program

• Objectives: statements of desired, specific, realisticand measurable program results SMART

• Specific: identifies concrete events or actions that will take place• Measurable: quantifies the amount of resources, activity, or change

to be expended and achieved• Appropriate: logically relates to the overall problem statement and

desired effects of the program • Realistic: Provides a realistic dimension that can be achieved

with the available resources and plans for implementation• Time-based: specifies a time within which the objective

will be achieved

Source: GAP 2003

Formulating Program Objectives

To [action]

The [specify what knowledge, attitudes, skills, behaviors]

Among [specific population or segment]

From – to [from baseline to desired level]

or by [ x percent ]

or to [specific level]

By [time frame]

Goals and ObjectivesNational AIDS Committee (NAC) 2005 Strategic Plan• Goal: Prevent the spread of HIV epidemic and

minimise its impact on nation by 2009• Objective 1: Reduction of HIV prevalence in nation• Objective 2: Improved health & quality of life of

people infected & affected by HIV/AIDS

• Objective 3: Strengthened capacity of NAC & stakeholders to respond to the HIV/AIDS epidemic at all levels through improved research, M&E and improved management & coordination

Goals and Objectives: Activity

Source: UNICEF, State of the World’s Children, 1998

Immediate causes

Underlying causes at

household/ family level

Basic causes at societal

level

Outcomes

Insufficientaccess to food

Potential Resources: environment, technology, people

Quantity & quality of actual resources - human, economic and organizational – and the way they

are controlled

Inadequatematernal & child

care practices

Poor water/sanitation & inadequate health

services

Child malnutrition, death and disability

Inadequate Diseasedietary intake

Results Frameworks

Diagrams that identify steps or levels of results and illustrate the causal relationships linking all levels of a program’s objectives.

Purposes:• Provides a clarified focus on the causal relationships that

connect incremental achievement of results to the comprehensive program impact

• Clarifies project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends

Other terms used:• Strategic frameworks

Results Frameworks

Goal: Improved Health Statusand/or Decreased Fertility

Objective: Improved Use of Health/FP Servicesand/or Appropriate Practices

IR1: Access/Availability IR2: Quality IR3: Sustainability IR4: Demand

IR1.1:Commodities/Facilities

IR1.2: Equity

IR2.2: Provider Performance

IR2.3: Training/Supervision

IR2.4: Information System

IR3.1: Policy

IR3.2: Health Care Finance

IR3.3: Private Sector

IR4.1: Attitude

IR4.2: Knowledge

IR4.3: Community Support

Source: David Marsh, 1999

Results Frameworks

IR1: Increased availabilityof quality services

IR1.1: Services increased

IR1.2: Practitioners’ skillsand knowledge increased

IR1.3: Improved program management

SO1: Increase tuberculosis case detection rate to 70%

IR2: Increased demand for quality services

IR2.1: Customerknowledge of TB improved

Tuberculosis (TB) Control Programs

Source: MEASURE Evaluation, 2004 (draft)

IR2.2: Social support for TB practices increased

Results FrameworksDonor/USAID Reproductive Health Program

IR1 Strengthened sustainability of FP/RH Program

IR1.1 Improved policy environment for the provision of FP/RH services in the public and private sectors

IR1.2 Strengthened NGO advocacy for FP program

SO1: Increased Utilization of Family Planning/Reproductive Health Services

IR2 Expansion of high quality FP/RH services in the public and private sectors

IR2.1 Increased availability of postpartum and postabortion FP services

Source: USAID/Turkey Performance Monitoring Plan, 1998-2001

IR2.2 Increased accurate knowledgeof clients about modern methods

and FP services

IR2.3 Improved job performance of health providers, trainers, and administrators

SO 14: Human capacity and social resiliency increased

FIVE-YEAR GOAL: Foundation Established forReducing Famine Vulnerability, Hunger and Poverty

IR 14.1: Use of high impact health, family planning, and nutrition services, products, and practices increased

IR 14.2: HIV/AIDS prevalence reduced and mitigation of the impact of HIV/AIDS increased

IR 14.3: Use of quality primary education services enhanced

IR 14.1.1: Community support for high impact health interventions increased

IR 14.1.3: Quality of key health services improved

IR 14.1.2: Availability of key health services and products improved

IR 14.1.4: Health sector resources and systems improved

IR 14.2.3: Access to care and treatment for people living with HIV and AIDS increased

IR 14.2.4: Care and support for orphans and vulnerable children expanded

IR 14.2.2: Mother-to-child transmission of HIV reduced

IR 14.2.1: Reduced risk behavior

IR 14.3.3: Quality of primary education improved

IR 14.3.4: Equitable primary education services strengthened

IR 14.3.2: Planning, mgmt and monitoring and evaluation for delivery of primary education services strengthened

IR 14.3.1: Community participation in the management and delivery of primary education services strengthened

IR 14.2.5: A more supportive environment for responding to HIV/AIDS

Results Frameworks

Source: USAID/Ethiopia Integrated Strategic Plan, February, 2004 (revised draft)

Logical Frameworks

Present a standardized summary of the project and its logic. Purposes:• Summarizes what the project intends to do and how• Summarizes key assumptions• Summarizes outputs and outcomes that will be monitored

and evaluated

Other terms used:• Logframe matrix

Logical FrameworksProject Description Performance Indicators Means of

VerificationAssumptions

Goal: The broader development impact to which the project contributes - at a national and sectoral level.

Measures of the extent to which a sustainable contribution to the goal has been made. Used during evaluation.

Sources of information and methods used to collect and report it.

Purpose: The development outcome expected at the end of the project. All components will contribute to this

Conditions at the end of the project indicating that the Purpose has been achieved and that benefits are sustainable. Used for project completion and evaluation.

Sources of information and methods used to collect and report it.

Assumptions concerning the purpose/goal linkage.

Component Objectives: The expected outcome of producing each component's outputs.

Measures of the extent to which component objectives have been achieved and lead to sustainable benefits. Used during review and evaluation.

Sources of information and methods used to collect and report it.

Assumptions concerning the component objective/purpose linkage.

Outputs: The direct measurable results (goods and services) of the project which are largely under project management's control

Measures of the quantity and quality of outputs and the timing of their delivery. Used during monitoring and review.

Sources of information and methods used to collect and report it.

Assumptions concerning the output/component objective linkage.

Activities: The tasks carried out to implement the project and deliver the identified outputs.

Implementation/work program targets. Used during monitoring.

Sources of information and methods used to collect and report it.

Assumptions concerning the activity/output linkage.

Logical Frameworks

GOAL

A. Reduced burden of TB to reach global average levels

B. Further development of multi-drug resistant TB (MDR-TB) prevented

 

PERFORMANCE INDICATORS

A. Notification rate

B-1. Treatment outcome

B-2. Prevalence of Multi-drug Resistance in “new” and previously treated TB patients

MEANS OF VERIFICATION

A. Annual notification reports (surveillance)

B-1. Annual reports on outcome of treatment (cohort analysis)

B-2. Periodic reports on surveillance of anti-TB drug resistance

 

 ASSUMPTIONS

 

- A dual HIV/TB epidemic causing increase in TB incidence does not occur

- Control of private practitioner and pharmaceutical sectors to prevent MDR

- Prevalence of resistance to second line anti-TB drugs low enough at the outset so as not to seriously compromise treatment success ratio

Taskforce on Communicable Disease Controlin the Region X: Tuberculosis

Logical Frameworks

PURPOSE

 

[Implementing] cost-effective measures for the prevention and control of TB operating within civil and penitentiary health services in the Task Force area

PERFORMANCEINDICATORS  

1. Coverage of TB programmes in line with international recommendations.

2. Proportion of patients defaulting out of patients treated.

3. Proportion of previously treated cases among all cases.

4. Proportion of patients on ambulatory treatment out of all patients treated.

MEANS OF

VERIFICATION

1. Annual reports

2. Annual reports

3. National / local annual notification reports (surveillance)

4. Annual record reviews during site visits (consecutive series of patients)

ASSUMPTIONS

 

- Stable political situation, sustained political commitment and financing

- Sufficient numbers of competent health care personnel in the government sector

Taskforce on Communicable Disease Controlin the Barents and Baltic Sea Regions: Tuberculosis

Logical Frameworks

OUTPUTS

 

8. Measures to increase awareness of TB and its treatment among all members of the community developed and tested

PERFORMANCEINDICATORS   8.1. Number of

pamphlets / posters printed and distributed annually

8.2. Awareness of TB among target groups

MEANS OF

VERIFICATION

8.1. Material produced / distributed

8.2. KAP or other surveys (before/after)

ASSUMPTIONS

 

1. Relevant persons motivated to participate

2. Professional interest, sufficient financing

3. Target groups interested in (their) health and able to participate

 ACTIVITIES

 

8.1 Identify groups at risk for TB

8.2 Develop advocacy material suitable for all target groups (not only risk groups)

8.3 Organize health education directed at all target groups

8.4 Involve the mass media

 INPUTS 

MEANS OF

VERIFICATION

 

Financial management reports

ASSUMPTIONS

Taskforce on Communicable Disease Controlin the Barents and Baltic Sea Regions: Tuberculosis(only one output and related activities shown here)

Activity

• Return to small groups from previous activity

• Develop goals and objectives for a program aimed at increasing utilization of FP/MCH health services

• Develop a results or logical framework for this program

• In 45 minutes, a member of each group will share the goals, objectives and framework

Logic ModelsDiagrams that identify and illustrate the linear relationships flowingfrom program inputs, processes, outputs, and outcomes. Inputs orresources affect Processes or activities which produce immediateresults or Outputs, ultimately leading to longer term or broaderresults, or Outcomes. Purposes:• Provides a streamlined interpretation of planned use of

resources and desired ends• Clarifies project/program assumptions about linear

relationships between key factors relevant to desired endsOther terms used:• M&E Frameworks, Logical Frameworks

Logic Models: Training

INPUT PROCESS OUTPUT OUTCOME IMPACT

Develop clinicaltraining

curriculum

Conducttraining events

Practitionerstrained in new

clinical techniques

Increase in clients served

by (newly) trained

providers

Declining morbidity

levels in target population

Problem Statement: HIV infection rates continue to rise, underscoring the importance for people toknow their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

Supervision and training for

VCT personnel

Risk behaviors decreased

Counseling and Testing Personnel*

Provide pre-test counseling

Refer HIV+ clients to appropriate

services (PMTCT, care and support,

treatment)

Financial Resources

VCT MIS**

HIV test kits

Referral system for

prevention & Tx services**

People know their HIV status

HIV+ people are referred to appropriate

services

Client records are

available

Clients (HIV+ and -) develop & adhere to personalized

HIV risk-reduction Increase in

care, prevention, and

treatment services for

HIV+, HIV-, and discordant

couples

HIV transmission rates

decreased

HIV incidence decreased

HIV morbidity

& mortality decreased

PROCESSES OUTPUTS OUTCOMES IMPACTSINPUTS

Service reports are produced

Health outcomes of

HIV + improve

VCT protocols,guidelines, and

training documents**

Infrastructure

Demand for Services

Provide HIV testing

Provide post-test counseling

Maintain patient records

Complete reporting

requirements

HIV+ Clients develop & adhere to

personalized HIV care,

support, and treatment plans

Program and services are improved

Continuity of care is

available

Logic ModelsVoluntary Counseling and Testing (VCT)

Results FrameworkTuberculosis (TB) Control Programs

IR1: Increased availabilityof quality services

IR1.1: Services increased

IR1.2: Practitioners’ skillsand knowledge increased

IR1.3: Improved program management

SO1: Increase tuberculosis case detection rate to 70%

IR2: Increased demand for quality services

IR2.1: Customerknowledge of TB improved

Source: MEASURE Evaluation, 2004 (draft)

IR2.2: Social support for TB practices increased

INPUT•Human and financial resources to develop and print educational brochure

PROCESS•Distribute brochure to health facilities

•Meet with physicians to promote distribution of brochure

OUTPUT•Brochure distributed to clients of facilities

OUTCOME•Increased customer knowledge of TB transmission and treatment

•Increased demand for quality TB services

Logic Model

IMPACT•Decreased TB infection, morbidity and mortality

Portion of model for tuberculosis control relating to increasing demand for quality services

Activity

• Return to small groups from previous activities

• Develop logic models for one of the activities of your program to increase utilization of FP/MCH health services

• After 20 minutes, a member of each group will share one of the logical frameworks

Frameworks for M&E Planning• Purposes:

– clarifying assumptions, goals, and interrelationships between factors relevant to the project or program

– defining objectives– selecting activities– defining levels of performance and desired results in terms

of planned activities and realistic, objective impacts• Monitoring and evaluation plans incorporate:

– program managers’ assumptions and objectives, in a given context

– a schematic design displaying the directional linkages between key program elements and/or planned results, and other relevant factors

Summary of Frameworks

Type of Framework

Brief Description Program Management

Basis for Monitoring and Evaluation

Conceptual Interaction of various factors

Determine which factors the program will influence

No. Can help to explain results

Results Logically linked program objectives

Shows the causal relationship between program objectives

Yes – at the objective level

Logical Logically linked program objectives, outputs, and activities

Shows the causal relationship between activities and objectives

Yes – at the output and objective level

Logic model Logically links inputs, processes, outputs, and outcomes,

Shows the causal relationship between inputs and the objectives

Yes – at all stages of the program from inputs to process to outputs to outcomes/ objectives

References• AusGuide. The Logical Framework Approach.

http://www.ausaid.gov.au/ausguide/ausguidelines/1-1-1.cfm• Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996.

Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project.

• Global AIDS Program. (2003) Monitoring and Evaluation Capacity Building for Program Improvement Field Guide, Version 1. U.S. Centers for Disease Control and Prevention, Atlanta, GA.

• Marsh, David. 1999. Results Frameworks & Performance Monitoring. A Refresher by David Marsh (ppt) http://www.childsurvival.com/tools/Marsh/sld001.htm

• Tsui, Amy. 1998. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill.

• Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill.

• UNICEF. 1998. State of the World’s Children.

Group Projects

• Form groups for your group projects

• For your project, develop– Goals and objectives– Conceptual framework– One other framework: results, logical or logic model