Frameworks 1 March 2011. Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Frameworks1 March 2011

Monitoring and Evaluation of HIV/AIDS ProgramsPretoria, South Africa 2011


Session Outline

Introduction to frameworks

Conceptual frameworks

Results Frameworks

Logical Framework

Hands on activity: developing frameworks


-- Why frameworks

Designing M&E Frameworks assists in the development of

Clearly understood program/project goals and measurable, long-term, short-term, and intermediate objectives

Clearly defined relationships between program/project inputs, processes, outputs, and outcomes,

Understanding between program/project activities and the external context (environmental factors)

Sound implementation of programs

Design sound M&E plans


Characteristics of frameworks

All types of M&E frameworks:

Inter-relate components, levels and directions of action

Enable understanding of how

programs influence health outcomes (conceptual framework)

programs should operate (logical framework)

Programs achieve goals (results)

Facilitate consensus building around a common paradigm




Types of frameworks

Different origins

Many types:

Conceptual

Results

Logical & Logic model

Specific types

Results Based Management framework


Learning Objectives: Conceptual Framework At the end of this session, participants will be able to:

Understand why and how conceptual frameworks are useful for understanding programs and planning M&E

Describe the role of conceptual frameworks in program design, program evaluation, and evaluation research.

Describe the components of conceptual frameworks

Design a conceptual framework to be most useful for M&E planning

Design a conceptual framework for an intervention program


Conceptual framework

Research framework


Group work- 1

Why did we have political revolt in Egypt Why do we have poverty in province B Why do we have crime in City Z Why do we have famine in Country X How come we have street abortion in South Africa Why do we have teenage pregnancy in Community B How come Spain won the FIFA soccer world cup How come China has a booming economy How come the people of Thailand are so hospitable How did Uganda manage to control an epidemic


What are concepts


What is happening?


What is happening

World

Society/community

Workplace

hospital




When we conceptualise

Definitions

What is the meaning of the word or term or idea

What is the actual use of the word

What are the boundaries within which a term operates

What are the contrary examples of the word


Concepts

Enable interpretation of a subject

Analyse complex subject matter

Synthesize separate pieces of a subject matter into a unit

Perceive similarities and differences

Enable us to make value judgments

Extend our knowledge


In program design/programming

What do you need to know

Why do have to know


M&E CONCEPTUAL FRAMEWORKS

Conceptual, or “research”, frameworks are diagrams that identify and illustrate the relationships among all relevant systemic, organizational, individual, or other salient factors that may influence program/project operation and the successful achievement of program or project goals.


Purpose of conceptual frameworks

Provides a perspective for understanding program objectives within a complete context of relevant factors in a program’s operating environment

Clarifies analytical assumptions and their implications for program possibilities or limitations on success, as well as measuring and analyzing that degree of success


Purpose of Conceptual Framework:cont.

Makes explicit connections among relevant contextual (environmental) factors and your program

Helps to clarify the “why” and “how” questions of program operation and design: Assumptions that underlie the chosen activities

Factors the activities are expected to affect

Causal linkages leading to achievement of program objectives

Guides identification of appropriate indicators

Guides impact analysis


Conceptual Frameworks

Individual characteristics

Technicalinputs

Program supply Institutionalcapacity

Programsustainability

Healthstatus

HealthypracticesService

utilization


Proximate Determinants Model for HIV/STI (Boerma and Weir)

Underlying Proximate BiologicalHealthOutcome

Demog.Outcome

ContextSocioculturalSocioeconomicProgrammaticInterventionsVCTSTD controlCondom promoteIEC

Partner acquis.Mixing patternsConcurrency

Condom useConcurrent STIRisky sexTreatment

Treatment

Exposure toinfected

Efficacy ofTransmissionPer contact

Duration of activity

HIVincidence

STIincidence

HealthImpacts:MorbidityMortality

Socioeconomic Impacts:Socio-Economic Decline (household and national)-Orphans and Vulnerable Children


Research Concepts –connotations-defines the problem and

constructs for measurement

Operationalise-denotations- variables

Framework allows integration of the program(research) with existing knowledge

( accepted thinking) of the problem.

Concepts are identified and linked to one another


Research cont..

Research starts from this conceptual or theoretical framework

Basis to Formulate our hypothesis

Expresses Assumptions

Testing/Falsification

This is the basis of deductive approach in research


Trying it out!!

Identify the problem ( concepts)

Define drivers of the problem

Define elements of your program( concepts)

Identify factors(concepts) in your environment

identify individual factors(concepts)

Establish relationship between the factors

Simplify and refine diagram

Create and present your framework


summary

To show where program fits into wider context

To clarify assumptions about causal relationships

To show how program components will operate to influence outcomes

To guide identification of indicators

To guide impact analysis (causal pathways)


Activity

Get into your project/program groups

Develop a conceptual framework for your intervention

After 45 minutes, a member of each group will share the framework with all participants

Strategic Framework


What are strategies

What are results




Revisit your conceptual framework

What is your vision?

What is your mission?

Where are your strengths?


M&E STRATEGIC (RESULTS) FRAMEWORKS

Strategic/Results frameworks are diagrams that identify steps, or levels, of results, and illustrate the causal relationships linking all levels of a program’s objectives.


Purposes:

Provides a clarified focus on the causal relationships that connect incremental achievement of results to the comprehensive program impact

Clarifies project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends


Strategic framework

Results framework

Diagrams

Summary of results

They identify and illustrate the causal relationships linking all levels of program strategy.

Objectives to impacts


Results Frameworks

Presents program strategy for achieving specific objective

Usually presented as diagram,

Includes objective and intermediate results

Includes assumptions behind hypothesis (why the program/strategy is expected to work)

Both a planning and management tool


Goals and Objectives

Goal: A broad statement of a desired,

long-term outcome of the program

Objectives: statements of desired, specific, realistic and and

measurable program results

Source: GAP 2003


Terminology !

Higher Level

Goal

Second Level

Strategic Objective

Third Level

Intermediate Result


Results Framework Example – PEPFAR funded ART Program

SO: Utilization of ART services

IR-1: Availability of quality services

IR-2: Demand for services

IR-1.1: Increase ART sites

IR-1.2: Supply sites with ARVs

IR-1.3: Training for providers

IR-2.1: Increase knowledge ofART

IR-2.2: Increase referral from VCT


EXAMPLE: STRATEGIC OR RESULTS FRAMEWORK

SO1: Increased use of VCT or HIV/AIDS preventive measures

IR1: Availability of quality services

IR2: Demand for services

IR1.1: Information and services increased

IR1.2: Practitioners’ skills and knowledge increased

IR1.3: Sustainable effective management

IR2.1: Clients or public knowledge of HIV/AIDS improved


Hierarchy of Program Dimensions

Availability Access Quality

Attitudes

Know ledge

Dem and

Objective

Goal


USAID labellingR e su lts fra m e w o rk

IR 1 .1

IR 1.2 .1 IR 1.2 .2

IR 1 .2

IR 1In te rm ed ia te ob jec tive

IR 2 .1

IR 2.2 .1 IR 2.2 .2

IR 2 .2 IR 2 .3

IR 2in te rm e d ia te o b jec tive

S tra te g ic o b jec tiveS O 1



Results Frameworks: Key Elements

Goal Strategic Objective

Should be appropriate & realistic Balance ambition and accountability

Intermediate Results Smaller goals Integral to achieving SO

Hypothesized cause-effect (based on conceptual theory) linkages Flow from one intermediate result or to many Move “up” in the results framework

Critical assumptions Relative to the achievement of the SO General condition which holds true for SO to be achieved Be realistic!


Results Frameworks: National TB Program

IR1: Increased availabilityof quality services

IR1.1: Services increased

IR1.2: Practitioners’ skillsand knowledge increased

IR1.3: Improved program management

SO1: Increase tuberculosis case detection rate to 70%

IR2: Increased demand for quality services

IR2.1: Customerknowledge of TB improved

Source: MEASURE Evaluation, 2004 (draft)

IR2.2: Social support for TB practices increased


SO 14: Human capacity and social resiliency increased

FIVE-YEAR GOAL: Foundation Established forReducing Famine Vulnerability, Hunger and Poverty

IR 14.1: Use of high impact health, family planning, and nutrition services, products, and practices increased

IR 14.2: HIV/AIDS prevalence reduced and mitigation of the impact of HIV/AIDS increased

IR 14.3: Use of quality primary education services enhanced

IR 14.1.1: Community support for high impact health interventions increased

IR 14.1.3: Quality of key health services improved

IR 14.1.2: Availability of key health services and products improved

IR 14.1.4: Health sector resources and systems improved

IR 14.2.3: Access to care and treatment for people living with HIV and AIDS increased

IR 14.2.4: Care and support for orphans and vulnerable children expanded

IR 14.2.2: Mother-to-child transmission of HIV reduced

IR 14.2.1: Reduced risk behavior

IR 14.3.3: Quality of primary education improved

IR 14.3.4: Equitable primary education services strengthened

IR 14.3.2: Planning, mgmt and monitoring and evaluation for delivery of primary education services strengthened

IR 14.3.1: Community participation in the management and delivery of primary education services strengthened

IR 14.2.5: A more supportive environment for responding to HIV/AIDS

Results Frameworks

Source: USAID/Ethiopia Integrated Strategic Plan, February, 2004 (revised draft)


Results Frameworks: basis for programmatic process

Agreement within operating unit and donor expectations

Selecting appropriate indicators for the M&E system

Use performance information for management decisions

Analyzing and reporting on program results


Building a Results Framework

Group work

Using your conceptual framework as a basis

Discuss and draw a strategic framework for your program

After 45 minutes, a member of each group will share the framework with all participants

Logical frameworks

M&E for HIV/AIDS Programs


LFA

Stakeholder analysis

Problem analysis

Objective analysis

selection of preferred implementation strategy


Logical Frameworks-logic matrix

Project Description Performance Indicators

Means of Verification Assumptions

Goal: The broader development impact to which the project contributes - at a national and sectoral level.

Measures of the extent to which a sustainable contribution to the goal has been made. Used during evaluation.

Sources of information and methods used to collect and report it.

Purpose: The development outcome expected at the end of the project. All components will contribute to this

Conditions at the end of the project indicating that the Purpose has been achieved and that benefits are sustainable. Used for project completion and evaluation.


Assumptions concerning the purpose/goal linkage.

Component Objectives: The expected outcome of producing each component's outputs.

Measures of the extent to which component objectives have been achieved and lead to sustainable benefits. Used during review and evaluation.


Assumptions concerning the component objective/purpose linkage.

Outputs: The direct measurable results (goods and services) of the project which are largely under project management's control

Measures of the quantity and quality of outputs and the timing of their delivery. Used during monitoring and review.


Assumptions concerning the output/component objective linkage.

Activities: The tasks carried out to implement the project and deliver the identified outputs.

Implementation/work program targets. Used during monitoring.


Assumptions concerning the activity/output linkage.


Logical Frameworks-Log frame matrix

Present a standardized summary of the project and its logic.

Purposes: Summarizes what the project intends to do and how

Summarizes key assumptions

Summarizes outputs and outcomes that will be monitored and evaluated

Other terms used: Logframe matrix


M&E LOGIC FRAMEWORKS or Logic model

Logical frameworks are diagrams that identify and illustrate the linear relationships flowing from program inputs, processes, outputs, and outcomes. Inputs or resources affect Processes or activities which produce immediate results or Outputs, ultimately leading to longer term or broader results, or Outcomes.


Purposes:

Provides a streamlined interpretation of planned use of resources and desired ends

Clarifies project/program assumptions about linear relationships between key factors relevant to desired ends


Logic Frameworks - Components

Inputs Program resources like staff, curricula, money

Activities What the program does like outreach, training, testing

Outputs Deliverables of program like people trained or tested

Outcomes Program results like change in knowledge, service use

Impact Long term change in health status like decreased HIV incidence

Source: CDC Evaluation Guidance Handbook at: http://www.cdc.gov/hiv/aboutdhap/perb/guidance/chapter4.htm


Log frame logic model

Inputs Activities Outputs Outcome

IMPACT


Logic Models: Training

INPUT PROCESS OUTPUT OUTCOME IMPACT

Develop clinicaltraining

curriculum

Conducttraining events

Practitionerstrained in new

clinical techniques

Increase in clients served

by (newly) trained

providers

Declining morbidity

levels in target population


INPUT•Human and financial resources to develop and print educational brochure

PROCESS•Distribute brochure to health facilities

•Meet with physicians to promote distribution of brochure

OUTPUT•Brochure distributed to clients of facilities

OUTCOME•Increased customer knowledge of TB transmission and treatment

•Increased demand for quality TB services

;

IMPACT•Decreased TB infection, morbidity and mortality

Logic Model: Portion of model for tuberculosis control relating to increasing demand for quality services


Problem Statement: HIV infection rates continue to rise, underscoring the importance for people toknow their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

Supervision and training for

VCT personnel

Risk behaviors decreased

Counseling and Testing Personnel*

Provide pre-test counseling

Refer HIV+ clients to appropriate

services (PMTCT, care and support,

treatment)

Financial Resources

VCT MIS**

HIV test kits

Referral system for

prevention & Tx services**

People know their HIV status

HIV+ people are referred to

appropriate services

Client records are

available

Clients (HIV+ and -) develop & adhere to personalized

HIV risk-reduction Increase in

care, prevention, and

treatment services for

HIV+, HIV-, and discordant

couples

HIV transmission rates

decreased

HIV incidence decreased

HIV morbidity

& mortality decreased

PROCESSES OUTPUTS OUTCOMES IMPACTSINPUTS

Service reports are produced

Health outcomes of

HIV + improve

VCT protocols,guidelines, and

training documents**

Infrastructure

Demand for Services

Provide HIV testing

Provide post-test counseling

Maintain patient records

Complete reporting

requirements

HIV+ Clients develop & adhere to

personalized HIV care,

support, and treatment plans

Program and services are improved

Continuity of care is

available

Logic ModelsVoluntary Counseling and Testing (VCT)


Frameworks for M&E Planning Purposes:

clarifying assumptions, goals, and interrelationships between factors relevant to the project or program

defining objectives selecting activities defining levels of performance and desired results in terms of

planned activities and realistic, objective impacts Monitoring and evaluation plans incorporate:

program managers’ assumptions and objectives, in a given context

a schematic design displaying the directional linkages between key program elements and/or planned results, and other relevant factors


M&E FRAMEWORKS

Designing an M&E framework assists in determining:

Appropriate program elements to measure

Appropriate indicators and data

Appropriate methodology




Summary of Frameworks

Type of Framework

Brief Description Program Management Basis for Monitoring and Evaluation

Conceptual Interaction of various factors

Determine which factors the program will influence

No. Can help to explain results

Results Logically linked program objectives

Shows the causal relationship between program objectives

Yes – at the objective level

Logic model Logically links inputs, processes, outputs, and outcomes,

Shows the causal relationship between inputs and the objectives

Yes – at all stages of the program from inputs to process to outputs to outcomes/ objectives


References Measure : A trainers guide to the fundamentals of Monitoring and Evaluation

for population, health, and Nutrition Programs. 2002. Carolina Population Centre. Chapel Hill. NC.

AusGuide. The Logical Framework Approach. http://www.ausaid.gov.au/ausguide/ausguidelines/1-1-1.cfm

Bertrand, Jane T., Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N.C.: The EVALUATION Project.

Global AIDS Program. (2003) Monitoring and Evaluation Capacity Building for Program Improvement Field Guide, Version 1. U.S. Centers for Disease Control and Prevention, Atlanta, GA.

Marsh, David. 1999. Results Frameworks & Performance Monitoring. A Refresher by David Marsh (ppt) http://www.childsurvival.com/tools/Marsh/sld001.htm

Tsui Amy. 2004. Frameworks (ppt). Presented at the Bill & Melinda Gates Summer Institute. John Hopkins, Townson. Maryland.

Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill.

Bloom S. 2007 Lecture notes . Asia HIV M&E Slides 2007


Activity

Return to small groups from previous activities

Develop Logical model for one of the activities of your program.

After 30 minutes, a member of each group will share one of the logical frameworks

Documents

Frameworks 1 March 2011. Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011