Foundations in Monitoring and Evaluation

Niamh Barry Niamh.Barry85@gmail.com

February 2010

Session outline

1. What is M&E?

2. Why do M&E?

3. What happens without M&E?

4. Tools and Approaches to M&E

5. Logical frameworks

6. Performance indicators

7. Practical work

8. Data collection and MOV’s

9. Reporting and reviewing

10. Documentation

What is M&E?

• Monitoring – routine & regular collection, analysis & use of information to track progress towards goals

• Evaluation: assessment of the extent to which a project is achieving or has achieved its stated goals

Or• Monitoring: Improvement and development of the community.• Evaluation: A process design to show the relationship

What's the difference? – Timing– Analysis

What do we M&E?• Inputs• Process • Progress towards goals • Impact of actions against stated goals

What M&E is can not do• M&E ‘s fundamental tenet is to inform judgments about project

performance

• Measure things that are immeasurable! i.e. if all your activities do not have a goal then how can progress be measured???

• The key is in program design if your program design is weak and inconsistent then you will not have a quality M&E framework

• First question should look at your program ‘is this the best way to achieve our goals’ then you can answer the question of how to best M&E the program

• M&E should always be built in – it must work with the program design

 

Why do M&E?

• M&E of development activities provides organizations with a better means for learning from past experience, improving service delivery, planning and allocating resources, and demonstrating results as part of accountability to donors & partners.

• Within the development community there is a strong focus on results— this helps explain the growing interest in M&E.

Why do M&E, when we should focus on implementation?

“If your not keeping score, your only practicing”

What happens without M&E?

• If you do not monitor progress, you do not know if you can

or are succeeding

• If you do not monitor progress, you can never recognize

program failures OR success

• If you do not evaluate Impact, you do not know if you have

succeed or failed

• You prevent learning and sharing

• If you never M&E you may repeat programs that have no

impact and are wasting organizational and target

participants time, resources and donor funds.

Tools and Approaches to M&E

• Various tools and methods exist to begin to help us M&E, the choice depends what is being being M&E (ed)

Progress/impact indicatorsThe logical framework approach

Formal surveys – BaselinesNeeds assessments

Participatory methods (FGDs)Cost-benefit and cost-effectiveness analysis

Impact evaluation

• Some of these tools & approaches are complementary; some are substitutes. Some have broad applicability, while others are quite narrow in their uses.

QUESTIONS?

PERFORMANCE INDICATORS AND LOGICAL FRAMEWORKS

What do we use?

A combination of logical framework and performance indicators is the most

commonly used in long term programming and planning

LOGICAL FRAMEWORKS

Causal Pathway (same but with out indicators)

Causal Pathways

• This is the initial step in creating the logical framework

• It is a program planning tool that ensure that program

activities will cause the desired impact

• This is a very important step – all inputs must be enough to

produce activities and these activities must eventually

produce the goal

• It’s logic is based on cause and effect or the ‘theory of

change’ but you must be careful to think of challenges

during this exercise i.e. you can not assume condom

distribution will lead to condom use.

Simple Logical FrameworkGoal Indicators MoVs Assumptions

Objectives

Outcome

Output

Activities Inputs

Terminology• Input-resources needed in the project

• Activities- process of using inputs to achieve results • Output –the results of the completed project activities • Outcome/objectives-takes place when the project targets uses

outputs as anticipated

• Impact/goal-long term

change/effectiveness of the project

• Many other terms are used-but effectively mean the same thing

…goals-objectives-results-effects-purpose…the point is to know when to use them in the process

Ou

tsid

e p

roje

ct

con

trol

M&E Components

The Soup Analogy HIV Program

Inputs Chicken, vegetables, Broth, SpicesPot, Stove

Condoms, Testing Kits, Staff time, Transport, Funding

Activities Chopping vegetables, cooking chicken, boiling and stirring

TrainingDelivery of service

Outputs Soup has been made VCT availableCondoms availableKnowledge on transmissionTrained staff

Outcomes Appeasement of hungerSatisfaction of eating soup

Increase in condom useIncrease in uptake of VCT Changes in sexual behavior Decrease in STI trends

Goal Improved nutritional status of person eating soup

Decrease in HIV transmission, morbidity and mortality Change in social normsEconomic impact

Important Notes on Log Frames

• You must know what you want to set out to achieve before doing the log frame, is your programme:

• Causal: strategies aimed at directly changing a situation

• Persuasive: strategies aimed at influencing a situation or thinking about a situation

• Supportive: strategies aimed at influencing the environment with which a focal problem is situated

• If you do not know exactly what you want to do then you can’t realistically expect it to be M&E(ed)

Important Notes on Log Frames

Remember that change happens of a period of time (Theory of Change)

That is, how a particular project is anticipated to contribute to social change through time.

Often and incorrectly the vertical logic is used to systematically disaggregate the problem – without consideration of time

Right and Wrong Approaches

Goal :a wallObjectives :the bricks that make up the wallOutputs :the sand and cement that make up the bricksActivities :the molecules that makes up the grains of sand and cement

Goal :significant and lasting changes anticipated in the lives of the ultimate beneficiaries (e.g. households with improved livelihoods)

Objectives: Change to action – peer educators using knowledge and increased capacity

Outcomes :changes in knowledge/attitude/practice influenced among intermediary stakeholders (e.g. Peer educators with better capacity)

Outputs :deliverables of the project implementing team aimed to initiate the change process (e.g. training of peer educators)

Activities: Trainings, follow ups

QUESTIONS?

PERFORMANCE INDICATORS

Performance Indicators

• Performance indicators are measures of inputs, processes, outputs,

outcomes, and impacts for development projects, programs, or

strategies.

• Only beneficial when supported with sound data collection—perhaps

involving formal surveys—analysis and reporting,

• Indicators enable us to track progress, demonstrate results, and take

corrective action to improve service delivery.

• Participation of key stakeholders in defining indicators is important

because they are then more likely to understand and use indicators for

management decision-making.

Developing indicators

• Working with indicators is not difficult

• Indicators are no more than criteria that will help us monitor and

evaluate our project

• To develop them, we have to ask ourselves:

What aspect will tell us whether or not something went well?

Whether we achieved our objectives or not?

Whether we followed our plans?

Example of Performance Indicators

• A program has the goal to increase community knowledge of HIV prevention and access to HIV testing treatment and care through targeted health talks

• What could the performance indicators of this programme be?

• Number of community members reached • Improvement in community members knowledge on HIV

• Improvement in number of people testing• Improvement in the number of people receiving treatment

• Increase in condom use• Reduction in HIV transmission

Important Note on Indicators

• When we talk about any of the below type of indicators:

Increase Decrease

Improvement Change

Etc• We can only measure these if we have a

BASELINE to compare it with• A baseline is information on whatever we wish to

impact prior to our program intervention.

Important note on Indicators

Keep them SMART

S – Specific M - MeasurableA – Achievable R – Realistic T – Timely

Resources to Help on Indicator Development

QUESTIONS?

PRACTICAL WORK Devising your own Logical frame work

Case study

• Country XX has one of the highest burdens of TB worldwide. XXX has quite a number of challenges that correspond to this high burden of TB:

Limited clinical infrastructure Poorly trained clinic staff Low health seeking behavior in the community Stigma regarding TB Low knowledge of TB Low adherence/Lost to follow up for those on

medications

• An TB NGO establishes itself in country XX and wants to decrease the incidence of TB. Of course to do this it most address all of the challenges.

• The NGO’s overall Goal is the decrease the incidence of TB mortality and morbidity in country XX

• Design a logical framework with activities, Outputs, Objectives and the stated goal.

• Start with your goal and then work with activities up!

• Draft a number of indicators for each of these steps.

Key Questions to Consider

• How will you track or measure the success of this project?

• What key indicators will allow to measure results of the work?

• What are some sources and/or means of data collection for these indicators?

• What challenges do you anticipate? What assumptions do you have?

DATA COLLECTION AND MOV’S

Means of Verification (MOV)

• For every indicator you create you must be able to verify it.• That is you must be able to collect data to prove that it is

true.

ExampleIndicator: Decrease in the number of TB patients lost to

follow upMOV: National TB registersIndicator: Increase in TB/HIV knowledge among clinic staffMOV: TB/HIV Knowledge questionnaire

• Essentially these are the data collection tools you will use to monitor progress and evaluate impact .

Data collection Tools

• These are integral to any M&E Framework with them you can not

monitor progress or evaluate impact.

• You must ensure all data collection tools capture the indicators you

have established

• When designing data collection tools – Less is more

• Do not ask for unnecessary information

• Do ensure the tool is user friendly

• Do pilot test the data collection tool (if possible) before you finalise

it

• Do design the tools as you are also designing the indicators – this

will tell you if your indicator is SMART

Important Notes

• Does your data collection tool also need to collect baseline information?

• Does each indicator require a new data collection tool? Are there MOVs such as

National Registers, KAP surveys already available with information on your target

population.

• Remember: data collection tools are anything that collects information on your

indicators

Attendance listsTraining evaluationsNational registers

QuestionnairesMonthly reports

Clinic formsSite assessments

Etc Etc Etc

QUESTIONS?

REPORTING AND REVIEWING

Reporting

• One of the key reasons for implementing an M&E

framework is because of reporting requirements

• Most funding agencies require this annually

• Additionally the organization should have its own

reporting structure, not only to managers but to

the beneficiaries and all stakeholders

Reporting ctd

• Monthly reports on progress are recommended to

ensure that the programme is progressing without

challenges and to review progress made against

targets

• Monthly reports are more desirable as they will

detect any challenges early

• Additionally data collected should be entered into a

program data base

Reviewing

• All programs need an internal review and replanning process at least yearly.

• Recommended that this take place as a prelude and guide to implementing new work plans

• Reviewing should focus on program’s overall achievements, challenges and emerging issues, based on information obtained through the M&E processes described in the preceding slides.

• Through this process, work plans are reviewed• The process should include Project partners and

beneficiaries

DOCUMENTATION

Documentation

• Information provided by a strong M&E framework is invaluable

• Organizations should always documenting: good practices, success

stories and case studies.

• Case studies will facilitate the sharing of ideas for action.

• They will provide information on what people are doing and what's

working, what communities have learned from their experience and

how it made a difference. They will provide inspiration and show

what's possible.

• Disseminating the M&E findings including the documentation should

form a critical part of the overall M&E activities

Concluding Messages• Keep it simple • Build it in• Meaningful as well as measurable• Develop evaluation capacity• Learn and do it yourself!

Thank you!