Module 5 - Questions and Criteria for Evaluations

Module 5 Module 5 - Questions - Questions and and CriteriaCriteria for Evaluationsfor Evaluations

23-04-20 2

Questions and

criteria

What needs to be evaluated

Methods and techniques

Quality

Aligning Questions and Aligning Questions and MethodsMethods

23-04-20 3

Role of Questions in Role of Questions in EvaluationEvaluation

• Expression of a concern in its simplest form, a need felt by the sponsor (future user?)

• Asked in common language • Best way for donors and evaluators to

understand each other and agree• Questions are refined using interpersonal

communication tools (listening, rewording, etc.)

23-04-20 4

Checking the Checking the FFeasibility of easibility of EEvaluation valuation QQuestionsuestions

• Specific: is there a topic and a stem for each question?

• Measurable: can the question be answered?• Actionable: will the answers provide the

information needed for decision-making?• Relevant: “nice to know” or “need to

know”?• Timely: is it important to ask this question

now?

23-04-20 5

What What CCan an an an EEvaluation valuation TTell ell UUs?s?

• Success– Are we achieving the results that we were

hoping for? – Are results being produced as planned at all

levels of our logic model?• Relevance

– Do stakeholders care about what we are doing? Are we solving the problem?

• Cost-Effectiveness - Are we getting value for money?

23-04-20 6

When When FFormulating ormulating QQuestionsuestions

• Ensure a clear link between questions and the issues of the evaluation

• Be realistic in the number of questions asked

• Choose questions selectively• Ensure questions can be answered• Focus on the important issues

23-04-20 7

Sorting Questions ProperlySorting Questions Properly

There are only 3 types of questions:

Case studies, discussions, file review Descriptive questions Comparative analyses Normative questions Experimentation with a control group or with a before/after analysis

Impact-focused questions

Case studies, discussions, file review Descriptive questions Comparative analyses Normative questions Experimentation with a control group or with a before/after analysis

Impact-focused questions

… that require the use of different methods:

Descriptive How did the program unfold, did the staff in place have the right profile, was the project coherent from the start...

Normative Were unit costs in keeping with those of similar organizations, was the insertion rate after training satisfactory,

Impact-focused (causality) Did the revenues of the target groups increase lastingly, to what extent did the improvement of sanitary condition result from the project...

Descriptive How did the program unfold, did the staff in place have the right profile, was the project coherent from the start...

Normative Were unit costs in keeping with those of similar organizations, was the insertion rate after training satisfactory,

Impact-focused (causality) Did the revenues of the target groups increase lastingly, to what extent did the improvement of sanitary condition result from the project...

23-04-20 8

Questions and TechniquesQuestions and Techniques

Descriptive Available dataCase studiesStatistical survey

Normative Available dataCase studiesStatistical surveyCost-benefit analysisCost-effectiveness analysis

Impact-focused Search for causal relationsStatistical analysisForecast analysis

23-04-20 9

Evaluation CriteriaEvaluation CriteriaC

oh

ere

nce

Relevance

Efficiency

Effectiveness

Impact

Su

sta

inab

ilit

y

Rep

licab

ilit

y

Goals, Purposes

Main Objective

Direct Results

Activities

Means

Logical structure

23-04-20 10

The CriteriaThe Criteria

What is their use ? • The ‘politically correct’ side of the

evaluation, based on the DAC’ standards• In discussions/reflections on evaluation

questions, criteria help ensure nothing is forgotten and helps sort out concerns

• A ‘total’ evaluation (on all criteria) reinforces the logical aspect of the findings, and therefore makes them sounder.

23-04-20 11

Criteria and Questions: Criteria and Questions: CorrespondenceCorrespondence

Relevance

Was the program well suited or adapted to reach the overall goal? Is the goal still relevant today? Does the program still meet the expectations of the stakeholders?

Coherence

In the program’s original design, were the objectives coherent (and are they still) with each other, did the planned activities make it possible to reach the operational objectives that were set, did the planned means make it possible to undertake the planned activities within the allocated time?

23-04-20 12

Efficiency How did the project or the policy work? Were the resources properly mobilized? Were the unit costs within the standard range? Were the expected immediate results obtained and under what conditions in terms of quality, costs and time frame? Were the results satisfactory given the amounts of money invested? Were the contractual demands met?


23-04-20 13


Effectiveness To what extent has the key objective of the project been reached? Has the program had the expected results from the point of view of the beneficiaries (addressing unbalances, reinforcing institutions, changing behaviours, etc.)?Impact Did the program help address the targeted problem (poverty level, for example) or produce the expected development? Did it have unforeseen negative impacts? Are its overall consequences positive?

23-04-20 14


Sustainability

Will the effects of the project or program be sustainable? Are the mechanisms put in place viable?

Replicability Can the policy be duplicated ? What would it take?

23-04-20 15

Hierarchy of project objectives Related indicators that help refine the objectives

Control means planned (or to be invented)

Purpose(s) Indicators Control means Objective Indicators Control means Results Indicators Control means Activities Indicators Control means Resources Indicators Control means

To ensure that all the questions have a concrete content, it is necessary to go back to the original objectives and indicators (project document and logical framework). When there is no logical framework, the evaluator must reconstruct it.

SummarySummary

Documents

Module 5 - Questions and Criteria for Evaluations