From Monitoring Through Evaluation To Impact Assessment

The case of NHDRs and vulnerable groups

Andrey IvanovHuman Development Adviser, Bratislava RSC

Main issues in this presentation Measuring what? Measuring how? The role of quantitative data

The case of NHDRs Links to poverty measurements

Some terminological clarity

Three distinct (but often confused) concepts: Monitoring Evaluation Impact assessment

All relate to quality but quality of different aspects of development process and have different purpose, applicability and scope

Measuring what?

Monitoring – primarily of progress. Relatively easy if major components are clear and traceable

Evaluation – of process but also outputs and outcomes. Feasible if these elements are replicable and provide grounds for comparability

IA – long-term changes in development situation. Difficult because of correlations and mutual influences

Applicability: monitoring

Applicable to process (NHDR elaboration) and implementation (applicable both to programs and

projects). For that purpose needed: Progress indicators related both to the outputs

and outcomes (reaching benchmarks) Consistency of procedures (administrative,

accountancy) Consistency of process (participation, consultation

of stake-holders etc.)

Applicability: evaluation

Applicable to process but even more so to its outcomes (the NHDR itself) For that purpose

needed: Measurable outputs and outcomes (linked to

benchmarks). But what is “NHDR outcome”? Measurable inputs (to assess efficiency) Again, consistency of procedures and process

(participation, consultation of stake-holders etc.)

One common “trap”: aligning evaluation to indicators available and biased towards

measuring inputs instead of outputs

Applicability: IA

Applicable only to outcomes (but what is the outcome – the NHDR itself or the process). “Sexy” but difficult to achieve

Applicable at different levels – that of programs (long-term change in development situation) but also of projects (small-scale development interventions

The higher the level, the higher the correlations Outcomes not always quantifiable (what is quantifiable

usually is not an outcome). When it is quantifiable, data is not available (disaggregated poverty or unemployment

The common “trap”: broadly used as a term but with vague conceptual justification and argumentation

Why impact assessment in different areas?Of legislation (regulations IA): Could outline the consistency with other pieces of legislation Could suggest additional areas needing legislative

involvementOf policies: Could outline the consistency with priorities Could measure the advance in strategic areasThe two areas – a kind of “advance warning” scenario buildingOf projects : Could measure the change after the involvement (and not

just count the inputs) Could measure the efficiency of the process (to what extent

there could be better alternative solutions)

Impact assessment: NHDRs

What is the outcome and impact: A book? Number of pages? Process? Change in the paradigm?

Possible indicators: Press coverage? Content analysis of political documents? Policies implemented? Projects implemented? “Ideas leakages”?

Impact assessment: Roma Report

What is the outcome and impact: Again, a book? A knowledge site? A data base behind it? New attitude to quantitative information in Roma

targeted programs and projects? Again, change in the paradigm?

Possible indicators (behind a “Policy Impact” Award): HR paradigm influenced Policies influenced Ideas “leaked”

What is the (often) existing practice?

When policies are concerned: Governments are usually convinced they possess “universal

wisdom”. Impact assessment - usually post factum by the opponents

politically biased.When projects are concerned: Usually focused on the “compliance with the budget” “Properly” defined objectives (easy to monitor) Confusion between means and objectives, outputs and

outcomes “Static” approach often applied “Corporate interest”: often nobody really needs adequate

assessment

How to reach there?

Clearly defined objectives, clear and realistic targets

Involvement of the target group and other beneficiaries

Measurable baseline indicators Consistent and adequate data Constituencies indeed interested in measuring the

impact Take into consideration externalities – both

positive and negative

Example:Employment generation projects

Inputs: training courses, presentations, practical exercises Outputs: number of people with requalification course passed Outcome: number of former unemployed who found jobs Sustainability: duration of the job Impact: HH incomes increased, poverty indicators improved Positive externalities: reduced drop-our tares, reduced societal

fragmentation

Minimum necessary data: Employment status, unemployment rates, incidence of poverty,

levels of income, qualification levels The costs of “reaching the beneficiary”, the costs of the alternative

approaches, opportunity costs The costs of non-involvement at all

Example:Employment generation projects

Possible sources: Targeted small-sample surveys Community level data collection (not sample-based) Interviews the beneficiaries on the specific involvement and

how did it influenced them Interviewing the other actors involved Comparing the trends within the group with the overall

trends

At the end the data should allow building alternative scenarios to compare with and measure the efficiency of the specific

project

Main conclusions

Clear understanding of the differences between monitoring, evaluation and IA is a must

All three should not be treated as substitutive Should not be susceptible to short-term political agenda Data is crucial. Quantitative information components

feeding progress indicators should be embedded from the very beginning of the project

Wide range of stakeholders should be involved Composite indices should be used carefully, for

evaluation of complex, multidimensional processes

What can we offer?

Statistical capacity development support (“Measuring Human Development” manual and KM)

Targeted data collection (“Vulnerable Groups” survey)

1. Methodological aspects of vulnerability research