PROPOSAL OF A FRAMEWORK FOR MONITORING AND EVALUATING PROGRESS TOWARDS THE MILLENNIUM DEVELOPMENT GOALS

Paper presented at the Regional MDG Meeting

by

Sylvain Lariviere, Frederic Martin, Soraya Mellali, and Diakalia Sanogo1

Dakar, February 25-28th, 2003

20 February 2003

1 Respectively (1) Senior Economist at IDEA International Institute and Professor at University Laval; (2) Senior Economist at IDEA International Institute and Professor at University Laval; (3) Poverty Reduction Specialist at UNDP/SURF West Africa; and (4) Senior Economist at IDEA International Institute and Research Associate at University Laval.

INTRODUCTION The Millennium Declaration was adopted in September 2000 by all 189 member states of the United Nations General Assembly, as the world leaders agreed to a set of time-bound and measurable goals and targets for combating extreme poverty, hunger, diseases, illiteracy, environmental degradation and discrimination against women (Quizon 2002). These goals, which have been part of the global development objectives for a number of years as endorsed by member countries of the United Nations, are now called Millennium Development Goals (MDGs). There are eight MDGs comprising 18 targets for which 48 indicators are suggested to measure their level of achievement by 2015 (Appendix 1). The MDGs symbolize a focus on results at the country level, and the achievement of the selected targets depends largely on the understanding of their qualitative and quantitative dimensions, their dynamics and interdependence, and the set of policies, programs, and projects (PPPs) being implemented to reach those targets. In 2002, the U.N. Secretary-General issued his first progress report on the implementation of the Millennium Declaration (UNDP 2002a). The report provides a global perspective on the status of the MDGs, based on internationally agreed indicators and using global and regional databases available within the United Nations system. Global monitoring of the MDGs is important for keeping all stakeholders accountable to agreed commitments but it does not necessarily identify development priorities and monitor progress in each individual country. For MDGs to be meaningful at the national level, targets need to be tailored to reflect national circumstances and development priorities. It is in this context that UNDP is promoting and supporting the preparation of MDG country progress reports (CPR) by national and sub-national authorities, in very close coordination with the United Nations Development Group (UNDG) (UNDP 2002b and c). The MDG country progress reports are intended to serve as a vehicle to keep the national focus on agreed development priorities, to keep the public informed on progress, to foster debate and dialogue, and to trigger action to meet localized targets in each country. They also aim at providing a platform for improving donor coordination around nationally defined priorities. In their current format, Country Reports are not meant to service global and regional data, but rather are based on existing data and analyses, contained in documents such as national development plans, Poverty Reduction Strategy Papers (PRSPs), National Human Development Reports (NHDRs), and the Common Country Assessments (CCAs). But experience to date shows that Monitoring and Evaluation (M&E) of MDGs still suffers from a number of flaws, as it is the case for M&E of PRSPs

2

(Marchant 2002). More fundamentally, good monitoring and evaluation requires rigorous strategic planning which needs to be reinforced in most countries. As a matter a fact, for a number of countries, there is not yet a clear translation of generic MDGs goals and targets into localized Specific Measurable Objectives (SMOs).2 Most important, the specification of the linkages between program level indicators and MDG outcome/impact indicators is yet to be made. The key question is establishing which consistent set of policies and programs will enable a country to reach its MDG targets and which level of resources is involved. A number of developing countries are currently engaged in major planning exercises and policy reforms such as national development plans, PRSPs, MDGs and budget reforms, all of which imply significant M&E needs. Those needs overlap to a significant extent which argues for a global, articulated and nationally owned M&E system. In addition, the development community shares this preoccupation for improving monitoring and evaluation (M&E) to provide government officials, civil society organizations and development partners with better means for learning from past experience, improving service delivery, planning and allocating resources more efficiently, and demonstrating results as part of accountability to key stakeholders (OED 2002). The objectives of this paper are: (i) to present the components of a generic system for monitoring and evaluating progress towards the MDGs ; (ii) to discuss major technical and institutional challenges to the development of such a system; and (iii) to propose guidelines for an action plan to help countries produce their Country Progress Reports (CPRs). The paper is divided accordingly in three sections. 1. COMPONENTS OF A GENERIC SYSTEM FOR MONITORING AND EVALUATING PROGRESS TOWARDS THE MDGS Measuring progress towards MDGs and other development objectives requires monitoring and evaluation using result-based management. Policy makers require analytical capacity as well as an effective information system to monitor and assess PPPs being implemented in order to reach MDG targets.

2 Simply put, a Specific Measurable Objective (SMO) represents a desired level at a certain date on a cardinal scale for a given indicator.

3

Even though MDG targets can be translated into either outcome or impact3 indicators, developing a generic M&E system for MDGs at the national level requires to go beyond a simple measurement of those indicators to trace back their exogenous causes (e.g., climate vagaries, world market price fluctuations, etc.) and endogenous causes (i.e., PPPs inputs, activities and outputs). This follows the impact chain conceptual framework which distinguishes various levels of monitoring and evaluation and their causal relationships (Figure 1):

1. For a given policy, program, or project (PPP) to have an impact on a given MDG, the starting point is to put in adequate inputs, especially budgetary inputs;

2. Then, as the PPP is implemented, activities indicators can be monitored. The direct results of the PPP are output targets, usually goods and services made available;

3. Those outputs are supposed to bring in outcomes for the PPP target groups. These outcomes are often measured in terms of use by the target groups of the goods and services produced by the PPP. Monitoring progress on outcomes means comparing real observed outcomes with several MDGs target indicators;

Figure 1: The Impact Chain

3 The distinction between outcomes and impacts can be fuzzy at times and depends on the perspective of the analyst. In general, outcomes are thought of as effects on intermediate variables while impacts are effects on final variables. In practice, causal paths might involve a number of intermediate and related steps with feed-back effects.

Input indicators

Output indicators

Activities indicators

Policies, programs and projects (PPP)

Context (exogenous factors)

Outcome indicators

Impact indicators

MDG targets

4

4. Finally, in the medium to long run, once all direct and indirect effects have taken place, one can expect the PPP to have an impact in terms of achievement of the remaining MDGs targets (the ones not measured by outcome indicators, but impact indicators) in the country. Assessing the impact implies linking the outcomes to the achievement of the MDGs targets. As for outcomes, monitoring progress on impacts means comparing real observed impacts (using impact indicators) with the MDGs target indicators.

As one moves from left to right on the impact chain, the social, economic, environmental, political and world context (exogenous factors) has more and more influence on the indicator results. This is illustrated by an increasing number of arrows from left to right on the diagram. Moreover, outcome and impact indicators might be influenced by a combination of various macroeconomic and sectoral PPPs (endogenous factors) rather than a single PPP. Monitoring is defined as a process that systematically and critically observes events connected to a project/program/policy, which enables decision makers to adapt policies and/or activities to given conditions. Monitoring aims at i) detecting as soon as possible any significant deviation from expected levels for inputs, activities and outputs related to major PPPs aiming at reaching MDG targets at specific time horizons and ii) assessing trends of MDG outcome and impact indicators. Hence, monitoring includes periodic recording, gap measurement and reporting, basic analysis and storage of data on key indicators. The data required for such monitoring activities include physical and financial information on inputs used, activities conducted and outputs achieved (secondary data coming from project sources) as well as data required to calculate outcome/impact indicators coming from routine administrative sources and from surveys. Evaluation involves a deeper level of analysis albeit less frequent than monitoring. Evaluating the progress toward MGD targets involves three interrelated dimensions: (i) an in-depth analysis of gaps between PPP targeted inputs, activities and outputs, and achieved levels (i.e., answering the question what went wrong in the implementation of PPP? and/or why did we achieve success?); (ii) understanding the underlying causes behind the observed trends of outcome/impact indicators (e.g., why did some groups fall into poverty? why others escaped the poverty trap? why did the young girls enrolment increased? why HIV/AIDS prevalence rate did not change? etc.); and (iii) assessing the impact of selected PPPs that were designed to contribute significantly to MDG target completion. The latter is usually called impact assessment.

5

Such evaluation activities require not only secondary data already used for monitoring purposes but also primary data collection combining qualitative and quantitative methods and associating the target beneficiaries in the process. There is no unique M&E system for MDGs, which is valid for all countries all the time. In fact, there is a clear danger in adopting a simple copy/paste approach since the replication of the same M&E system may not fit well with local institutional realities. The development of an effective country-level M&E system for the MDGs should be seen as an evolutionary process and it should take into account the needs and the institutional specificities of each country, making sure that the system will be operational and primarily oriented toward satisfying the country decision-makers’ needs. In a context of democratisation and decentralisation, the M&E system should also be decentralized and participatory to improve the relevance of analyses conducted, the operational character of policy recommendations and their acceptance by all stakeholders. Hence, an efficient monitoring and evaluation system for MDGs requires that the government of each country, in consultation with other stakeholders (civil society, NGOs and other development partners), assesses its needs in terms of M&E since various needs along with institutional and human capital characteristics will lead to different M&E objectives and designs. However, the experience in a number of countries suggests that a generic MDG M&E system should probably include three core components:

(1) Component 1: Monitoring the trends of outcome/impact localized indicators related to MDG targets and trend analysis; (2) Component 2: Monitoring the implementation of PPPs that contribute to MDG targets achievement and explaining the gaps; (3) Component 3: Assessing the impact of selected PPPs on MDG target indicators.

We discuss each of these core components in the following sub-sections. MONITORING THE TRENDS OF OUTCOME/IMPACT LOCALIZED INDICATORS RELATED

TO MDG TARGETS AND TREND ANALYSIS The objectives of this component are (i) to keep track of the trends of MDG outcome/impact indicators and (ii) to understand them. Trend monitoring at country level should monitor the country’s progress towards reaching the set MDG targets captured through a limited number of localized indicators, and identify areas in which expected outcomes are not achieved. Trend monitoring is what most countries are

6

currently doing in their first Country Reports, albeit with significant limitations on the analytical side. Trend analysis implies identifying endogenous and exogenous factors behind the trends of a given MDG (e.g., why do we observe a reduction or an increase in the proportion of people suffering from hunger). This introduces the causal analysis of trends. It may also involve keeping track of the dynamics of MDGs selected targets and its underlying causes (e.g., who is getting out of extreme poverty, who is getting into poverty and who remains in extreme poverty and why?). Trend monitoring and analysis also requires a good understanding of the real on-the-ground economic, social and cultural situation. All of these analyses, to be meaningful, must be broken down from national level to sub-national levels, be they administrative units (e.g., province, district, commune) or agro-ecological homogenous zones. Moreover, the analyses might be disaggregated, when appropriate, according to other criteria as well (e.g. poor/non poor, gender, urban/rural areas). MONITORING THE IMPLEMENTATION OF PPPS THAT CONTRIBUTE TO MDG TARGETS

ACHIEVEMENT AND EXPLAINING THE GAPS Implementation monitoring is part of the day-to-day activity of the MDGs’ operationalization (EC, 2001) and may have three objectives:

• Monitoring the budget allocations of the PPPs; an effective allocation of funds (i.e., right use of funds at the right time), is a precondition to reaching output targets and improving MDGs indicators.

• Monitoring activities indicators for those PPPs; • Monitoring the level of achievement of the output targets of PPPs related to

MDGs achievement. Implementation monitoring requires that programs and projects have been planned and budgeted using a result-based budgeting approach. Such an approach makes it possible to link the inputs, activities and output targets of the PPPs, e.g., monitoring (i) budget allocation and disbursement to build primary health care units (input); (ii) the number and current status of construction projects (activities); (iii) number of constructed units (outputs). In addition to implementation monitoring, there is a need to understand and explain the gaps between what was projected to be achieved with given resources and what was

7

actually realized (implementation results analysis), e.g., are there any delays in budget allocation, disbursement, activities and outputs with regard to planned schedule and understanding the causes of those delays and why? Explaining the process of transforming inputs into outputs through activities is usually referred to as “project evaluation” in the literature. ASSESSING THE IMPACT OF SELECTED PPPS ON MDG TARGET INDICATORS

Impact assessments are required to inform policymakers and other stakeholders on which public actions have been effective and which one have not worked so well in achieving a given goal. Hence, impact assessment aims at understanding the linkages between PPPs’ outputs and outcome/impact indicators, e.g., to what extent building schools (output) leads to increased girl school enrolment (outcome) and, eventually, increased girl literacy level (impact)? Impact assessment implies addressing a series of methodological challenges, including the isolation of the impact of the policy or program under scrutiny on a given MDG target from the influence of all the other endogenous and exogenous factors that may affect this target indicator. The notion of impact assessment also implies tracing the indirect effects on non-target group welfare (e.g., under-five-year children benefiting from a maternal health improvement project) as well as the medium and long run sustainable effects of the policy or program. Impact assessments studies can be characterized according to several criteria. A first criterion is the reference period for the study: ex-post or ex-ante. Ex-post impact assessment helps understand the past effects of policies and programs to improve upon their future design. Ex-ante impact assessment simulates the impacts on MDGs of a given policy or program so that better policy recommendations can be formulated. A second criterion is the focus of the impact assessment, i.e., (i) either focusing on an analysis of transmission mechanisms of outputs into impacts for the target group (e.g. beneficiaries’ assessments), or (ii) comparing the situation of the target group with that of a control group (counter factual analysis). In addition to the above three core components, two complementary components also need to be considered for MDG monitoring and evaluation. Experience has shown that it is useless and a waste of public money to have data collected and analyzed if it is not extensively available, accessible, discussed, validated, and eventually used by decision-makers.

8

The first complementary component (Component 4), the MDG Management Information System (MDG/MIS), aims at creating a physical, technical, institutional and human environment conducive to the circulation of data, information and results among producers and users of information. MDG/MIS can take advantage of the new technologies of information and communication (NTIC) to provide cheap and reliable computer and communication facilities. The second complementary component (Component 5), Communication/Advocacy, aims at improving information dissemination, favoring a participatory and informed policy dialogue among stakeholders, and eventually, contributing to better design and implementation of public policies. Figure 2 presents in a schematic way the objectives and components of a generic M&E system at country level to monitor progress towards MDG targets.

9

Figure 2: Structure of a generic country-level monitoring and evaluation (M&E) system for MDGs

Communication & Advocacy

Implementation Monitoring Analysis

Trend Monitoring Analysis

Data Collection and Validation

Decision Making: Determination of Policies,

Programs and Projects (PPPs)

Impact Assessment

Primary (survey) Data Secondary data

M&E Objectives

Input indicators

Output indicators

Activities indicators

Policies, programs and projects (PPP)

Outcome indicators

Impact indicators

Trend monitoring Implementation monitoring

Impact Evaluation

MDG Management Information

System

Data storage & management

Data processing, &

analysis

Analysis results

dissemination

MDG Targets

10

2. MAJOR TECHNICAL AND INSTITUTIONAL CHALLENGES IN TERMS OF MDG MONITORING AND EVALUATION AT COUNTRY LEVEL GLOBAL INSTITUTIONAL AND TECHNICAL CHALLENGES RELATED TO M&E The practice of country-level M&E must overcome substantial challenges. Both monitoring and evaluation activities need to be carried out by institutions that possess enough expertise and have strong linkages to key decision-makers, if they are to be useful in the implementation of PPPs aimed at contributing to reaching MDGs targets. Unfortunately, so far, much M&E takes place without adequate development of in-country capacity and without strong links to key decision-making processes, which leads to the loss of precious opportunities to learn what works and what does not, sometimes along with critically needed funds. The most obvious institutional challenge in developing an efficient M&E system at the country level lies in the institutional setup to strengthen national capacity to provide policymakers and other partners with feedback on the impact of the PPPs. It also is crucial that efforts are undertaken to ensure that M&E results are widely disseminated through mechanisms adapted to different groups of civil society organizations, so that these results are used, hence avoiding the waste of resources spent to achieve them. Until recently, participatory M&E methods were underutilized, and there was insufficient advocacy for the use of the impact chains to make transparent linkages between programs inputs and MDG outcomes and impacts. INSTITUTIONAL AND TECHNICAL CHALLENGES RELATED TO MDG OUTCOME/IMPACT

TREND MONITORING AND ANALYSIS As mentioned in the above section, trend monitoring of the MDGs depends on the existing situation in each country which varies from one to the other. However, below are listed often encountered technical and institutional challenges:

• Identification of MDG outcome/impact localized dimensions to reflect local realities: Because of noticeable differences between countries, MDGs proposed targets must be translated into local development objectives that reflect the realities for each concerned country.

• Identification of quantitative MDG outcome/impact indicators: Each

country should keep a balance between (i) selecting its own quantitative

11

indicators related to the MDG localized dimensions and (ii) keeping generic indicators that are the same from one country to another for regional and international comparison purposes. This choice is made all the more difficult since the number of outcome/impact indicators must be limited for practical cost and level of effort considerations. Whatever the final choice, all selected indicators have to be defined clearly and very precisely.

• Calculating outcome/impact indicators to establish a baseline situation in

1990, and an update situation in 2000: Establishing the baseline situation in 1990 and an update in 2000, requires calculating all selected MDG indicators for those dates. In a number of cases, data required to calculate those indicators can be missing, unreliable or conflicting.

• Setting up realistic targets for 2015 and intermediate time horizons (2005

and 2010): Targets should be selected based on the current situation and what is attainable in a given country at a given time. An estimate of realistic targets can be made on the basis of an analysis of the gap between desired situation and current situation/trends. However, a more rigorous specification of realistic targets would require the identification of the set of PPPs that together might probably enable to reach desired MDG targets, costing those PPPs, and matching them with the pool of mobilizable resources (both from public and private sources).

• Articulation of MDG targets with other targets established in other

strategies and plans: In many countries, there might exist a number of already defined national and sectoral strategies and plans which are currently being implemented such as a national development plan, a PRSP, a health sector or education sector development plan, an agricultural development strategy, etc., each of them strategies having their own targets. Even without MDGs, there are already possible problems of intersectoral consistency among those targets, be it between macroeconomic targets (in particular on the public budget and on debt management) and sectoral targets or between sectoral targets as such. Adding MDGs targets in the picture might further add to the complexity of reaching consistency among all those targets.

• Organization of data collection, processing, analysis and dissemination:

This issue is twofold. First, data used for MDG outcome/impact trend monitoring may come from a variety of sources such as national accounts data,

12

budget data, and administrative data (i.e., secondary data). The demands for MDG trend monitoring further outline the existing weakness in the statistical system, both from a technical and institutional point of view. Second, the tendency is too often to move too fast into primary data collection, while it might be more efficient to encourage and/or improve secondary data collection by concerned administrative or technical departments.

• Choosing the levels of disaggregation for data collection and analysis: There

are inevitable trade-offs between (i) the desire for further disaggregation of the calculation of outcome/impact indicators to capture the heterogeneity in situations and behaviours across various groups in society (gender, regions, rural/urban, administrative units, poor/non poor, etc.); (ii) the additional level of effort and cost associated with larger primary data collection; and (iii) the limits of existing routine administrative data.

• Data quality: Data quality is a major preoccupation since it will have major

consequences on the values obtained for the outcome/impact indicators and poor quality data will translate into wrong indicator values, providing a false vision of reality and possibly leading to wrong policy recommendations. Data quality criteria relate to relevance, accuracy, timeliness, accessibility, interpretability, coherence, and representativity (Brackstone 1999; IEA 2001). These data quality indicators constitute a good data quality assessment checklist. However, most of the questions related to the listed indicators remain often unanswered because of the lack of ‘‘metadata4,’’ which makes it difficult to perform a correct data quality assessment. Unless the constraint related to data quality is adequately addressed by improving quality in data collection, poor data quality will always affect the reliability of trend monitoring results.

• Surveys coordination: Household surveys, and community and price surveys

data (i.e., primary data) constitute other major data sources for M&E activities. Substantial difficulties arising with respect to primary data collection are related to the lack of coordination of major surveys resulting in variable data collection workload, with either lows (no data available on key indicators over several years, making it impossible to monitor trends) or highs (too many surveys conducted at the same time, overwhelming data collection capacity, leading to respondent fatigue, differences in methodologies leading to possible

4 ‘‘Metadata’’ concerning numeric data, comprises information such as age, means of acquisition, literature references, geographic and other representativity, etc., i.e., all the information that permit assessment of the data quality (IEA 2001)

13

contradictions in indicator values across surveys, and in some cases duplication of data collection efforts).

• Data storage and management capacity: All too often, databases from costly

surveys are not properly stored, documented, and maintained, hence making them unusable. Moreover, even if properly maintained, databases are often not easily accessible by potential users for a variety of reasons, among which excessive control by the data producers, a lack of information technology allowing for user-friendly and cheap dissemination of those data.

• Data analysis: Very often, survey data and routine administrative data are

poorly valued, simply because of a shortage of national capacities, in particular in data management and processing and in analytical skills to be able to draw the policy implications from results.

• Implications of decentralisation and democratization: Decentralization and

democratization are becoming heavy trends in governance and they have major implications for M&E institutional set up. Respective roles of national level, sub-national levels and local levels in data collection and analysis should be redefined and, even more, must be supported by institutional reinforcement and capacity-building. Deconcentrated structures have often existed for a long time, but with few means to conduct adequate data collection and even more data management and analysis. Community involvement in data collection, result discussion and validation is relatively recent, but is of paramount importance to endogenize results in localized PPP design and implementation. In addition, the lack of coordination in data collection, entry, and processing leads to duplication of efforts at local and regional levels, reporting at times more or less the same information to satisfy different line ministries or donors at central level.

INSTITUTIONAL AND TECHNICAL CHALLENGES RELATED TO MDG IMPLEMENTATION

MONITORING AND ANALYSIS As for MDG outcome/impact trend monitoring, here are often encountered technical and institutional challenges for MDG implementation monitoring and analysis:

• Identification of priority programs or sub-programs that contribute directly to the achievement of a given MDG: Given scarce resources, the

14

government through sectoral ministries needs to identify and prioritize PPPs that will most likely contribute directly to the achievement of MDG targets. This involves a major strategic planning exercise and significant challenges: (i) Prioritization is already a difficult exercise at the sectoral level given interests at stake and implications for budget allocation. Prioritization of PPPs related to MDGs are all the more complex since most MDGs concern several sectors, which means the prioritization of PPPs includes intersectoral trade-offs; (ii) Prioritization of PPPs requires the participation of all stakeholders if the selected PPPs are to be relevant, appropriated and effectively implemented; (iii) most programs are not yet focused on MDG targets, making it difficult to assess their specific contribution to reaching those targets.

• Establishing the linkages first between MDGs and, second, between the

PPPs that contribute to MDG targets: as mentioned in Section 1, outcomes and impacts are closely interlinked and simultaneously affected by a number of endogenous (PPPs) and exogenous factors. There is still too much emphasis on (i) working on one MDG at a time without proper attention to the effect of progress/deterioration of other MDG indicators on the MDG under consideration. For example, several countries or line ministries in those countries are underestimating the impact of HIV/AIDS on their sector; and on (ii) sectoral perspectives when identifying PPPs for a given MDG and not enough consideration of the contribution of other sectors’ PPPs. For example, increasing literacy level requires not only an increase in the quantity and quality of education services, but also progress in non-education PPPs, such as PPPs that aim at improving nutritional and health status of children, raising income-earning opportunities of the parents to let their children go to school, and promoting gender equity.

• Costing selected PPPs: As mentioned in Section 1, once a set of PPPs most

likely to reach MDG targets has been identified, it is essential to cost those PPPs and compare the estimated cost with resources availability in order to check for realism of MDG targets. Establishing linkages between PPPs inputs, activities and outputs can be difficult for countries in which public expenditure allocation and management is based on line budgeting and not on program budgeting. Traditional line budgeting makes it difficult to isolate the activities and related costs that are directly relevant for achieving an MDG. Moving to result-based budgeting in the public sector is a major process which will take years, but is central to efficient PPP implementation monitoring. This process requires

15

significant changes in public management at national level for more result-based management, transparency and accountability, but requires also a simultaneous shift of attitude on the part of a number of donors to move away from project management toward support to more integrated and coordinated approaches led by the government such as budgetary support.

• Multiple M&E sectoral systems and incompatible reporting format: Apart

from the above mentioned lack of result-based budgeting and management which impedes PPP implementation monitoring, many countries are facing a coordination problem in implementation monitoring due to the multiplicity of M&E sectoral systems, or even worse, M&E project systems reflecting the different requests of the various donors. This creates major problems for aggregation at national level for comparison of MDGs target indicators across sectors and regions in the country. This coordination problem is often compounded by the multiplicity of hardware and software used by various ministries to report on PPP implementation.

• Capacity building of M&E at central and sub-national levels: Obviously,

reforming existing institutional and technical PPP implementation monitoring systems require major capacity building efforts at both national and sub-national levels. Priority training areas include strategic planning, result-based budgeting and management, M&E systems, and IT. Training should go beyond a mechanical understanding of tools to include analytical skills reinforcement. In particular, civil servants should be trained on gap analysis, causal analysis, and drawing policy and program implications.

INSTITUTIONAL AND TECHNICAL CHALLENGES RELATED TO MDG IMPACT

EVALUATION5

• The attribution problem: As mentioned in Section 1, the basic methodological challenge in impact assessment is how to link PPP outputs to MDG outcome and impact indicators. In practice, very few real impact evaluation studies use rigorous methods to control for all other factors than the PPP that may explain the obtained outcome or impact.

• The complacency problem: Unfortunately, too many so-called impact

assessments are quick and dirty self-satisfying reports justifying donor programs

5 This sub-section draws upon IDEA International Institute (2002).

16

and allocation to the executing agencies. The challenge is to minimize conflicts of interest in conducting such studies.

• Time and budgetary constraints: Rigorous impact assessment study takes

time and cost money. However, the cost of such a study needs to be put in perspective with the magnitude of the funding allocated for years to a given type of PPP. Moreover, the payoffs of improved PPP design resulting from a rigorous impact assessment study need to be factored in.

• Insufficient demand for and use of assessment results to feed into policy-

making process: A number of national and international PPP managers do not necessarily welcome an impact assessment which might point out possible weaknesses while others are not convinced of the importance of impact assessment and do not know how to use this information for improving PPP design.

• National capability in impact assessment: In many countries, the problems of

limited national capabilities to assess the impacts of public policies right now lie not so much in a shortage of well trained national specialists in this field than in the institutional setup which so far has been unable to deliver enough quality policy impact assessment in time for decision-making.

3. PROPOSAL OF AN ACTION PLAN TO HELP COUNTRIES PRODUCE THEIR CPRs WHAT DO COUNTRIES WANT TO ACHIEVE IN TERMS OF MDG COUNTRY PROGRESS

REPORTS? The global objectives of the MDG Country Progress Reports (CPR) are the following:

• Move the Millennium commitments from the global to the local level; • Create necessary links between global target setting and national priority

setting; • Re-energize a broad political constituency to accelerate progress towards the

goals; • Generate public awareness, scholarship and debate for action around the

development challenges of our times;

17

• Build alliances across and within countries, working with national governments, civil society, the private sector, international financial institutions and other development partners (UN MDG web site – Fact sheet 1)

Key characteristics of those reports are the following:

• Their orientation towards a wide audience: including not only decision-makers and development partners, but also the general public, including NGOs, private sector, population, media, etc. The heterogeneity of audiences for these CPR create a challenge. The orientation of the document toward the general public means it should be easy to understand by the non specialist, but without falling into vague generalities and populist and simplistic slogans. The orientation of the document towards decision-makers means it must be oriented on action and be based on a rigorous conceptual framework and serious data collection and analysis. Our recommendation to reconcile those two audiences is to have one summary report including key findings presented in a clear and graphic way, complemented with a series of technical annexes that would present a more thorough and detailed analysis.

• Reporting on progress to satisfy the major MDG monitoring and

evaluation needs, i.e. the ones identified in Section 1 of this paper. More specifically the contents of the CPR would include the following: For each MDG: 1. Trends and gaps (where are we and why?): This section would provide

an update on the progress towards meeting its MDG targets by presenting the current level and trend of each selected MDG target indicator.6 It would highlight possible gaps between desired level and current level at intermediate time horizons. It would highly desirable to move beyond a mere description of trends and gaps to undertake causal analysis, i.e. understanding why do we observe those trends and what are the challenges that confront each country in meeting its targets;

2. Review of existing Policies, programs and projects (PPPs) (what are we doing and is it working?): This section would identify all significant PPPs that are relevant for each MDG and provide a short description of the PPP. It would also report the major results of any impact assessment study that has been conducted on the impacts of any of those PPPs in terms of progress toward MDGs. Then, the section would describe the major challenges

6 It is assumed that localized MDG target indicators have already been defined in a participatory way.

18

encountered in implementing such PPPs. Finally, it would discuss linkages with macroeconomic, intersectoral and sectoral action plans.

3. Identification of a set of PPPs that will most probably enable to reach MDGs (what should we do and can we afford it?). On the basis of the results of Section 2, Section 3 would (a) first identify a mix of existing PPPs (kept as such or expanded to a wider target group), modified existing PPPs, or new PPPs that together will most likely enable to reach the MDG targets; (b) specify the critical conditions in terms of endogenous or exogenous factors required so that the set of PPPs enables to reach the MDG targets; (c) cost this set of PPPs; and (d) assess resource availability .

4. Improving MDG Monitoring and Evaluation (how can we better monitor and asses progress towards MDGs?): This section would include the current status of MDG monitoring and evaluation activities and propose a workplan of activities to be conducted in the coming year (before the publication of the next CPR) to improve the various components of the information system as presented in Section 1 of this paper, e.g. strengthening data gathering and analysis, statistical capacity building, larger access and use of IT, better institutional setup for wider participation, etc.

• Ownership by national stakeholders: Even more important than the quality of

the CPR per se, is the participatory nature of the process leading to this report. Even though the UN Country Group is playing a key support role in initiating work on these reports, it is critical that these CPR be a national product encompassing the ideas and discussions of all major stakeholders (NGOs and other civil society organizations, private sector, public sector at central and regional levels). This appropriation process assumes first that all major stakeholders at national and regional levels are aware and knowledgeable about MDGs and are convinced of their relevance as far as they are concerned. The MDG awareness campaign is a critical pre-requisite since a number of national stakeholders, especially at regional level, are simply note aware of MDGs. Second, the localization of MDG targets and the choice of priority policies, programs and projects to reach those targets require a number of national and regional consultations and discussions to come up with a final version of the CPR that is representative and relevant. One must not underestimate the time and resources required to conduct a real participatory process as exemplified by the PRSP process.

19

• Articulated with other Monitoring and Evaluation initiatives: The MDG initiative should build upon existing initiatives, be they at the global level such as long term planning, national development plan, PRSP and HIPC, gender promotion, environment protection or at sectoral level such as initiatives against HIV/AIDS and tuberculosis, initiative to promote universal primary education, etc. The monitoring and evaluation of MDGs should in turn benefit and be articulated with existing monitoring and evaluation initiatives, in particular the PRSP monitoring and evaluation system which is being designed and/or implemented in most countries. The MDG country progress reports should themselves be articulated with other reporting mechanisms for those initiatives (e.g. Human Development Report, report on PRSP progress, etc.).

• Conceived of as a stepwise process with several generations: improving as

national capacities improve, in particular, data quality, data analysis capacities, use of information technology, capacity to conduct national and regional debates involving all major stakeholders, communication flows, etc. By December 2002, 17 MDG country progress reports had been issued, and another 50 were under preparation7 at world level (UNDP 2002b). In West and Central Africa, only 3 CPR have been published and a number of countries are right now struggling to publish their own report.8

The published CPRs are of unequal quality from country to country, but they constitute a starting point. Except for few countries (e.g., Vietnam, Tanzania) (UNDP, 2002d; Evans and Coyle, 2002), where CPR were drafted in the context of a somewhat more comprehensive approach, this first generation of country-level M&E system for MDGs provides useful information mainly on: (i) the baseline situation for the 18 MDG target indicators, usually in 1990; (ii) the required trend to reach the MDG targets in 2015 compared with the current short run trend (from 1990 to 2000) to measure overall progress; and (iii) a first qualitative diagnostic of monitoring and evaluation national capacities. Given the commitment of all countries to have an MDG report by the end of 2004 in order for the UN Secretary General to make a global report early 2005, we suggest, wherever possible, to have two rounds of CPR: one in 2003 and a second improved version by the end of 2004.

7 The list of countries that have completed their report are available at www.undg.org and www.undp.org 8 Please see the paper presented in this MDG meeting on the current status of MDG reports in West and Central African countries.

20

It is suggested that the next generation of reports, the 2004 CPR, puts, as much as possible, stronger emphasis on: (i) causal analysis for a better understanding of the situation; (ii) defining a set of PPPs that will most probably enable the country to reach its MDG targets by 2015; (iii) costing those PPPs, i.e., the level of resources required to reach the MDG targets; and (iv) simulate alternative combinations of PPPs to reach MDG targets in order to promote a more informed policy dialogue.

ACTION PLAN TO HELP COUNTRIES PRODUCE THEIR 2003 AND 2004CPRS The action plan that we suggest to help countries produce their 2003 and 2004 CPRs is guided by the following principles:

• A country level information system in support to MDG monitoring and evaluation should be conceived globally from the start with all of its components, but with a modular structure to allow for flexibility and account for human and financial resource constraints.

• A stepwise approach should be adopted in order to account for the level of effort and methodological challenges involved in designing and implementing such a system on one hand and the absorption capacity on the other.

• There should be a strong capacity-building perspective to contribute to the progressive reinforcement of national capacities, full appropriation of the information system, and progressively increased national demand for the results of such a system.

The action plan includes two phases:

• Phase I: Diagnostic of the country’s capacity to product a CPR respecting the above-mentioned characteristics and identification of needs in terms of supporting activities

Each country would fill up the template presented in Annex 2 to assess its current situation and identify its needs in terms of supporting activities. Each country is facing a different situation in terms of MDG reporting, but shares a certain number of common challenges. A few countries have already published their first 2003 CPR; others are on the verge of doing it while a number of countries are lagging behind. The diagnostic will help point out these

21

differences and common points. This activity could be started at the MDG meeting in Dakar, but would need to be continued in the country. This diagnostic phase could run from February 25th till April 30th.

• Phase II: Implementing supporting activities to help each country produce its

2003 and 2004 CPRs.

Since each country is a separate case, the supporting activities need to be tailor-made to fit its specific needs. We suggest to start supporting activities in all countries to improve the upcoming 2003 report and/or the next 2004 report as soon as the Diagnostic phase is completed. The objective is to make as much progress as possible in terms of scope and quality of the CPR within the time frame allowed and to ensure that each country 2004 CPR meets a minimum number of requirements to put the UN Secretary General in a position to make a more comprehensive global report on MDG progress early 2005.

The diagnostic in Phase I would be performed by the national team in charge of MDGs M&E consulting with other major stakeholders, in particular line ministries planning units, and representatives of NGOs, civil society, private sector and development partners. This team could be supported by national consultants. Supporting activities in Phase II would be conducted by a wider array of actors from public service, private sector, NGOs, civil society, the research community, with the support of national consultants depending on the nature of the supporting activity and the expertise of various actors. For example, (a) designing a system of surveys on living conditions would be conducted by the Central Statistical Office in consultation with sectoral statistical departments; (b) setting up a result-based planning and budgeting system would be done by the Direction of Budget of the Ministry of Economy and Finance consulting with the administrative departments in line ministries; (c) identifying the priority set of PPPs in a given region would be conducted by the regional authorities in close consultation with NGOs, private sector and community associations active in this region; (d) an impact assessment study of a major PPP on a given MDG could be conducted by a team of national consultants and researchers, etc. Both for diagnostic and support activities, technical assistance could be provided by (a) UNDP country office economic units (b) UNDP/SURF professional staff and (c) international consultants. They would support nationals by providing methodological guidelines, training, and reviewing documents for comments and suggestions. To

22

minimize costs, technical assistance would be provided by regional training and discussion seminars and by on-line technical assistance. The minimum expected contents of each country 2004 MDG CPR would include, for each MDG: 1. Trends and gaps:

• Localized MDG outcome/impact indicators; • Current level and trend of each selected MDG target indicator; • Possible gaps between desired level and current level at intermediate time

horizons; • A simple causal analysis of those trends and gaps.

2. Review of existing Policies, programs and projects (PPPs): • List and short description of major PPPs deemed relevant for the MDG under

consideration; • Synthesis of any impact assessment study that has been conducted on the

impacts of any of those PPPs in terms of progress toward MDGs; • Major challenges encountered in implementing such PPPs; • Discussion of linkages with macroeconomic, intersectoral and sectora action

plans.

3. Identification of a set of PPPs that will most probably enable to reach MDGs • Suggestion on a mix of existing PPPs (kept as such or expanded to a wider

target group), modified existing PPPs, or new PPPs that together will most likely enable to reach the MDG targets.

4. Improving MDG Monitoring and Evaluation:

• Current status of MDG monitoring and evaluation activities; • Workplan of activities to be conducted in the coming year (before the

publication of the next CPR) to improve the various components of the information system as presented in Section 1 of this paper.

For those countries that are farther ahead and that can go beyond the basic CPR requirements outlined above, we suggest three complementary lines of action to improve the scope and quality of MDG reporting.

23

1. First line of action: Improve the localization of MDGs dimensions and deepen the MDG debate among national and sub-national stakeholders This would involve conducting a series of national and sub-national debates leading to an in-depth qualitative analysis of the dimensions of each MDG and their linkages as perceived by major stakeholders to formulate locally meaningful performance outcome and impact indicators. On that basis, generic MDG target indicators may be adapted or complemented by other relevant indicators to localize MDG targets. Let us underline that this localization aims at reflecting local realities and existing development programs, but does not preclude using a sub-set of core indicators valid for all countries to allow for regional and international comparison. It would also contribute to a better and common understanding among national and regional stakeholders of the prevailing socio-economic and institutional settings, opportunities and constraints of the target populations/regions and the underlying inter-sectoral and inter-regional linkages and dynamics of growth in order to achieve the MDG targets. The social, institutional and manpower constraints including governance issues that need to be addressed to reach the MDGs would be highlighted together with the conventional economic factors. Opportunities to strengthen stakeholder partnerships, integrate the ongoing development efforts, would be explored as well. Similarly, identification of factors that contributed to the failure of similar efforts or inhibited their progress in the past will be useful to suggest precautionary measures. The proposed methodological tools would involve SWOT analysis, focus group interviews, Delphi method using in-depth interviews with knowledgeable stakeholders (NGOs, youth, women group, academics and research institutions, private sector and civil society organizations, government, development partners) and secondary data collection and analysis, mostly from local agencies and administrative records. The data collection and analysis needs of course to be conducted at the central level, but also in representative sub-national areas, disaggregating between urban and rural areas and by gender, to capture differences in perceptions, situations, opportunities and constraints. This process should be conducted along with awareness raising activities in favour of the MDG campaign. As a matter of fact, a number of development actors, especially at the regional and local levels, are still not aware of what MDGs are about. For those who know about them, a large proportion still perceive the MDG campaign as a donors’ initiative to which countries need to comply just like too many other development initiatives. There is a risk of having some confusion among several national and regional actors, coming fresh out of participatory exercises for the design of the PRSP,

24

wondering how MDGs fit with all existing strategic planning initiatives and whether this is not another theoretical concept geared toward the dustbin of development community buzzwords. Therefore MDG awareness campaigns coupled with the above data collection proposal are of paramount importance to explain that the MDG initiative is building upon past and existing initiatives on education, health, poverty, etc. to provide at the same time a more focused (8 MDGs), but more integrated view of development with time-bound objectives for all countries. In a nutshell, the value added of this line of action would be the following:

• In-depth understanding of MDG dimensions at national and regional levels, including disaggregation by rural/urban areas and by gender;

• Localized MDG target indicators and targets; • In-depth understanding of the causes of the current situation and trends in terms

of MDG target indicators at national and regional levels; • In depth understanding of opportunities and constraints to reach MDG targets at

national and regional levels; • Understanding of the reasons why existing PPPs relevant to MDGs might not

have delivered expected results; • Above all, an endogenization of the MDG process as a result of a large

participatory process going beyond mere consultation to contribute significantly to appropriation of the MDGs by major stakeholders.

2. Second line of action: Improve the quality of the MDG debate by providing quantitative information on the costs and impacts of various sets of PPPs on MDG targets Quantitative analyses would contribute to a more informed and rigorous debate about policy options to reach the MDG targets. More specifically, it could involve the following inter-related activities:

• The identification and prioritization of a consistent set of PPPs that will most likely enable a country to reach intermediate and final MDG targets;

• The costing of this set of PPPs to estimate the total cost of reaching MDGs targets and assess the realism of those targets by the set time horizon given mobilizable resources (base scenario);

• The simulation of alternative level of those PPPs to assess which combination of levels of those PPPs will minimize the cost of achieving the set MDG targets given the current institutional setting and performance;

25

• Assessing the level of achievable MDG targets given existing resource availability;

• Evaluating the impact of exogenous shocks (e.g. world prices changes, HIV/AIDS, climate vagaries) on the resource availability and, subsequently on PPP levels and achievable MDG targets;

• Assessing the impact of variable transfer efficiency (affecting the output level for a given input level) on PPP levels and achievable MDG targets.

The methodological tools to conduct those activities would involve the construction of a user-friendly and non-black-box-type MDG Policy Simulation Model to carry out policy simulations as an aid for planning and policy analysis and dialogue to:

• Assess the impact (ex ante) of ongoing and proposed policies and programmes, taking into account the inter-sectoral linkages, on reaching the time-bound targets specified under the MDGs and to estimate their resource needs;

• Suggest what policies and programmes need to be strengthened and/or what additional policies/programmes need to be introduced to eliminate the shortfalls in realising the MDG targets and their corresponding resource requirements;

• Recommend policy matrices, corresponding time-bound targets and resource requirements.

A modular approach could be followed to design and estimate the model as follows:

• Macro-economic module providing the macro-economic structure and growth and public resource availability parameters (relying on existing models);

• MDG outcome/impact module; • PPP cost module using a unit cost matrix of inputs/outputs and, progressively

incorporating the result-based budgeting approach as it is being implemented in Component 2 of the information system;

• Programme module linking the first 3modules. In addition to outlining the major policy options and implications for reaching efficiently MDG targets, work on this model would also contribute to reinforce economic governance by stimulating a more informed and rigorous policy dialogue, and raising decision-makers’ awareness of issues at stake and potential implications of various policy options. The model should also take into account factors such as institutional reforms, legal and regulatory framework, privatisation, globalisation and efficiency of expenditure through proxy variables or dummy variables where applicable to simulate their implications for reaching MDG targets.

26

3. Third line of action: Reinforcing the institutional setup for a more operational and efficient MDG monitoring and evaluation This would involve reinforcing or, if necessary, establishing in each country what we call, in generic terms, a Strategic Economic Governance Unit (SEGU) (IDEA International Institute 2002). This SEGU would be attached to the highest decision-making level (Presidency, Prime Minister’s Office) to ensure direct connection with decision-makers and articulate an inter-sectoral vision. Let us underline that there is no need to create a new structure in all countries where exists a structure performing the functions of a SEGU or that can be adapted to do so. Clearly the institutional setting is country specific and a SEGU can already exist under varying names and be attached to various structures in most countries. Often, this will be the unit in charge of monitoring and evaluation of the PRSP at the Ministry of Economy and Finance or at the Ministry of Planning. In this case, this unit could be supported to extend its mandate to integrate MDGs M&E. In the end, what matters is that this structure exists and performs well. For generic purposes, an institutional setup is proposed in Figure 3. Its mandate would be to provide top decision-makers strategic advice to improve economic governance, in particular:

• Valuing existing information and results of ex ante and ex post impacts of strategies, policies, programs and projects to feed into the policy-making process with respect to the MDGs;

• Educating decision-makers in the usefulness of giving more weight to policy analyses results in policy formulation;

• Contributing to rationalize resource programming with transparent budgets and public investment plans, output-oriented and activity-based budgeting, etc. through the introduction of more efficient budgeting processes and PPPs implementation monitoring;

• Inserting the short run perspective (e.g., the time horizon for PRSP is 3 years) within the medium and long run perspective (the time horizon for MDGs is 15 years) and, at the same time, drawing the short run operational implications of long run planning exercises such at the National Long Term Planning Strategy (NLTPS). The SEGU could help the Government determine a clear agenda of top priorities balancing the short, medium and long run issues;

• Supporting line ministries in the implementation of M&E components.

27

Figure 3: Potential institutional set up of a generic information system for monitoring and evaluation

Parliament / National Assembly

Presidency / Primer Minister’s Office

Strategic Economic Governance Unit - SEGU

Technical Unit

Task Force Chair: President / Prime Minister Secretary: Finance Minister Members: Directors of cabinet of relevant ministries

National Policy Forum

Direction of planning

CSO Line

Ministries Planning,

Monitoring & Evaluation Units

Civil Society Organizations: NGOs, Private consulting firms, Associations, Unions, Academia, etc.

Regional Representatives (Administrative region level)

Local Representatives (Administrative Commune level)

28

Each SEGU would be composed of two structures:

• A task force which would be directly related to the policy-making process. It would be lead by the President or the Prime minister with the Minister of Finance as Permanent Secretary and all Directors of Cabinet of relevant line ministries as members;

• A technical high-level cluster composed of high-level technical experts.

A SEGU would not work as an isolated group. The technical unit would entertain close relationships with relevant national structures such as Study and Planning Units in line ministries and the Central Statistical Office through a network of focal points in each of those structures. Apart from the SEGU, a National Policy Forum would be created (or reinforced if it already exists) in which all stakeholders (SEGU, technical experts of line ministries, beneficiaries and administrative sub-national representatives, and civil society organizations) would meet several times a year to assess M&E results and voice their opinion on suggested policy orientations. Apart from (a) those three lines of action and (b) supporting activities that could be drawn from the template in Annex 2, here is a list of possible complementary activities that are not directly related to MDG reporting, but that would contribute in the medium to long run to the improvement of the information system for monitoring and evaluating progress towards MDGs. These activities are presented by component of the Information system outlined in Section 1: Proposed activities for Component 1: Monitoring the trends of outcome/impact localized indicators related to MDG targets and trend analysis

• Improving secondary routine administrative data useful for calculating localized MDG outcome/impact indicators;

• Designing a system of surveys articulating: (i) qualitative rapid participatory appraisals on population’s perceptions of MDG dimensions and beneficiaries’ assessments; (ii) quantitative surveys to measure MDG localized outcome/impact indicators surveys, including (a) cross-sectoral baseline surveys every 5 to 7 years and intermediate small-scale follow up surveys; and (b) sectoral surveys and (c) census;

29

• Defining and putting out a regular series of publications on trends and trend analysis, including MDG CPRs, PRSP CPRs, etc.;

• Training on MDG outcome/impact trend analysis. Proposed activities for Component 2: Monitoring the implementation of PPPs that contribute to MDG targets achievement and explaining the gaps

• Training on strategic planning, result-based budgeting and management methodologies, IT, gap analysis, causal analysis, and policy and program design.

Proposed activities in Component 3: Assessing the impact of selected PPPs on MDG target indicators

• Training in qualitative and quantitative ex ante and ex post impact assessment methodologies;

• Institutional support to help independent teams in conducting impact assessment studies.

Proposed activities in Component 4: Designing and implementing a MDG Management Information System (MDG/MIS)

• Training in related IT. Proposed activities in Component 5: Communication/Advocacy

• Improving the institutional setup and operating procedures for improved communication between line ministries and non sectoral ministries related to MDG M&E activities;

• Improving the institutional setup and operating procedures for improved communication between central and sub-national levels related to MDG M&E activities;

• Improving the institutional setup and operating procedures to better supply top decision makers (e.g., President, Cabinet Office, Parliament) with MDG M&E results.

30

REFERENCES Brackstone, G. 1999. Managing Data Quality in a Statistical Agency. Statistics Canada,

Survey Methodology, Catalogue No. 12-001 – XPB, Vol. 25 No. 2, Dec. Available on the web at: http://dsbb.imf.org/dqrs_intro.htm

Dixon, A., 2002. Monitoring, Evaluation and the Role of MDGs. A Power Point

presentation available on the web at: www.worldbank.org/wbi/attackingpoverty/events/kazakhstan_1202/2b_dixon_eng.pdf

EC (European Commission), 2001. Draft Guidelines for the Definition of Development

Indicators. DG Development, DEV/B/2 D (2001). Available on the web at: http://europa.eu.int/comm/development/sector/poverty_reduction/infopack_ann1.pdf

Evans, A. and E. Coyle, 2002. Reporting & Monitoring: Post-full PRSP Challenges.

Briefing Note 2, March 2002. Groupe des Nations Unies pour le Développement, 2001. Reporting on the Millenium

Development Goals at the Country Level. A Guidance Note. Available on the web at: www.undp.org/mdg/reportingguide.doc (English) or www.ceasurf.org /mdg/guidfr.pdf (French)

IEA (International Energy Agency), 2001. Data Quality Assessment Indicators. Paris. Marchant, T., 2002. From Project to Program – The New Development Paradigm and

its Implications for M&E. Available on the web at: www.worldbank.org/poverty/strategies/events/attackpov/presents.pdf

IDEA International Institute, 2002. How to Reinforce African Capabilities to Assess the

Impact of Public Policies? Draft Report for UNPD, Quebec City, Canada. OED (Operations Evaluation Department), 2002. Monitoring & Evaluation: Some

Tools, Methods & Approaches. Washington D.C.: World Bank; Available on the web at: www.worldbank.org/html/oed

31

Picciotto R., 2002. Development Cooperation and Performance Evaluation: The Monterrey Challenge. OED Working Paper, World Bank 2002, Washington, D.C.

Prennushi, G., Gloria R. and Kalanidhi S., 2001. “Monitoring & Evaluation”. PRSP

SourceBook, Chapter 3. Available on the web at: http://poverty.worldbank.org/library/view/4480

Quizon J. B., 2002. Measuring, Monitoring and Management for Development Results;

An Approach to Monitoring and Evaluation in World Bank Institute (WBI) Country-Focused Activities. World Bank Institute, Washington D.C..

UNDP, 2000. What is a Medium-Term Expenditure Framework (MTEF)? VIE/96/028:

Public Expenditure Review, Phase II. Available on the web at: www.undp.org.vn/projects/vie96028/whatis.pdf

UNDP, 2001. Managing for Results: Monitoring and Evaluation in UNDP – A Results-

Oriented Framework, Evaluation Office, New York, November 2001. UNDP, 2002a. Status of Millennium Development Goals Country Reporting. Note by

the Secretary-General. E/CN.3/2003/22, United Nations, New York. UNDP, 2002b. The United Nations and the MDGs: A Core Strategy. Available on the

web at: www.itu.int/osg/spu/wsis-themes/UNMDG/MDG-strategy-paper.pdf.

UNDP, 2002c. MDG Fact Sheets. Available on UN Millennium Development Goals

web site, www.un.org/millenniumgoals/index.html. UNDP, 2002d. Millennium Development Goals: Bringing the MDGs closer to the

People. Viet Nam United Nations Country Team, Ha Noi, November.

32

Appendix 1: The Millennium Development Goals (MDGs)

Goals Targets Goal 1. Eradicate extreme povertyand hunger

1. Halve, between 1990 and 2015, the proportion of people whose income isless than one dollar a day

2. Halve, between 1990 and 2015, the proportion of people who suffer fromhunger

Goal 2. Achieve universal primaryeducation

3. Ensure that, by 2015, children everywhere, boys and girls alike, will be ableto complete a full course of primary schooling

Goal 3. Promote gender equality andempower women

4. Eliminate gender disparity in primary and secondary education preferably by2005 and to all levels of education no later than 2015

Goal 4. Reduce child mortality 5. Reduce by two-thirds, between 1990 and 2015, the under-five mortality rate Goal 5. Improve maternal health 6. Reduce by three-quarters, between 1990 and 2015, the maternal mortality

ratio Goal 6. Combat HIV/AIDS, malariaand other diseases

7. Have halted by 2015 and begun to reverse the spread of HIV/AIDS

8. Have halted by 2015 and begun to reverse the incidence of malaria and othermajor diseases

Goal 7. Ensure environmentalsustainability

9. Integrate the principles of sustainable development into country policies andprogrammes and reverse the loss of environmental resources

10. Halve, by 2015, the proportion of people without sustainable access to safedrinking water

11. By 2020, to have achieved a significant improvement in the lives of at least100 million slum dwellers

Goal 8. Develop a global partnershipfor development

12. Develop further an open, rule-based, predictable, non-discriminatorytrading and financial system; Includes a commitment to good governance,development, poverty reduction - both nationally and internationally

13. Address the special needs of the least developed countries; Includes: tariffand quota free access for least developed countries’ exports; enhancedprogramme of debt relief for HIPC and cancellation of official bilateral debt;and more generous ODA for countries committed to poverty reduction

14. Address the special needs of landlocked countries and small islanddevelopment states – (through the program of action for the SustainableDevelopment of Small Island Developing States and the outcome of the twenty-second special session of the General Assembly)

15. Deal comprehensively with the debt problems of developing countriesthrough national and international measures in order to make debt sustainable inthe long term

16. In co-operation with developing countries, develop and implementstrategies for decent and productive work for youth

17. In cooperation with pharmaceutical companies, provide access toaffordable, essential drugs in developing countries

18. In cooperation with the private sector, make available the benefits of newtechnologies, especially information and communications.

33

Annex 2: Suggested template to conduct the diagnostic of the country’s capacity to product a CPR respecting the above-mentioned characteristics and identification of needs in terms of supporting activities The template should help determine the current status of the country in terms of the 5 components of the information system for MDG monitoring and evaluation outlined in Section 1 of this paper by answering a series of questions. If the answer to a question is positive, i.e., such an activity has been or is being conducted in the country, the respondent should provide some key information describing the activity under way. If the answer to a question is negative, i.e., the activity is not yet being undertaken, this activity should probably benefit from supporting activities to be undertaken in Phase II. In this case, the respondent should provide indications on both the area and type of desired support: Identification of desired support areas:

• Strategic planning • Primary data collection • Secondary data collection • Data analysis • IT • Participatory processes • Communication • Institutional setup

Type of support:

• Methodological guidelines • Training • Reviewing documents • On line assistance (answering queries made by nationals).

34

Component 1: Monitoring the trends of outcome/impact localized indicators related to MDG targets and trend analysis For each MDG: Identification of MDG qualitative dimensions

1. Are the MDG dimensions identified at national level based on a qualitative debate and assessment of major stakeholders?

2. Are the MDG dimensions identified at regional level based on a qualitative debate and assessment of major stakeholders in all regions? In case of partial coverage, indicate which regions have been covered and which have not.

3. Are the MDG dimensions identified at national level disaggregated by urban/rural areas? (when relevant)

4. Are the MDG dimensions identified at national level disaggregated by gender? 5. Are the MDG dimensions identified at regional level disaggregated by

urban/rural areas? (when relevant) 6. Are the MDG dimensions identified at regional level disaggregated by gender?

(when relevant)

Identification of MDG outcome/impact quantitative indicators, level and current trend 7. For each MDG dimension identified above, have MDG outcome/impact

indicators been identified? If yes, specify name, unit and definition of indicator 8. Is it possible to calculate each identified MDG outcome/impact indicator at

national level for the base year (1990)? If yes, indicate value. 9. Is it possible to calculate each identified MDG outcome/impact indicator at

national level for year 2000? If yes, indicate value. 10. Is it possible to calculate the trend for the MDG outcome/impact indicator at

national level between 1990 and 2000? If yes, calculate the % variation and graph it.

11. Is it possible to calculate the values for 1990 and 2000 and the trend 1990-2000 for the MDG outcome/impact indicator disaggregated at regional level? If yes, calculate for each region.

12. Is it possible to calculate the values for 1990 and 2000 and the trend 1990-2000 for the MDG outcome/impact indicator at national level disaggregated by gender? If yes, calculate for each gender (when relevant).

13. Is it possible to calculate the values for 1990 and 2000 and the trend 1990-2000 for the MDG outcome/impact indicator at national level disaggregated by urban/rural areas? If yes, calculate for rural and urban areas (when relevant).

35

14. Is it possible to calculate the values for 1990 and 2000 and the trend 1990-2000 for the MDG outcome/impact indicator at regional level disaggregated by gender? If yes, calculate for each gender in each region (when relevant).

15. Is it possible to calculate the values for 1990 and 2000 and the trend 1990-2000 for the MDG outcome/impact indicator at regional level disaggregated by urban/rural areas? If yes, calculate for rural and urban areas in each region (when relevant).

Setting targets for MDG outcome/impact indicators

16. Has the target (in absolute numbers) been established for each outcome/impact indicator at country level for 2015? If yes, specify.

17. Has the target (in absolute numbers) been established for each outcome/impact indicator at country level for intermediate horizons? If yes, specify target level and time horizon.

18. Has the target (in absolute numbers) been established for each outcome/impact indicator at regional level for 2015? If yes, specify target level for each region.

19. Has the target (in absolute numbers) been established for each outcome/impact indicator at regional level for intermediate horizons? If yes, specify target level for each region.

20. Has the target (in absolute numbers) been established for each outcome/impact indicator by urban/rural areas for 2015? If yes, specify target level for rural/urban areas (when relevant).

21. Has the target (in absolute numbers) been established for each outcome/impact indicator by gender for 2015? If yes, specify target level for each gender (when relevant).

22. Has the target (in absolute numbers) been established for each outcome/impact indicator by urban/rural areas for intermediate horizons? If yes, specify target level for rural/urban areas (when relevant).

23. Has the target (in absolute numbers) been established for each outcome/impact indicator by gender for intermediate horizons? If yes, specify target level for each gender (when relevant).

Component 2: Monitoring the implementation of PPPs that contribute to MDG targets achievement and explaining the gaps For each MDG:

24. Have PPPs contributing directly to MDG targets been identified and prioritized through an inter-sectoral and participatory strategic planning exercise, taking

36

into account linkages across MDGs and PPPs? (the output of this exercise would be a set of PPPs that would most probably enable reaching localized MDG targets)? If yes, indicate the set of PPPs and the nature of the exercise that led to this identification and prioritization;

25. Is result-based budgeting and management being used in sectoral ministries at central level? (this would facilitate costing of the set of PPPs required to achieve MDG targets and the information required to monitor the implementation of those PPPs). If yes, specify in which ministry and describe shortly the procedures.

26. Is result-based budgeting and management being used in sectoral ministries at sub-national level? If yes, specify in which ministry, at which sub-national level and describe shortly the procedures.

27. Are sectoral M&E systems harmonized in terms of subject matter? (i.e., core information related to financial inputs, activities implementation, and outputs of the various public programs conducted by ministries and other public institutions)

28. Does an IT Policy and a Workplan exist to progressively harmonize technological support of the M&E systems across ministries? (in particular regarding software and hardware and networking activities to link (a) line ministries and the non sectoral ministry in charge of global PPP implementation monitoring on one hand and (b) central level ministries and sub-national administrative units on the other hand);

Component 3: Assessing the impact of selected PPPs on MDG target indicators For each MDG:

29. Have any rapid impact assessment studies of significant PPPs of relevance for MDGs been conducted? If yes, indicate the title of the study, provide a short description of methodology and results;

30. Have any quantitative impact assessment studies of significant PPPs of relevance for MDGs been conducted? If yes, indicate the title of the study, provide a short description of methodology and results;

37

Component 4: Designing and implementing a MDG Management Information System (MDG/MIS)

31. Has a MDG Management Information System been designed and implemented? ( articulated with sectoral data warehouses and a central data warehouse)? If yes, provide a short description of the MIS;

Component 5: Communication/Advocacy

32. Are there any initiatives to raise awareness of MDGs and related PPPs among the population and to better convey to the Government population’s perceptions and needs in terms of MDGs? If yes, provide a short description of target groups and activities being conducted.

33. Are there any initiatives to consult all major stakeholders (public, private sector, NGOs and other population association representatives, regional representatives) on preliminary results of MDG work for discussion and validation? If yes, provide a short description of stakeholders involved and activities being conducted.

34. Has a communication strategy specifying all the above elements been elaborated and implemented? If yes, provide a short description of the strategy.