DFID-Review of Multilateral Effectiveness

8/4/2019 DFID-Review of Multilateral Effectiveness

1/30

Multilateral Effectiveness andFinancing Meeting

June 20thLondon

A Selection of key papers

Multilateral Effectiveness and Financing Meeting, 20 June 2007, London 1


2/30

Foreword by Mark Lowcock,

Director General, Policy and International,

Department for International Development (DFID)

International organisations have a critical role to play in helping

developing countries achieve the Millennium Development Goals(MDGs). Three institutions the European Commission, the WorldBank and the United Nations already account for around 30% ofglobal aid. Multilaterals have a strong comparative advantage in thefight against poverty because of the global reach of theiroperations, unique legitimacy, breadth and depth of knowledge andexpertise of staff.

The UK is strongly committed to working through the internationalsystem to reduce poverty in developing countries. In 2005/06 theUK channelled 38% of our aid through multilateral organisations.

This is likely to increase in the future. As levels of aid tomultilaterals increases, we shall need better evidence todemonstrate that this is contributing to improved results on theground. This is important both for accountability purposes and topromote ongoing multilateral reforms. It will also enable donors tomake more informed resource allocation choices between agencies.

At present, there are no common reporting standards formultilaterals and the volume and quality of published data onperformance varies immensely. Partly in response to this, a number

of instruments have been developed to assess different aspects ofmultilateral effectiveness. For example, the annual survey of theMultilateral Organisation Performance Assessment Network(MOPAN) looks at multilateral performance at country level asassessed by a group of ten bilateral donors. The Paris baselinesurvey also monitors multilateral performance against keyindicators contained in the Paris Declaration on Aid Effectiveness. Inaddition, several bilateral donors, including DFID, have developedtheir own methodologies to assess multilateral effectiveness.

On June 20 2007, DFID hosted a meeting of bilateral donors to

discuss multilateral effectiveness and financing issues. The purposeof this meeting was threefold:

(i) to share experience, tools and methodologies for assessingmultilateral effectiveness;

(ii) to explore the potential for streamlining some of thevarious tools into a joint approach; and

(iii) to discuss the possibility of developing clearer criteria formaking decisions on aid allocations between multilaterals

This package is the main output from that meeting and includessummaries of the key presentations on:



3/30

1. DFIDs Multilateral Development Effectiveness Summaries(MDES);

2. The Swedish Ministry of Foreign Affairs: Framework forAssessing Relevance and Effectiveness;

3. A Canadian International Development Agency (CIDA) studyon a comparative analysis of assessment tools anddevelopment of a joint approach;

4. DFIDs multilateral allocative efficiency methodology; and

5. An Overseas Development Institute (ODI) survey of partnercountry perceptions of multilateral performance

We hope that these materials can make a useful contribution to theemerging discussion around multilateral effectiveness. This is stillwork in progress. Future challenges include refining and

harmonising existing multilateral assessment methodologies;engaging multilaterals and partner countries in a substantivedialogue around effectiveness issues; and developing a more joinedup international approach to resource allocations.

Beyond the goal of increasing individual agency effectiveness, thereis also a need to develop a shared vision for the overallinternational development architecture. In recent years, the globalaid system has seen a proliferation, fragmentation andverticalisation of institutions. The average number of donors in eachlow income country has risen from around 12 in the 1960s to more

than 30 now. In 2004, the total number of donor activities wasestimated at 60,000, compared to 20,000 in the late 1990s.Changes in the multilateral system with the establishment of singlepurpose multilateral funds for health and education, together withthe emergence of new bilateral donors and Foundations, lie behindthis. The effects have been to increase transactions for recipientsand reduce coherence. This is particularly problematic for those lowincome countries which are highly reliant on aid resources and havethe weakest administrative capacity to manage donors.

Without fundamental system reform, there is risk that the

international system will be unable to meet the significant globalchallenges which we currently face, for example in relation toconflict, fragile states, HIV/AIDS and climate change. The challengefor the future will be to promote an international system fit for the21st century. This should include a clear vision of the respectiveroles, responsibilities and comparative advantages of multilateraland bilateral actors in delivering international development goals.DFID looks forward to working with others over the coming periodto progress this important agenda.

Mark Lowcock



4/30

Contents

Page

5 Department for International Development: MultilateralDevelopment Effectiveness Summaries Methodology Note

11 Swedish Ministry of Foreign Affairs: Framework for AssessingRelevance and Effectiveness

14 Canadian International Development Agency: AssessingMultilateral Effectiveness: A Comparative Analysis of DataCollection Tools

17 Department for International Development: Bilateral andMultilateral Allocative Efficiency Methodology Note

22 Overseas Development Institute: Recipient CountryStakeholder Perceptions of Multilateral Donor Effectiveness



5/30

1. Multilateral Development EffectivenessSummaries

Presented by Department for International Development

(DFID)

Introduction

1. This note explains the methodology the Department forInternational Development (DFID) is using to measure theeffectiveness of multilateral organisations. Just as there is nouniversal definition of multilateral effectiveness there is no oneway for organisations to achieve it and no one way to measure it.

Current Approaches to Measuring MultilateralEffectiveness

2. There are a number of tools to measure multilateraleffectiveness. Each concentrates on a different part of the resultschain and considers the issues from the perspective of differentstakeholders. Figure 1 shows how some of these tools fit within themonitoring or evaluation process.

Figure 1: Different Aspects of Measuring Effectiveness

FactorOne

FactorTwo

Factor

Three

DataCollection

Surveys

Censuses

Administrative

data

PerformanceMonitoring

Inputs

Outputs

Outcomes

Impact Donor Behaviour

Evaluation Projects

Programmes

Themes

Policy

instruments

Institutions Interventions

RESULTS

ImpactEvaluation

ParisDeclaration

MEFF,MOPAN

FactorOne

FactorTwo

Factor

Three

DataCollection

Surveys

Censuses

Administrative

data

PerformanceMonitoring

Inputs

Outputs

Outcomes

Impact Donor Behaviour

Evaluation Projects

Programmes

Themes

Policy

instruments

Institutions Interventions

RESULTS

ImpactEvaluation

ParisDeclaration

MEFF,MOPAN

Annual ProgressReportsAllocation

Models

3. Some of the main tools used to assess multilateral effectivenessare:

Multilateral Effectiveness Financing Framework(MEFF) developed by DFID in 2004, this looks atwhether an organisation incorporates a results focus into



6/30

all its business processes and continually uses results toimprove its performance.

Multilateral Organisations Performance AssessmentNetwork (MOPAN) this looks at multilateralperformance at country level as assessed by a group of

eight bilateral donors, including DFID.

Paris Declaration (PD) on Aid Effectiveness a set ofindicators and targets developed by OECD DevelopmentAssistance Committee (DAC) to make aid more effective.

4. None of these tools encompasses the full range of informationwhich is needed to measure effectiveness although they eachprovide useful information from particular perspectives.

A New Approach

5. Rather than attempting to create a new all encompassingsingle tool to measure agency effectiveness, DFID have devised anew system, based on a Balanced Scorecard Approach, whichattempts to collate existing information and to present it within acoherent framework. A standard Balanced Scorecard Approachrecognises the need for organisations to consider a wider range ofinformation beyond the financial measures they traditionally used.

6. DFID has adopted the Balanced Scorecard to reflect its ownbusiness needs and uses this approach in its Quarterly ManagementReport. In particular it recognises the importance of partnershipsand relationships in DFIDs work (see Figure 2).

Figure 2: DFID Quarterly Management Board Report Framework

DFID Mission:Poverty reduction

throughpartnerships

Managing ResourcesHow well are we planning

and managing our resources?

Managing ExternalRelationships

Are we managing ourrelationships effectively?

Building for the FutureAre we delivering our peopleand organisation for the future?

Delivery of ResultsAre we delivering against

our PSA targets?



7/30

DFIDs Multilateral Development Effectiveness Summaries

7. In developing a new framework for considering multilateraleffectiveness DFID has adapted the Balanced Scorecard Approachand tailored it to meet the specific context in which multilateral

organisations operate. This is the Multilateral DevelopmentEffectiveness Summary (MDES).

8. The summary looks at four broad areas of effectiveness:managing resources, building for the future, partnerships andcountry/global performance. Each area in turn affects the next onehence they are shown in Figure 3 as a continuous circle. While thesummary gives a comprehensive picture of the overall effectivenessof the organisation, it is not intended to directly measure actualdevelopment results on the ground.

Figure 3: Multilateral Development Effectiveness Framework

Partnerships

Country/

Global Results

Building for the

Future

Managi

ng

Resources

Effectiveness

Partnerships

Partnerships

Country/

Global Results

Country/

Global Results

Building for the

Future

Building for the

Future

Managi

ng

Resources

Ef ssfectiveneIMPACT

9. The MDES are short publications which cover the followingfour areas of the organisation:

Building for the Future -How is the organisation buildingfor the future through sharing information, learning andinnovating?This area considers whether the organisation iscommitted to a culture of continual learning andimprovement, how it manages information and its results andwhether it is investing in its staff.

Managing Resources- How is the organisation managingits activities and processes?This area looks at how the

organisation uses its financial and human resources. Itincludes indicators such as disbursement ratios, resource



8/30

allocation criteria, staff recruitment, postings and promotionsprocesses and degree of decentralisation.

Partnerships -How is the organisation engaging withother development partners?This area looks at how theorganisation works with other development partners including

developing country governments themselves. It encompassesParis Declaration Indicators on alignment and harmonisation.

Country/ Global P erformance - How well is theorganisation performing either at the country level and/or atthe global level?The final area considers any informationwhich is available on actual performance either at the countrylevel or at the global level. This includes results of anyevaluations, by the organisation themselves or by otherorganisations, information on portfolio quality, any data onprogramme targets and any other indicators on outcomes andimpact.

10. Each of these four areas is considered equally important andonly by considering the four areas together can one begin tounderstand how effective the organisation is. The inclusion of anarea on building for the future helps us attempt to measure howwell an organisation is placed to perform in the future. Within eachof the four areas we have attempted to collate a number ofindicators under broad headings as shown in Annex 1. The headingsand indicators are not designed as an exhaustive list. Rather they

should provide some suggestions on the types of indicators whichcan be considered under each heading.

11. In the future, DFID is thinking of collating information foreach of these areas, summarising and possibly assigning a trafficlight based on assessment, as shown in Figure 4. The traffic lightassessments should be based on objective evidence but they willalso rely on judgement depending on what evidence is available forthat particular organisation.

Figure 4: Traffic Light Assessment



9/30

Purpose of the Multilateral Development EffectivenessSummaries

12. The Multilateral Development Effectiveness Summaries giveusers a general overview of an organisations effectiveness. They

include basic data on the organisation as well as a summary of theassessment and the context in which the organisation is working.The summaries are aimed at providing senior managers and staffworking on multilaterals with a concise summary of all availableinformation on the effectiveness of a particular multilateralorganisation.

13. The information is not designed to produce a ranking ofmultilateral organisations by their effectiveness or to provide asingle basis for expenditure decisions as there are otherconsiderations which would also feed into this type of decision, suchas the scale, scope, coverage and risk involved with each particularorganization. Rather the information should be used to help toinform decision making and to ensure senior managers have aconsistent picture of each multilateral.

Limitations to the Multilateral Development EffectivenessSummaries

14. The multilateral development effectiveness summaries are atool to present the latest available information on the effectiveness

of an organisation. They rely on the collation of existing data. Assuch there are issues of comparability both across organisationsand also across time. One organisation may collect a lot moreinformation than another which may influence the overall picture ofits effectiveness. The quality of data will vary considerably acrossorganisations. The data which does exist may not always becomparable across organisations as it may be collected in differentways and using different definitions.

15. For these reasons the multilateral development effectiveness

summaries should not be used to rank organisations. Rather theyprovide a snapshot of existing information, collated into a coherentframework.



10/30

Examples of MDES IndicatorsCommitment to Continual Improvement

Commitment to ContinualImprovement

Does the Board require management to act on performance results (MEFF 1Ac)?

Building Knowledge and Lesson Learning

Lesson Learning Does the organisation have adequate mechanisms for spreading lesson learning

across the office (MEFF 7Ac)?

RBM System To what extent does this organisation manage by results?

Staff Development

Training & Development What is the average number of days training per staff member?

Staff satisfaction What are staff resignation rates?

Does the organisation have IiP accreditation or similar?BUILDING

FORTHE

FUTU

RE

What is the level of staff satisfaction (e.g. % of staff proud to work for the organisation)?

Corporate Strategy

Strategy Is the organisation's corporate strategy based on a clear definition of mandate and comparative advantage(MEFF 2Aa)?

Resource Management

Disbursement

Administrative Efficiency

What is the disbursement ratio i.e. commitments versus disbursements (Paris Declaration Indicator 7)?

What is the administrative efficiency ratio?

Aid Allocation How well is the organisation's resource allocation criteria aligned with its corporate strategy and comparativeadvantage?

Staff Management

Recruitment & postings To what extent is staff recruitment, postings and promotions meritocratic and transparent (MEFF 6Aa)?

Operational Management

Decentralisation Is the organisation sufficiently decentralised to enable it to respond flexibly to country demand (MEFF 4Ba)?MANAGING

RES

OURCES

Programme Efficiency What is the speed of disbursements (e.g. time taken from project approval to implementation)?

Voice

Voice What mechanisms exist for developing countries to influence the strategy of the multilateral?

Civil Society How actively is this organisation promoting the participation of civil society (MOPAN Support to CivilSociety)?

Partnership BehaviourPartnership Behaviour What mechanisms are in place to seek feedback on partnership behaviour and what do the results show?Alignment

Alignment To what extent does this organisation promote or enable government ownership throughout theproject/programme cycle (MOPAN Capacity Development and Technical Advice Qu.)?

What % of aid flows to government sector is reported on national partners budgets (PD 3)

Use of Country Systems Is the number of Project Implementation Units decreasing or non-existent? (MEFF 4Bb, PD 6)?

What % of TC flows are provided through coordinated programmes consistent with partners' nationaldevelopment strategies (Paris Declaration Indicator 4)?

Do they use countries own public financial management and procurement systems (PD 5a & 5b)?

In what ways has this organisation been aligning its strategy/programme/projects with national strategies(MOPAN Alignment with National PRS, Policies and Procedures Qu.)?

Harmonisation

Coordination To what extent does this organisation participate in local donor coordination activities such as donor workinggroups (MOPAN Inter-Agency Coordination)?

Information Sharing To what extent does this organisation share information with other donors (MOPAN Inf. Sharing Qu.)

PARTNERSHIPS

Harmonising Systems What evidence is there of harmonising procurement and consulting services procedures, disbursementpolicies and evaluation practices (e.g. % of donor missions and country analytical work which is joint PD 10aand 10b)?

Country/Global Results

Evaluation What information is available on the organisations' performance at country level?

What evidence is there of the independence, credibility and utility of the organisations' own evaluations (e.g.Operations Evaluations Departments, DAC Peer Reviews, NGO reports)?

Results What impact is the organisation having at the country/global level?

Portfolio Quality

Quality of projects andprogrammes What % of projects/programmes met their targets?

COUNTRY/GLOBAL

RESULTS

How does this vary across sectors, regions and countries?



11/30

2.Swedish Framework for Assessing

Relevance and Effectiveness

Presented by the Swedish Ministry for Foreign Affairs

Introduction

1. In 2006 Sweden channelled 13 billion Swedish kronor or 50%of its development budget through multilateral organisations (MOs).As with other donors, this has generated increased demands forimproved monitoring and accountability of multilateralperformance. In April 2007 the Swedish government adopted aStrategy for Multilateral Development Cooperation. The strategygives normative guidance on defining and developing criteria forpriorities, financial principles and methods for strategic influence onmultilateral work.

2. The strategy proposes that relevance and effectivenessshould be the main criteria in assessing and deciding on aid to allmultilateral channels. Relevance means, firstly, the compatibility ofthe activities with Swedish development goals and, secondly, therole of the organisation in the international multilateralarchitecture. Effectiveness means, firstly, whether the organisationcontributes to the relevant goals set and, secondly, whether the

activities are organised so as to lead to results and employ aidresources effectively.

3. The Ministry for Foreign Affairs developed a pilot frameworkfor assessing relevance and effectiveness of the multilateralorganisations. 27 multilateral organisations (UN organizations,development banks, EU and vertical funds) were assessed duringthis first cycle.

Purpose

4. The assessment framework serves two main purposes,namely to:

Promote strategic dialogue with the MOs, and provide thebasis for developing specific organizational strategies.

Guide decisions on allocations of funds to MOs,



12/30

Design Characteristics

5. The overall objective of the organisation performanceassessment model is to provide a snap-shot of where theorganisation is at and in what direction it is moving. To do this an

assessment template or scorecard has been developed, whichincludes five separate sections covering:

(i) Relevance of goals to the main priorities set out inSwedens Policy for Global Development

(ii) Relevance of the organization to the international aidarchitecture.

(iii) Internal effectiveness e.g. the organizational structure,results-based management focus, monitoring andevaluation systems, and transparency/disclosure policy,

including reporting on auditing.

(iv) External effectiveness e.g. selectivity, cooperation withothers (i.e. in line with the Paris declaration), and thedegree of focus at country level.

(v) Reform potential and other expected changes (or trends).

6. Against each of these criteria a number of qualitative andquantative questions are asked in order to build up an overallpicture of the relevance and effectiveness of the organizationsconcerned.

Methodology

7. The process for conducting the assessment during 2007 wasled by a project team in the Swedish Ministry for Foreign Affairs,and for each organization a special sub-team was set up with stafffrom the Swedish International Development Cooperation Agency(Sida), other ministries, embassies, constituency offices and EU/UN

representations, all working in different capacities with the specificorganization. The assessment was made jointly by the subgroup.The time-frame for completing the assessments was fairly short(four weeks).

8. Both internal and external sources of information were usedin compiling the assessment. A reference group was alsoestablished for quality assurance purposes, which reviewed thequality of assessments at the end of the process.



13/30

Relevance to resource allocation

9. According to the strategy, a number of core principles shallapply to the financing. These include:

Non-earmarked contributions Predictable long-term financing Thematic Financing (vertical funds)

Multi/bilateral support through country programmes Burden sharing

10. The assessment framework has provided further informationon which to base decisions around resource allocations tomultilaterals. Generally, the following principles are likely to beapplied in the future:

Enhanced effectiveness and relevant = increased contributions,non-earmarked and multi-year contributions

High effectiveness and relevant = unchanged contributions,non-earmarked and multi-year contributions

Low effectiveness but relevant = reduced contributions andshort-term financing

Not relevant = reduced contributions and possible phasing out

Lessons Learned

11. A number of lessons have been learnt from the initial phaseof the Swedish assessment process. For example:

(i) It is difficult to directly compare multilateral organisationswhich have different roles and mandates within theinternational system

(ii) Assessments of this type make an important contributionto organisational learning

(iii) They also provide a good basis for dialogue withmultilaterals, organisational strategies and budgetallocations

(iv) There is limited knowledge/evidence on the overallinternal effectiveness of multilateral organisations



14/30

3. Assessing Multilateral OrganisationEffectiveness: A Comparative Analysis of

Data Collection Tools

Presented by Canadian International Development Agency(CIDA)

Introduction

1. The Multilateral Organization Performance AssessmentNetwork (MOPAN) is a network of ten like-minded donor countries1.At MOPAN meetings in 2006 there was consensus that membersrequired further information on the effectiveness of individualmultilateral organisations (MOs) beyond the Annual MOPANSurvey2. This was in order to meet growing demands for

accountability, and to make more informed policy choices asmultilateral budgets increased.

2. The members therefore established a Working Group to beginmoving towards a common approach for the assessment ofmultilateral effectiveness. CIDA has led the initial phase of work,which has included a comparative analysis of existing bilateraldonor assessment approaches3 and recommendations on how amore harmonised approach might be developed and applied in thefuture.

Main Findings

3. The main findings from the comparative analysis of existingbilateral assessment approaches were as follows:

The main drivers for assessing multilateral effectiveness are to(i) increase accountability (ii) promote organisational learningand change, and (iii) inform resource allocation decisions

1

Current MOPAN members include: Austria, Canada, Denmark, Finland, France, The Netherlands,Norway, Sweden, Switzerland, United Kingdom

2MOPAN conducts an annual perception survey of 3 multilateral organisations in 8-10 countries. In

each country, MOPAN members complete questionnaires which feed into a country report and anoverall synthesis report for the agencies concerned. The focus of the Survey is on multilateralpartnership behaviour towards national stakeholders (governments, NGOs, private sector) indeveloping countries as well as towards other international development agencies. The Survey is notan evaluation and does not cover actual development results on the ground.

3A total of seven bilateral donor tools were assessed as part of this work: Performance Management

Framework (Danida); Scorecard/Multilateral Monitoring System (Netherlands MFA); Framework forAssessing Relevance and Performance (Sweden MFA); Multilateral Evaluation Relevance andAssessment System (CIDA); Multilateral Effectiveness Framework (DFID); Results-BasedManagement Assessment Program (Danida); Multilateral Effectiveness Summary Balanced Scorecard

(DFID).



15/30

Comparison of the design characteristics reveals a common setof four criteria for assessing effectiveness: domestic policyrelevance, international relevance, internal performance andexternal performance.

The seven multilateral assessment tools reviewed included over250 indicators of effectiveness, many of which are overlappingand duplicatory. A few information gaps related to corporateindicators, quality assurance processes, and partner governmentperspectives.

There is a need for greater transparency in conductingeffectiveness assessments, particularly where these are linkedto resource allocation decisions

Comparison of the indicators reveals a common frame ofreference focused on s t ra teg ic , ope ra t iona l , r e la t i onsh ip a n d k n o w led g e ma n a g em e n t .

Good potential exists to develop a common donor approachbased around these core dimensions of effectiveness.

Common Approach: Key Principles

4. In developing a common approach five key design principles(based on agreed Managing for Development Results principles)have been proposed. The assessment approach should:

i) Generate relevant and credible information to support bilateraldonors in fulfilling their governance responsibilities for holdingmultilateral organizations to account.

ii) Generate useful information to support multilateralorganizations in their efforts to foster a results-basedmanagement culture and develop the requisite monitoring andevaluation systems to facilitate organisational learning, improve

development effectiveness and produce credible performancereports

iii)Should apply the keep it simple principle, keeping the initialnumber of indicators reasonable and designing the datacollection tools with a view to facilitating data analysis andinterpretation.

iv)Should be designed as a collaborative management exercise, asopposed to a bilateral donor measurement exercise.

v) Should be designed to engage multilateral organisations in alearning and improvement process.



16/30

Developing a Common Approach: Recommendations

5. This study proposes the use of a Balanced Scorecard as acommon approach to assessing multilateral organisationeffectiveness. The Balanced Scorecard below is based on a commonframe of reference focused on s t ra teg ic , ope ra t iona l ,

r e l a t i o n s h ip a n d k n o w le d g e ma n a g e me n t categories derivedfrom the concept mapping exercise undertaken as part of thecomparative analysis. To further develop the balanced scorecard,MOPAN Members will have to agree on a core set of commonindicators, which will build on the strengths of the existingassessment approaches, and modify them as required to collectdata from at least three data sources representing differentperspectives on the same indicators of interest (e.g. donor, clientand multilateral organisation perspectives). When selecting the coreset of performance indicators, they will also have to forego their

stakeholder role in favour of a shareholder role, thereby putting themultilateral organisation in the foreground, while the uniquelynational development policy objectives of each bilateral donorcountry are put in abeyance.

6. A second phase of work is now underway. This will seek toagree a core set of indicators to populate the scorecard, as well asa commonly agreed process for collecting the relevant data.

June 3, 2007 16

STRATEGICSTRATEGIC

MANAGEMENTGovernance

Corporate StrategyCountry-Level Strategies

Corporate Indicators/Targets

KNOWLEDGEKNOWLEDGE

MANAGEMENTOrganisational Stewardship

Performance MonitoringEvaluation ManagementPerformance Reporting

Lessons Learned RELATIONSHIPRELATIONSHIP

MANAGEMENTDonor Harmonisation w/

Multilaterals and BilateralsAlignment with

Country Partners

OPERATIONALOPERATIONAL

MANAGEMENTFinancial ResourcesHuman Resources

Quality @ EntryCountry Portfolio

MfDR

The MOPAN Balanced ScorecardThe MOPAN Balanced Scorecard



17/30

4. DFID Multilateral Al locative Efficiency:Methodology note

Presented by DFIDBackground

1. DFIDs programme expenditure is set to increase significantlyas the UK continues to make progress towards its commitment tomeet the UNs oda/GNI target of 0.7% and the commitments andexpectations resulting from the Commission for Africa, the 2005Gleneagles G8 Summit and DFIDs third White Paper Makinggovernance work for the poor. Within this context, the Value forReform programme has taken a robust and critical look at how and

where the aid budget is deployed with a view to improvingprogramme efficiency and effectiveness.

2. This note provides an explanation of the methodology used inDFID to measure non-cashable efficiency savings.

Efficiency Methodology

3. Allocative efficiency for the bilateral programme is aboutchoosing the country allocation scenarios which would maximise

the highest number of poor people lifted out of poverty for a givenbudget. In the multilateral case, allocative efficiency is defined asthe extent to which a portfolio of multilateral investments and theircorresponding country allocations, have greater poverty impact(measured by the number lifted out of poverty) than a set ofprevious decisions made on the multilateral institutions.

4. The methodology adopted here is based on a Collier Dollarmodel of poverty efficiency modified for DFIDs purposes. Themodel basically looks at an aid growth equation which shows themarginal and average impact of aid on growth. The growth is thenlinked to poverty reduction by using an average poverty elasticityof growth of 2 (a 1% increase in GNI leads to a 2% fall in thepoverty headcount). The size of this impact then is dependent onthe aid flows to a particular country, the level of poverty and thequality of policies (as proxied by CPIA).

5. Poverty efficiency is maximised by reallocating aid to equalisethe marginal efficiency in each country. The marginal efficiencyestimates derived from the model determine how much extragrowth and therefore, how much extra poverty reduction we will

achieve by scaling up aid to a particular country. Thus, we canthen estimate for each country the number of poor people that are



18/30

removed from poverty for a given aid allocation and sum this upover the aid programme and alternative portfolio of investmentsscenarios. The efficiency savings figures are derived by takingthe percentage of the additional people lifted out of poverty from apoverty efficient scenario multiplied by the total budget.

Models shortcomings

6. Note that the model is imperfect and is highly dependent bothon the parameters of the aid growth relationship and on theavailable data which results in a large margin of error. Some ofthe shortcomings of the model are outlined below:

This model only compares bilateral and multilateralinvestments according to how poverty efficient the allocationsare; no other information is taken into account in determiningthe efficiency savings.

The model derives the marginal efficiency (ME) estimates foreach country based on a dataset of global aid in 2002.Updating the data for subsequent years would result indifferent ME estimates and different efficiency savings figures.

Changing the parameters of the aid growth relationship willyield a different set of marginal and average efficiencyestimates of the aid impact on poverty reduction.

20% of current aid goes to countries where there are data gapswhich means we cannot calculate a poverty efficient allocationto these countries.

This model is about allocative efficiency only. The otherdeterminant of aid effectiveness the quality of the aid itself is assumed to be the same across institutions and across time.

The model currently only has information of historical andcurrent disbursement patterns for 9 or 10 of the multilateralinstitutions which means not all of the positive choices madeon portfolio investments will be wholly reflected in the

efficiency savings.

The model does not incorporate any further efficiency gainsthat might be achieved from technical efficiency gains oradditional information that may influence the allocationdecision.

Bilateral efficiency

7. The efficiency programme for our bilateral allocationscompares a baseline case of proportionate scale up from 2007/08

each year to 2010/2011 to a more poverty efficiency scenario.The efficiency savings measure is calculated by taking extra people



19/30

lifted out of poverty permanently as a percentage of the total andtranslating that into an efficiency savings figure.

8. Using this methodology, an efficiency savings of100 millionhas corresponding number of people lifted out of poverty

permanently. This means that by allocating in a more povertyefficient way, we can save 100 million or lift an additional100,000 people permanently. Efficiency savings figures can alsobe cumulated over multi-year budgeting cycles because theadditional people we lift out poverty in each year is a permanenteffect.

Multilateral efficiency

9. The efficiency programme we are expecting to deliver through

the multilateral portfolio will be calculated in three ways:quantitatively based on a Collier Dollar global ideal framework,other quantitative estimates for individual institutions and finallyby bringing in qualitative effectiveness information of institutionscomparative advantage.

10. The baseline scenario for the multilaterals, unlike in thebilateral programme, cannot simply be based on a proportionalscale up from 07/08 as there are other factors influencing thedecisions around the multilaterals. The baseline is based on a set

of low case assumptions and excludes any investments to meetWhite Paper 3, and spending decisions made in the last 12 months.Therefore, the annual efficiency savings figures represent theextent to which a portfolio of multilateral investments and theircurrent and historical country disbursement patterns is morepoverty efficient than a set of previous decisions made individuallyon the multilateral institutions.

11. We have looked at the allocative efficiency implications ofseveral multilateral spending scenarios. Here again, the efficiencyfigures we derive reflect that by making portfolio choices in a moreefficient way saves DFID x million or lifts an additional x thousandpeople out of poverty permanently.

Relative efficiency by institution

12. Other qualitative information should be used in makingportfolio choices between the institutions, including institutionsrelative contribution to sectoral goals. This note also includesallocative efficiency estimates of the major institutions according totheir global aid ideal. The calculations are based on an assumption

that were there to be an additional 100 million above thepreferred budget scenario, how would investing all of it in one



20/30

institution compare with investing it proportionately through all theinstitutions.

13. These estimates could be seen as the opportunity costs ofdifferent scenarios of scaling up within a growing envelope.

Therefore, an efficiency savings of17 million for IDA means thatby investing the additional 100 million in IDA above that which has

been allocated under the preferred spending scenario, we couldspend 17 million less to lift an additional 15,600 people versusinvesting it through all the institutions in proportionate amounts tothe preferred scenario allocations. By the same token, investingthe 100 million in the EC, means we need to spend an additional41 million pounds to lift the same number of people out of povertyas we would by investing the additional 100 million proportionatelyto all the institutions.

14. Again, these efficiency savings are based on the Collier Dollarmodel and the same margins of error exists. It should be noted thatthese figures demonstrate at the margin what the allocativeefficiency savings would be. In deciding portfolio balance, there willbe many other efficiency and effectiveness considerations central tothat decision.



21/30

Relative efficiency by institution

2010/ 2011

RatingAdditionalNumber of

people liftedout of poverty

Efficiency

savings( m)

AfDF

41,400

45.52

EDF Very strong

38,300

42.04

FTI

27,100

29.78

IDA Strong

15,600

17.19

GFATM

7,000

7.74

UNDP

2,100

2.28

Other Medium

1,100

1.16

UN

-14,900

-16.33

AsDF

-20,500

-22.50

EC Weak

-37,700

-41.40DFIDbilateral

33,600

36.89



22/30

5. Recipient country stakeholder

perceptions of multilateral donoreffectiveness

Presented by Overseas Development Institute (ODI)

1

Key Points

The perceptions of 261 senior recipient countrystakeholders about the performance of a range ofmultilateral organisations are a challenge for donors.

Respondents and donors appear to use different sets ofcriteria to assess overall effectiveness.

Perceptions of governance and ownership appear to affect

recipient preferences about which multilateral organisationsshould disburse additional sums of ODA

1. This briefing paper describes the results from a pilot projectaimed at seeking the views and perceptions of key stakeholders inrecipient countries on the performance of a range of multilateralorganisations and their preferences for which organisations shoulddisburse additional aid.

2. Despite the dip in aid volumes in 2005, OECD/DACpredictions still suggest that aid volumes will rise to around $130bn by 20102. At the same time, a number of bilateral donors arereducing costs by holding or even cutting staff numbers. Thesedonors are therefore seriously considering increasing the aid theychannel through multilateral organisations, because they will find itdifficult to spend the additional money bilaterally. In addition, theremay be a number of positive effects of increasing disbursementsthrough the multilateral system such as reducing procurementcosts and streamlining political conditionality.

3. If bilateral donors are to increase their spending through themultilateral system substantially, they will need to demonstratethat the money is being spent well. This will require them to have abetter understanding of the effectiveness of multilateralorganisations. There have been a number of attempts to assess theeffectiveness of donors, including multilateral agencies. Some ofthese use results-based management toolkits, a few examineorganisational policies and processes, but most are based on

1This paper is taken from ODI briefing paper of same title.

2 OECD/DAC April 2007, http://www.oecd.org/dataoecd/13/26/38341348.pdf



23/30

perception data, usually of staff from donor governments andagencies3. A few, like the Strategic Partnership for Africa (SPA)Survey and the OECD/DAC's Paris Baseline Survey also seek theviews of recipient government officials. Until now there have beenno systematic attempts to seek the views of a wider range of

stakeholders in recipient countries, a gap that is all the moresurprising considering the range of stakeholders that interacts withdonors in-country and is responsible for implementing donorprogrammes.

The Study: seeking stakeholder perceptions

4. Stakeholders from six countries gave their views andperceptions about various dimensions of effectiveness of sevenorganisations. The six countries were: Bangladesh, Ghana, India,South Africa, Tanzania and Zambia. The seven organisations were:

the African and Asian Development Banks, the EuropeanCommission, the Global Fund for AIDS, TB and Malaria, UnitedNations Childrens Fund (UNICEF), United Nations DevelopmentFund (UNDP) and the World Bank. Locally based countrycoordinators administered a questionnaire seeking views from fivedifferent stakeholder groups;

Figure 1 shows the total number of respondents by country, whileFigure 2 shows the total number of respondents by stakeholdergroup. In total, despite a very short data collection period, 261

well-informed and senior individuals answered the questionnaireand contributed over 2300 text comments which provide valuablecontext and additional insight.

55

49

49

38

35

35Bangladesh

Ghana

Tanzania

India

South Africa

Zambia

Figure 1: Number of respondents by country, total 261

3Donor attempts to assess donor effectiveness where the results are publicly available include: The

Multilateral Organisation Performance Assessment Network (MOPAN) Survey, the OECD/ DAC ParisBaseline Survey, the Performance Management Framework (Denmark), and the Strategic Partnershipfor Africa survey.



24/30

52

69

44

43 53

Business

Civil Servants

Civil Society

Govt.Ministers

MPs

Figure 2: Number of respondents by stakeho lder group, total 261

6. Much of the attention of donor and recipient governments hasbeen focused on assessing progress against the Paris Declaration asa key measure of effectiveness. The survey sought respondents

perceptions of multilateral organisation (MO) performance againstthree elements of the Declaration.

7. In addition, the project developed its own effectivenesscriteria. This exercise drew on two separate projects which broughttogether recipient country stakeholders, one coordinated by theCommonwealth Secretariat together with la Francophonie, and thesecond by Debt Finance International. Respondents were asked torank fifteen of the most important criteria from the two exercises inorder to see if recipient stakeholders take into account a broaderrange of factors as they think about organisational effectiveness.They were then asked to rank the performance of the MOs againsteach of the criteria. Finally, the survey asked respondents to rankthe MOs in the order in which they would prefer them to act asdisbursement channels for any additional ODA flows.

The Results

8. Figure 3 summarises the results. Each bar is split into fourelements. The top three elementsshow the ratings given tothe MOs for the three Paris Declarationindicators. Thebottom element shows the scores received by the MOs whenrespondents were asked to rate overall effectiveness.Ratings for each element were on a five point scale giving anoverall maximum potential score of twenty. The crosses onFigure 3 shows respondents preferences for whichorganisations shoulddisburse additional ODA. The maximumpotential score is three.



25/30

Large scalefunders

SpecialisedAgencies

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

18.00

20.00

AsDB

AfDB W

B ECUN

DP

UNICEF

GFAT

M

MultilateralOrganisation

EffectivenessRatings(maximums

core

20)

0.00

1.00

2.00

3.00

RespondentPreferences

forDisbursement(maximums

core3)

Overall developmenteffectiveness

Harmonises with otherdonors

Aligns with governmentpriorities

Promotes governmentownership of developmentprocess

Respondent disbursementpreferences

Figure 3: Multilateral organisation effectiveness ratings andrespondents preferences for disbursement of additional aidflows.

9. Respondents have a clear idea about the elements that makeup an effective donor. The fifteen additional effectiveness criteriacan be split into two groups: criteria relating to the way donorsprovide funds, and criteria relating to donors policies andprocedures. Respondents ranking of the criteria, see box 1,suggests that the criteria relating to policies and procedures aremore important; six of eight criteria are rated by 70% or more ofthose who answered the question as highly important. Incomparison, only two of the seven funding criteria are rated as highas this. Respondents were also offered the opportunity to addadditional criteria that they believe to be important to donoreffectiveness. Few did this, suggesting that the list presented in thesurvey was generally seen to be adequate.



26/30

Funding Criteria Policies and Procedures Criteria- Disburses funds quickly - Facilitates the participation of

stakeholders in its work- Flexible in the types of funding provided - Is transparent in the way it makes

funding decisions- Makes long-term commitments - Is cost-effective

- Provides predictable funding - Undertakes constructive policydialogue

- Provides funds with low conditionality - Monitors and evaluates its workeffectively

- Provides untied aid - Programmes and projects aligned togovernments development priorities

- Provides highly concessional funds - Harmonises with the procedures ofother donors

- Uses government procurementprocedures

These criteria are presented in the order they were ranked by respondents,highest at the top.Box 1: Important additional criteria affecting donor effectiveness

10. Respondents perceive there to be little difference in theperformance of multilateral organisations against the ParisDeclaration. In aggregate there is limited differentiation betweenrespondents ratings of the different organisations against three ofthe Paris Declaration indicators, ownership, harmonisation andalignment. Where there are differences, the United Nations (UN)

agencies tend to be rated higher than the Banks and EuropeanCommission, though this is country specific.

11. Respondents rank the Asian Development Bank, UNDP andWorld Bank highest for both the funding and the policies andprocedures criteria. In contrast to the ratings received by the MOsagainst the Paris Declaration indicators, their rankings against thefifteen additional effectiveness criteria show more differentiation.Asian Development Bank, UNDP and World Bank receive a greaternumber of highest place rankings than the other organisations in

the survey. The European Commission receives only three highestplace rankings, the African Development Bank one and the GlobalFund for Aids, TB and Malaria receives none.

12. Respondents perceive there to be no difference in the overalleffectiveness of different multilateral organisations. Figure 3 showsthat respondents give equal ratings, in the range 3.30 3.52, foroverall effectiveness to the seven organisations studied.This finding is surprising for two reasons. First as described above,there is clear differentiation in respondents rankings of the MOsagainst the fifteen effectiveness criteria. Second, the results ofother methodologies assessing MOs find differences in their



27/30

effectiveness. Despite the limited differentiation against the ParisDeclaration indicators, it might be expected, therefore, that thedifferent MOs would receive different ratings for their overalleffectiveness. That they do not suggests that the respondents tookinto account other factors as they assessed multilateral

organisation effectiveness.

13. Respondents have complex views about the factors thatinfluence the effectiveness of different multilateral organisations.Analysing the data suggests that respondents take different factorsinto account for different MOs when they assess effectiveness. Forexample, for the majority of organisations in the study, there is anassociation, although weak, between the promotion of governmentownership and rankings of overall effectiveness, i.e. respondentsgiving higher ratings for the extent to which an organisationpromotes government ownership are also more likely to give it ahigher rating for overall effectiveness. A number of associationsbetween the fifteen effectiveness criteria and overall effectivenesswere also found. But these associations were all weak, raising thepossibility that respondents were taking into account a range ofother factors as they thought about MO effectiveness.Respondent open-ended comments on the questionnaire providefurther insight into the factors they appear to be taking intoaccount when assessing overall effectiveness.For example, thefocus of activities appears to be important. UNICEF and the GlobalFund for Aids, TB and Malaria are valued for their health focus; thetwo regional development banks, the World Bank, and, to a lesserextent the European Commission, for their focus on productivesectors and infrastructure; and UNDP for its technical assistance.The scale of activities also appears to be a factor, as the Banks andEuropean Commission receive more positive comments about theamount of money they can disburse, while the UN agencies receivealmost exclusively negative comments in this regard.

14. Respondents preferences for which multilateral organisationshould disburse additional aid appear to be independent of theirratings of effectiveness. Figure 3 shows that, unlike the ratingsgiven for overall effectiveness, respondents preferences for whichof the MOs should disburse additional ODA are clearlydifferentiated. In aggregate, respondents rank UNDP as firstpreference for disbursing additional aid. Comparing the multilateralorganisations that can disburse significant sums of extra financing,stakeholders preferred the two Regional Development Banks to theWorld Bank and European Commission. Analysing the data confirmsthat respondent preferences for disbursement are only weaklycorrelated with the overall effectiveness, the Paris Declaration and



28/30

other effectiveness indicators explored in the study. This suggeststhat other factors also influence disbursement preferences.

15. Governance appears to be an important influence ondisbursement preferences. Overall the African Development Bank is

rated poorly against most of the fifteen effectiveness indicators. Itis surprising therefore that it is preferred over both the EuropeanCommission and World Bank as a disbursement channel foradditional aid in three of the four African countries studied.Respondent comments suggest that one key reason for this couldrelate to the more equitable governance structure of the Bank. Thissupports the hypothesis that perceptions of ownership of theinstitution and its policies can trump perceptions about itseffectiveness.

16. Country context is important. The comments made byrespondents on the questionnaire provide a helpful insight into whythey gave the different MOs particular scores. A number of thecomments suggest that a broader context is important indetermining stakeholder perceptions of effectiveness. Thesecontextual factors will vary across countries but may include thehistory of the organisation in that country, the visibility of itsactivities, the type of activities it undertakes, as well as personalfactors such as the performance of the head of the agency incountry.

17. The pilot study has demonstrated that key stakeholders in aidrecipient countries want to have their voices heard.It invited theviews of senior individuals from five respondent groups. Theindividuals invited to complete the questionnaire are all very busyand at the beginning of the project there was a real fear that theywould not find the time to complete a questionnaire. In the end, themajority of respondents also contributed textual comments,totalling over 2300.



29/30

Box 2: Comments about multilateral organisationeffectiveness

The Global Fund for Aids, TB and Malaria and UNICEF's funding especially as

regards to HIV/AIDS and children, its effectiveness can be seen through activitiescurrently implemented in these two areas(Tanzania Civil Service)

The World Bank and African Development Bank are likely to favour the

productive sectors for growth promotion.(Tanzania Civil Society)

The UN and European Commission offer more than just funding, they build

capacity in government and civil society. They offer a package

(South Africa Parliament)

The World Bank with its high lending of fund is relatively effective than UNDPwhich with less funds is content to publish more reports(Business India).

UNICEF [] releases funds timely(South Africa Government)

The European Commission is very bureaucratic and takes a long time whencontracts are signed and when it is implemented. UNDP and World Bank are

much better, even though there are delays also(Ghana Business)

The World Bank's loan disbursement policy is guided by the head office. Theirprescription never give results for the LDCs. (Bangladesh Business)

The World Bank is sometimes bogged down with inflexible conditionalities.

(Tanzania Business)

The World Bank gives loans which have to be paid back at some future date.

We need more grants.

(Zambia Business)

The African Development Bank is directly accountable to African countries

(Tanzania Government)

The African Development Bank is a regional organisation and is more in touch

with continent issues and has a stake in its development.(South Africa Government)

I believe in the potential of the UN system and its political accountability.

(Ghana Civil Society)

Organisations such as the European Commission hide behind their regulationsforcing recipients to use their procedures arguing that these cannot be changedquickly because of their bureaucracy etc. These regulations impede efforts atattaining aid effectiveness.(South Africa Government)

What next?

18. If donors want to ensure that they have a full picture of MOeffectiveness, then systematic studies of stakeholder perceptionswill become increasingly important. The result that the surveyrespondents perceive no significant difference in overalleffectiveness is, at first sight, surprising. Other assessments ofeffectiveness might lead to the expectation that the World Bank, forexample, would rate higher than it does. The results from the pilotproject suggest that donor governments conceive and assesseffectiveness using a different range of factors and criteria to those

used by stakeholders in recipient countries. Bilateral donors placemore emphasis on outputs and outcomes, while recipient



30/30

stakeholders appear to place at least as much importance onownership and governance. Given the clear consensus thatownership of the development process is critical to aideffectiveness, understanding stakeholder perceptions must be acritical element of any overall assessment of donor effectiveness.

This difference in emphasis on the range of factors which could betaken into account while assessing effectiveness suggests that asingle tool is not a viable way to assess donor effectiveness.Instead a suite of tools will be needed to capture the variousdimensions of effectiveness necessary to obtain a complete view.These would include results based management tools, assessmentsof organisational processes, alongside different stakeholderassessments. Donors, recipient governments and key stakeholderswill then need to use the range of evidence generated to formbalanced judgements about which multilateral organisations to fundand engage with.

19. If donor governments wish to take stakeholder preferencesfor multilateral funding channels into account, then the resultssuggest that respondents have clear preferences. Surveyrespondents rated the African Development Bank relatively poorlyagainst the fifteen effectiveness criteria in the survey, yet theyprefer it as a disbursement channel to both the EuropeanCommission and World Bank. Comments drawn from thestakeholders suggest that perceptions of ownership and governanceare, at least in part, responsible for these preferences. Yet, donorswill find it hard to justify significantly increasing funding throughthe African Development Bank in the immediate future given therelatively mixed results from the various attempts to assess itseffectiveness. This suggests that bilateral donors need to workharder to assist the African Development Bank to build its capacityand increase its effectiveness in order to be able to provide biggervolumes of funding to it as the institution which stakeholdersappear to feel the greatest sense of ownership.

20. If the views and perceptions of recipient stakeholders become

part of bilateral donor assessments of effectiveness thenmultilateral organisations will need to pay closer attention to issuesof governance and ownership. The issue of country ownership willcontinue to be important. As highlighted above, it is likely that theviews of recipient stakeholders from inside and outside ofgovernment about donor performance will therefore become moreimportant and methodologies of this type will be increasinglyemployed by donors assessing MO effectiveness as part ofdisbursement decisions. Given that governance and ownershipappear to play an important role in shaping stakeholders views,

multilateral organisations wishing to improve their ratings will needto pay closer attention to these issues.

Documents

DFID-Review of Multilateral Effectiveness