EUROPEAN COMMISSIONJOINT RELEX SERVICE FOR THE MANAGEMENT OF COMMUNITY AND TO NON-

MEMBER COUNTRIES (S.C.R.)Resources, Relations with the other institutions, Evaluation and Information

EVALUATION OF E.U. DEVELOPMENT AID TO ALA STATES

Phase III – Synthesis ReportFinal Report

March 15, 1999

Società Italiana di Monitoraggio S.p.A., SIM (I)

This report was prepared with financial assistancefrom the Commission of the European Union.

The views expressed herein are those of theConsultant, and do not represent any official view

of the Commission.

EVALUATION OF EU DEVELOPMENT AID TO ALA STATESPHASE III: SYNTHESIS REPORT

Table of Contents

Pag. n.

EXECUTIVE SUMMARY i./xiii.

I. THE ALA EVALUATION 1Background 1The Data Constraints 1Country and Project Selection 2The Desk Study 2The Field Study 3Methodology for the Desk and Field Studies 3The Synthesis Report 3

II. THE EVOLUTION OF EC RELATIONS WITH THE ALA COUNTRIES 5

III. THE FINDINGS OF THE ALA EVALUATION 8Relevance 8Effectiveness 11Efficiency 12Impact 17Sustainability 18Dissemination of Information 20Co-ordination, Complementarity and Coherence 21

IV. REGIONAL CO-OPERATION AND CO-ORDINATION 23

V. RECENT DEVELOPMENTS THAT AFFECT THE QUALITY OFEC MANAGEMENT 26SEM 2000 26The External Relations Common Service (SCR) 26EC Delegations 29The ALA Committee 30The Quality Support Group (QSG) 32Internal Manuals 33

Pag. n.

VI. ISSUES AND RECOMMENDATIONS 34Recommendations to the Council of Ministers 34A Diversified Strategy for ALA Countries 34Clarification of Rationale and Conditions for Assistance toPriority Objectives 35Recommendations to the Council, the Member States and the Commission 351. Co-ordination, Complementarity and Coherence of Aid 352. Regional Co-operation 36Recommendations to the Commission Managers 36The Information Base 36The Project Cycle 37Aid Management within the EC 40Recommendation to the ALA Committee and Commission Managers 40

AnnexesI. Distribution of EC aid to ALA countries by countryII. Distribution of EC aid by objective by regionIII. Organigramme of the Relex Common ServiceIV. Comparative outline of CR 442/81 and CR 443/92V. Terms of reference for the ALA Synthesis ReportVI. Summary of main findings of Desk StudyVII. Executive Summary for the Field Study (Revised)VIII ALA Committee Questionnaire

LIST OF ACRONYMS

ADB Asian Development BankACP African, Caribbean and Pacific RegionsALA Asian and Latin American RegionsASEAN Association of South East Asian NationsAWPs Annual Work PlansBCF Business Co-operation FundBIDS Business Information and Development ServicesCECAP Central Cordillera Agricultural ProjectCEPAL The United Nations Economic Commission for Latin AmericaCPA Comprehensive Plan of ActionCPM Cycle Project ManagementCR Council RegulationCSP Country Strategy PaperDc Development co-operationDG Directorate GeneralEBICs European Business Information CentresEc Economic co-operationEC European CommissionECCP European Chamber of Commerce in The PhilippinesECIP European Community International Programme for Reintegration of

Vietnamese ReturneesECIP European Communities Investment PartnersEIB European Investment BankERP Earthquake Reconstruction ProjectEU European UnionEWG Evaluation Working GroupERP Earthquake Reconstruction ProgrammeFA Financial AgreementFONCODES Fondo Nacional de Compensacion y Desarrollo SocialFP Financial ProposalGDO Geographical Desk OfficerGDP Gross Domestic ProductHADP Highland Agricultural Development ProgrammeHoD Heads of DelegationINADE Instituto Nacional de DesarrolloIWP Interim Work PlanLF Logical FrameworkMBO Management By ObjectiveMSF/B Médecins Sans Frontiéres - BelgeM&E Monitoring and EvaluationMED South and East Mediterranean RegionsMICTI Ministry of Industry, Commerce, Tourism and IntegrationMPI Ministry of Planning and InvestmentMSI Mary Stopes InternationalMSs Member StatesNGO Non Governmental OrganisationNORLU Northern Luzon Co-operative Development CentreODI Overseas Development InstituteOWP Overall Work PlanPIT Tumbes Irrigation ProjectPPM Pilot Project MajesPWP Provisional Work PlanRRs Rules and RegulationsSECTI Secretaria Ejecutiva de Cooperaciòn Tecnica InternacionalSEM 2000 Sound and Efficient ManagementSMEs Small and Medium EnterprisesTA Technical AssistanceTDI Trade Development Institute of IrelandTO Technical OfficerToR Terms of Reference“3Cs” Co-ordination, Coherence, ComplementaryUNHCR United Nations High Commissioner for RefugeesUS United States of AmericaWB World BankWID Women In Development

i

Evaluation of EU Development Aid to ALA States

Phase III: Synthesis Report

EXECUTIVE SUMMARY

The ALA Evaluation

1. Origins. The Council of Ministers of the European Union (EU) decided, on June 1, 1995, to evaluate theEuropean Commission (EC) aid programmes1 to Asian and Latin American (ALA) countries, as well asto the Africa, Caribbean and Pacific region (ACP) and Mediterranean (MEDA) countries. The purposeof the evaluation is to help shape future policies and programmes for EC aid as a result of the lessonslearnt from the evaluation and improve the effectiveness and efficiency of aid procedures andinstruments.

2. Evaluation Constraints. The choice of countries and projects to be evaluated was affected by the scarcity ofdocuments about EC activities in ALA countries. In mid-July 1997, the EC was able to deliver less than100 evaluation studies for the ALA countries, of which only 45 contained sufficient information forevaluation in the Field Study. Yet EC aid to ALA countries had financed approximately 2000 projects on2 continents and in 33 countries for 20 years. The quality of information on crosscutting issues, such asgender, poverty alleviation, human rights, etc., was poor. Records about co-ordination with MemberStates are kept but these are spread throughout a large number of files, which makes them difficult toaccess. More positively, recent evaluations contain better quality information and country strategy papersare becoming increasingly available.

3. The Scope of the Work. In the Desk Study the consultants analysed the policies and instruments of the ECin all 15 Asian and 18 Latin American countries in the period 1986-1995. The Field Study scope wasdecided according to criteria discussed with the Evaluators Working Group (EWG). Peru and ThePhilippines were identified as countries with the capacity to act as full-fledged partners of the ECbecause of their relatively strong public administration and local authorities, and the availability of non-profit organisations with the capacity to reach final aid recipients. By contrast, Honduras and Vietnamwere chosen as examples of countries with more limited capability to absorb and manage externalassistance. The selection of only two countries on each of the two continents out of the thirty-threecountries that comprise the ALA region, limits the possibility of generalising about major issues, such asthe quality of country strategy papers, from the country findings to the region as a whole.

4. The Desk and Field Studies. For the desk study, the consultant reviewed documents at EC headquarters inBrussels in June and July and in September and October 1997. The Desk Study was distributed andapproved in December 1997. For the Field Study, the consultants visited Costa Rica (the regionalheadquarters for EC aid to Central America), Honduras, Peru, The Philippines, Thailand (where the ECdelegation used to manage aid to Vietnam until 1996), and Vietnam, during the first six months of 1998.The Field Study Report, including Annexes, was completed in November 1998.

5. The projects were purposely selected from as wide a range of sectors as possible. Project coverage waslimited to those initiated or completed between 1986 and 1995 and to projects that represented a varietyof management and performance issues, as well as a balance between revenue earning and socialdevelopment. No weight was given to performance outcomes. The sample of projects reviewedrepresents a small share of the total EC portfolio in each country and cannot be regarded as arepresentative sample of EC projects in any one country or in the ALA region as a whole.

6. The Synthesis. This report presents the Synthesis of the findings of the Desk and Field studies. The reportassesses the main strengths and weaknesses of the EC’s programmes and instruments, analyses theirunderlying processes, and highlights the key measures needed to improve the future effectiveness of ECaid to ALA countries. This report also includes new work done subsequent to the Field Study, including:

• Interviews of the members of the ALA Committee, through a questionnaire, to obtain their viewsabout their roles and responsibilities.

• A review of the EC role in regional integration in the ALA countries.

1 Although the EC is the aid-giving instrument of the EU, the term «EC aid programme» rather than «EU aid programme» is used in this report inorder to be able to differentiate when needed between the Community and the Commission. Similarly, the term EU is used throughout rather thandifferentiating historically between the EU and EEC.

ii

• A preliminary assessment of very recent changes within the EC that affect aid management.

Conscious of the small size of their project sample, the consultants have been careful to select only thosefindings that are common to most of the projects reviewed, when they generalise about aid managementin the ALA portfolio as a whole.

The Evolution of EC Relations with ALA Countries

7. The EC designation of Asian and Latin American countries as «ALA» is a terminology unique to the ECthat serves to classify its diverse aid recipients in Asia and Latin America into one category. No provisionwas made in the Treaty of Rome for tariff preferences or financial support for ALA countries. The ALAevaluation covers a period of evolution in EC policies towards these countries. Although aid to ALAcountries began in 1974, the first major policy statement was in 1981.

8. Council Regulation 442/81 defined the purpose of development aid to non-associated countries2,including ALA. The five objectives of EC aid were: i) to assist the poorest countries, ii) to improve theliving standards of the most marginalised strata of the population, iii) to promote rural development andagricultural production, iv) to promote a regional approach to development, and v) to meet humanitarianneeds in cases of natural disasters. EC programming was done annually, there was no continuity infunding of programmes, which were largely limited to poverty and humanitarian aid issues, with a focuson agriculture and, iii) rural production. Between 1981 and 1992, the EC expanded its ALA aid involume, instruments used and beneficiary sectors.

9. Regulation CR 443/92 in 1992 and subsequent regulations have broadened aid objectives and policies toembrace other key issues in addition to poverty, i.e., human rights, gender, democratisation and goodgovernance, environment and culture. The 1992 Regulation set aside 10% of the budget for the 1991-95period for environmental initiatives. The Regulation recommended that five-year programming shouldbe established for each objective, country or region, where possible. Economic Co-operation (Ec) wasspecifically added as a development tool. The EC would provide financial and technical grant assistanceto the poorest countries and neediest populations within countries, while those countries or regions withhigh economic growth rates would receive Ec.

10. A stronger body of EC policy towards ALA countries has only evolved since 1992. Given the time lagbetween policy formulation and its implementation, few projects in the evaluation portfolio reflect theimpact of CR 443/92.

EC Aid Commitments to ALA Countries

11. In 1995, EC aid commitments to Asia and Latin America were at a record level of 1.2 billion ECU, or17% of total allocable EC aid, as compared with under 300 million ECU in 1986. By comparison, aid toACP countries in 1995 amounted to 2.6 billion ECU and to 869 million ECU for MEDA countries.From 1991 to 1995, EC aid commitment to Latin America increased by 68% and that to Asia by 82%.

The Findings of the ALA Evaluation

12. The consultants applied the criteria of relevance, effectiveness, efficiency, impact and sustainability totheir analysis of each step of the project cycle for projects in the evaluation portfolio.

13. Relevance of aid to EC policies. The distribution of EC commitments as between Asia and Latin America in1986-1995 amounted to ECU 3,871 million to Asia and ECU 2,808 million to Latin America, a roughly60:40 split. Informally the EC has advocated the delivery of two-thirds of aid to Asian countries. In fact,Latin America receives a higher per capita allocation than Asia, so this is one poverty target that has notbeen attained. Both the Asia and Latin America regions were major beneficiaries of food andhumanitarian aid and assistance to and through NGOs, as well as for the social sectors. These are themajor areas targeted by the EC in its policy statements for the ALA region. In 9 out of the 13 projectsreviewed for the Field Study, development co-operation (DC) was specifically directed to extremely pooror socially excluded populations within countries, but, as previously stated, the consultants may not havereviewed a representative sample of projects. Environmental aid just attained the 10% of total aidallocation required for the 1990-1995 period. Aid allocations for gender issues are difficult to

2 The associated developing countries are the ACP countries

iii

disaggregate, though specific issues related to rural women can be identified mainly in Asia. The analysissuggests that the EC has only given prominence to issues such as gender and environment in ratherrecent years.

14. Relevance of aid to country policies. Because of annual programming and the lack of dialogue about countrystrategy, EC aid operations until 1992 were open to the criticism that the programmes lacked a sense ofownership by the governments and that the projects were designed in an ad hoc manner. While thedegree of participation in decision-making about EC programmes by governments depends on theircapacity and time availability, the EC has progressively introduced measures since 1992 to make thecountry policy dialogue participatory and the decision-making process transparent and collaborative,including specifically the preparation of country strategy papers. In the two most developed countries ofthe four that they visited, Peru and The Philippines, the consultants gained the impression that theseGovernments now (i.e., in 1998) consider themselves partners of the EC in the decisions that are takenabout the priority sectors and the stakeholders that will be recipients of EC aid.

15. Effectiveness. An assessment of whether the EC attained the objectives for its aid programmes as a wholein the ALA region has to be measured against very broad policy objectives contained in CR442/81 andCR 443/92. Furthermore, the consultants reviewed a relatively small slice of the EC’s aid programme ineach of the four countries visited and, therefore, an insignificant portion of the EC’s aid for the region asa whole. Also, the country and project selection process could not guarantee that the evaluation portfoliowas representative of the whole.

16. Despite the inefficiencies that characterised much of project preparation and implementation (discussedbelow), the projects reviewed did, in their majority, attain their targeted physical outputs. A major caveatis that the performance targets identified in the appraisal reports are limited to physical targets and donot address the wider issue of country strategy or the sustainability of the project.

17. Efficiency. The project analysis identified the most common weaknesses in EC aid management.

• The absence of country strategy papers and of long-term programming until the early nineties led toad hoc financing of projects.

• Only recently, as a result of a request by the ALA Committee, will the EC evaluate options beforeproject selection.

• The EC tends to compact identification, pre- and feasibility studies, project formulation and ex anteevaluation into a single and overly brief stage called preparation and appraisal. This crucialpreparation and appraisal stage of the project cycle has often not been well done in the past. Theappraisals suffer from a lack of time for more thoughtful analysis and data collection andpreparation, as well as the absence of discussion of the preliminary results with a peer group.

• Headquarters and delegation staff have perceived a tendency to select the lowest biddersystematically in EC project contracts. The immediate savings are obtained at the risk of loweringperformance standards and sometimes even the modification of the project to accommodate alower consultants’ budget. Given the complexity of the EC projects and the need for theirsustainability after implementation, the policy for awarding contracts needs to be reviewed.

• EC Financing Agreements are formal instruments, whose modification is burdensome even whenthe rationale for modification is irrefutable and the need to proceed is urgent. Difficulties in projectimplementation could often be traced back to the FA.

• EC projects suffered from two main problems during implementation—delays and poor projectmanagement. The former is often attributable to the FA and EC Rules and Regulations and thelatter to the EC tradition that two directors are appointed as joint project managers—one a nationaland the other a European expert.

• For project management to be effective considerable support is required from the country, whichrequires a sense of ownership that was not always present prior to 1992. The EC Delegations havenot been staffed to provide intensive support to project management in ALA countries. The weakmonitoring in EC projects has hindered mid-term adjustments in project implementation and hasalso limited the quality of ex post evaluations. This in turn limits the lessons that could have beenlearnt from previous aid projects. A main cause of weak monitoring is the lack of baseline dataagainst which project activities can be monitored.

iv

• Compounding the supervision weaknesses are the complex MIS system at EC headquarters and thelack of adequate systems, exacerbated by lack of manpower, for collecting, collating and internaldissemination of the findings in project reports.

• The paucity and poor quality of evaluation studies for ALA projects handicaps judgements aboutproject effectiveness and efficiency, sustainability and impact of most of the projects that wereevaluated by the consultants.

18. Impact. An in-depth evaluation of the impact of EC aid in the four countries visited by the consultants isnot possible because such studies were not done during the course of project implementation andevaluation. Nevertheless, based on the data available to them, the consultants consider that four out ofthe fourteen EC aid initiatives they evaluated had a substantial actual or potential localised impact. Theconsultants further conclude that EC interventions may have been too scattered in terms of sector andregion to have an impact at the national level.

19. Sustainability. A review of sustainability criteria: technical design, financial viability, institutionalsustainability, project management, and stakeholder participation, suggests that a pervasive weakness inEC aid projects has been the lack of attention to these factors. Sustainability is defined as the ability ofthe project to continue once the external financing is withdrawn.

Dissemination of Information

20. The consultants noted that the EC Delegation in Costa Rica, which has a regional mandate that includesHonduras, has established a well-staffed and proactive public information programme for the area. Theprogramme can be considered a model for information about the EU, as is the initiative of thePhilippines Delegation to set up a web site about EU operations that are of interest locally. Elsewhere,the generation of information about EC activities does not seem to have been effective, which may berelated to the dearth of data within the EC about ALA programmes.

Co-ordination, Complementarity and Coherence

21. The relative lack of success in implementation of «the three Cs» policy established by the Council ofMinisters may reflect lack of political will. Otherwise, it should not be too difficult to find ways to co-ordinate and complement EC and Member States activities. The responsibility for implementation of thethree Cs is a collective one by the Member States and the EC. Given the lack of success so far inimplementation of these policies, the Ministers may wish to take the matter up again not just with theCommission but also with the Member States.

v

Regional Co-operation and Co-ordination

22. Some 9% of EC aid commitments to Latin America directly supported regional programmes and 3.4%of EC aid to Asia directly supported regional programmes from 1986-95. The consultants attempted toidentify components that clearly support regional objectives and regional projects directly or indirectly inthe projects they evaluated. With specific exceptions, such as maternal and child care in Central America,it is not at clear that the EC takes regional objectives into account in designing national aid programmes,unless such projects are specifically designed as regional projects. By implication, this suggests thatGovernments also tend to ignore opportunities for promoting regional objectives in nationalprogrammes.

Developments since 1995 that Affect the Quality of Aid Management

23. The consultants identified recent changes in the EC’s structures and processes to evaluate whether theyaddress the main weaknesses in EC aid management discussed above. Most of these changes within theEC are very recent, some are ongoing, and, therefore, only preliminary judgements about their impactsand efficacy are possible.

24. SEM 2000. An internal drive for improved quality in EC operations began in mid-1995, known as‘sound and efficient management for the Year 2000». One output is the current reorganisation of thetechnical and financial staff. Whether or not induced by external pressures, the existence of an internal«change team» that keeps the institution’s objectives, policies and administration under continuingreview normally creates a healthy organisation.

25. The External Relations Common Service. The current reorganisation of staff from the geographicalDirectorate Generals (DGs) into a «common structure for the implementation of Community aid tothird countries» is intended to lead to economies of scale and a greater degree of responsibility by allconcerned in the management of EC aid programmes. The new SCR results from the merger oftechnical and financial staff previously responsible for such work in DG 1, DG 1A, DG 1B, and DGVIII. The technical staff, however, is still divided geographically in the new SCR between the EC’straditional regional groupings, e.g., ALA and MED in Direction B and ACP in Direction C. Certaincross-sector aid themes, such as NGOs, environment, gender issues, human rights, drugs, etc. areconcentrated in one or other Direction, which then has global responsibility for these themes.

26. The considerable emphasis on evaluation during this restructuring addresses a major weakness in theEC’s management of the project cycle. A separate Evaluation Unit has been created that has globalrather than regional responsibility for evaluation. Currently, the Unit has nine evaluators, which includesthree external experts on short-term contracts, in addition to the head of the Evaluation Unit. Thiscompares with ten evaluators and three heads of units in the previous DG IA, IB and VIII evaluationstructure. While the concept of integrating evaluation resources is good, it is difficult to see how the Unitcan effectively evaluate the EC’s development aid programmes world-wide or help to ensure thatcommon standards and methodologies are applied at all stages of the project cycle with this level ofstaffing. Therefore, recommendations in this report for improved monitoring and evaluation, inparticular, will be difficult for the Evaluation Unit to implement. The Unit will have to ensure thatappropriate monitoring systems are included in project preparation to facilitate later work on evaluation.The Unit will also need to expand its internal resources by using experienced consultants frequently. In awelcome development, the SCR plans to post on a web site to be constructed by April 1999 thesummaries and full texts of all completed evaluations.

27. Project management may be adversely affected by the absence of formal links between the geographictechnical departments and, therefore, by no obvious means of cross-fertilisation of ideas. The separationof project specialists by geographic units, when some of the units are very small, raises the question ofwhether economies of scale have indeed been achieved. Would it be more effective to group, forexample, all the infrastructure specialists in one department and give them responsibilities for all theEC’s programmes worldwide? That in turn would require that a mechanism be established to ensure thateach geographic DG had access to specialists on an equitable basis.

28. A practical question is whether topic specialists in specific geographical units, e.g. NGO specialists in theACP country Directorate (C), will respond with equal priority to demands for their services from otherregional Directorates.

29. A major question is whether the sharing of responsibilities during the project cycle between thegeographic DGs and the SCR will work, or whether it will lead to overlaps and/or underlaps. As withrelations with the Delegations, the sharing of responsibilities during the project cycle will work smoothly

vi

only if the managers of the geographic DGs and the SCR provide leadership and create incentives forEC staff that encourage co-operation.

30. EC Delegations. The consultants’ assessment of relations between the Delegations and Headquarters werethat: «Headquarters relies heavily on delegations for technical and administrative tasks, withoutdelegating enough power to resolve situations locally. On the other hand, the Delegation’s countryknowledge and insight may be under-utilised in functions such as country strategy formulation, projectappraisal, tailoring the Framework Agreement to country conditions». The consultants singled out Peruas a Delegation that is successfully involved in project management and has enhanced the image of theEC/EU in the process. In the EC’s ongoing decentralisation process, numerous initial problems willarise in ensuring that delegation of real responsibilities occurs and that it works properly. The respectiveEC managers will need to keep this situation under review.

31. The ALA Committee. The Committee is made up of representatives of fifteen Member States, who meetmonthly for one day to review and approve all EC programmes exceeding ECU two million. As a resultof its deliberations over time, the ALA Committee requested that alternative aid initiatives should bediscussed in each proposal presented for financing, to ensure that the project selected is the highestavailable priority. The ALA Committee also noted the lack of regularity in presentation of Mid-termReviews of project implementation.

32. Country strategy papers covering a 3-5 year period are now required for EC aid recipients. TheCommittee is starting to review these in order to assess the context in which projects are prepared andfinanced. In tandem, the Commission will prepare longer-term indicative budgets and programmes foreach country or regional programme, which the Committee will also review.

33. In an attempt to understand better the work of the ALA Committee, the consultants sent aquestionnaire to the thirty-three Committee members and received twelve replies from ten countries. Bya slight majority the Committee members stated that the most important aspect of their work wasapproval of projects. One member noted the difficulty of changing project proposals at the Committeestage as a result of the Committee’s comments because the meetings occur too close to implementation.An only slightly lower number saw their role as a means to foster co-ordination, complementarity andcoherence, though one Member stated specifically that it did not provide the best forum for pursuit of«the three Cs». With one exception, all the Members expressed satisfaction with the Commission’sparticipation in the Committee, though a minority said that the Commission’s follow up to thecomments was not always satisfactory.

vii

Summary of Recommendations

Recommendations to the Council of Ministers

1 - A Diversified Strategy for Aid to ALA Countries

ISSUEImportant issues related to the ALA countries are their heterogeneity, the breadth of the policy frameworkfor ALA aid and the lack of diversified strategies for aid delivery among countries at very different incomelevels and with different patterns of income distribution. The timing is appropriate for a thorough review ofthe EC’s policies towards and relations with the ALA countries.

RECOMMENDATIONS• the EC should re-evaluate the objectives, delivery systems and expected outcomes from EC• assistance to the ALA countries• the EC could recommend strategies that differentiate between ALA countries at different• income levels or with large segments of poor people, or that are in the process of regional• integration• the EC should review the rationale for its blanket grant assistance even for priority objectives• to all ALA countries• more thought may need to be given to ways in which Ec can effectively support a• diversification of aid strategies within ALA countries and in the best interests of those• countries• an analysis should be done by the EC to assess the reasons for and possibly correct the• apparent bias in assistance to Latin America.

2 - Clarification of Rationale and Conditions for Assistance to Priority Objectives

ISSUEAn overriding objective of EC aid, incorporated in every Council Regulation, is the reduction of poverty.Over the years the Council has added other objectives: gender, NGOs, environment, food and humanitarianaid, regional cooperation, democratisation and governance, trade, etc.).

RECOMMENDATIONS• the number of objectives and the lack of clear guidance about the priorities among them now require

that there be a re-examination of the targets of EC assistance in ALA countries• once the Council has established new policies for EC aid to ALA countries, EC managers should ensure

that clear guidelines are issued promptly to staff to ensure that the Council regulations can beimplemented efficiently.

viii

Recommendation to the Council, the Member States and the Commission Managers

1 - Co-ordination, Complementarity and Coherence of Aid

ISSUEThe responsibility for implementation of “the three Cs” is a collective one by the Member States and the EC.The consultants found little evidence of co-ordination and complementarity of aid by the Member States withthe EC. The relative lack of success in implementation of “the three Cs” may reflect lack of political will.

RECOMMENDATIONSGiven the lack of success so far in its implementation, the Ministers may wish to take the matter up againamong themselves and within the countries they represent. As a first step, the Ministers could ensure that theconcepts are clearly defined and that implementation guidelines are drawn up and widely circulated tonational aid agencies and their representatives in beneficiary countries, as well as to EC Delegations and staff.

2 - Regional Co-operation

ISSUEThe EC in Brussels and the EC Delegations in aid recipient countries have a major responsibility for helpingto design programmes and projects that further the objectives of countries that have formalised theirintentions for regional co-operation. The Member States of the EU are responsible for reviewing ECproposals for regional policies and, once approved, for co-ordinating their aid policies with the EC.

RECOMMENDATIONSThe EC and MSs should review in greater depth the processes for co-ordinating and/orcomplementing efforts. It may well be that the processes that are likely to work in one region may not be theright ones to adopt in another. The holding of a major consultative group meeting is recommended prior toEC support for a regional programme to plan the sectors or local regions where each MS will focus its aidefforts.

Recommendation to the Commission Managers and the ALA Committee

The Responsibilities of the ALA Committee

ISSUEMembers of the ALA Committee expressed concerns about their responsibilities in their responses to theconsultants’ questionnaire.

RECOMMENDATIONSThe ALA Committee and Commission managers should establish a dialogue to find ways toenhance the contributions of the Committee to the EC’s aid management.

ix

Recommendations to the Commission Managers

1 - The Information Base

ISSUEThe serious absence of information about ALA aid initiatives not only constrained the quality of this ALAevaluation but also has prevented and probably still prevents internal analysis of the effectiveness of ECpolicies and programmes for ALA countries.

RECOMMENDATIONS• It is essential that the EC improve systematically the generation, collection and collation, and regular

dissemination of information. Appropriate information technology systems that capture data generatedby EC activities immediately and disseminate it for information, analytical, monitoring and evaluationpurposes, as well as the creation of an on line handbook of best practices and lessons learnt, should becreated within the EC as a major priority

• The consultants recommend that, in future, the findings of ALA committee reviews be recorded in adatabase, categorised and periodically analysed by EC staff, as a continuous input into improved designof country strategy papers and projects.

• The EC should ensure that the information base about its operations in ALA countries is complete,current and easily accessible. Given the size of its aid flows and its global importance, the EC needs toimprove its databases urgently. Much of the information that the EC requires to improve its operationsshould be produced routinely and as a by-product when policies, programmes and projects aredeveloped, approved, implemented and, in the case of programmes and projects, completed andevaluated.

2 – The Project Cycle

ISSUEThe extensive analysis of projects at each step of the project cycle undertaken in this evaluation indicates theweaknesses frequently apparent in the project cycle in ALA countries.

RECOMMENDATIONS• a methodology, with guidelines, should be developed to improve the quality of country strategy papers.

The EC will have to continue to improve its dialogue with Governments about priority projects andprogrammes through joint preparation of the CSPs, which should conform to the country’sdevelopment priorities. The comments of the ALA Committee about such strategy papers should becollated for periodic review by EC evaluators or consultants. In addition, the Delegations and EC staffshould provide, in a format that can supplement the ALA Committee’s initial review, regular monitoringof the CSPs during the course of project implementation.

• as a result of a recent request by the ALA Committee, the EC will evaluate options before projectselection. During project selection, there should be greater emphasis on the project context, which ispartly resolved by the now required country strategy papers, but also needs a better understanding of thesector context and the reasons for existing projects’ failure or success

• project preparation and management budgets and project cost coefficients must be adapted toaccommodate more careful project preparation, including stakeholder participation, appraisal andmonitoring. At the same time, managers of the earliest projects that are based on a LogFrame approachto project management should be asked to record constraints on monitoring caused by the EC’sadministrative or budgetary procedures; monitoring is the key to informed project management and thebasis for an effective evaluation

• given the complexity of the EC projects and the need for their sustainability after implementation, thepolicy for awarding contracts needs to be reviewed, so that they are not given virtually automatically tothe lowest bidder

• to avoid exogenous constraints on project implementation, the FAs and R&Rs that govern projectadministration must be adapted to facilitate and not hamstring procedures for project implementation. Itis urgent that a solution be found quickly to this recurring problem.

x

• to resolve problems in project management, the consultants recommend that the national expert isappointed as the Project Director and the European expert as an Advisor

• the Commission’s objective of decentralising evaluation has been achieved but the critical mass ofexpertise created may be too small to be effective and to implement recommendations for extending therole of evaluation within the EC. For example, the consultants believe that the EC should beundertaking more cross-sector analysis that targets issues that are common to projects in a variety ofsectors, e.g., financing of recurrent cost. Another area of interest is the objectives that cut acrossindividual sectors, like gender and poverty. The purpose of such evaluations would be to build up a bodyof experience with these project elements and to track compliance with EC objectives. But, anysuggestions for an increase in responsibilities of the Evaluation Unit either require an increase in staff ora redesign of the Unit’s traditional way of doing business.

3 – Aid Management within the EC

ISSUEThe need to strengthen aid management within the EC

RECOMMENDATIONS• the consultants recommend that a Quality Support Group similar to that already established for DGVIII

and consisting of peer reviewers from within the EC should be created to review project proposals forthe ALA countries. This would help to improve the quality of the proposals presented to the ALACommittee at a time when that Committee is shouldering new responsibilities for review of CSPs andsome members believe their inputs are made too late in the process to effect change. With Delegationsmore involved in support to implementation, these combined efforts should result in more relevantprojects that are more efficiently managed and that result in an improvement of the cost-effectiveness ofEC aid.

• changes in aid management can be recommended and even adopted but will require leadership by ECmanagers to ensure that the reforms are institutionalised. The creation of the Common Service and thedecentralisation of responsibilities to the delegations should have a positive impact on projectmanagement. There are likely to be numerous initial problems in ensuring that the sharing anddelegation of real responsibilities occurs between the SCR and the geographic DGs and with theDelegations. To achieve the targeted outcomes will require leadership by the respective EC managersand the creation by them of incentives to staff that encourage co-operation and the sharing ofresponsibilities for project management.

1

Evaluation of EU Development Aid to ALA States

Phase III: Synthesis Report

I. THE ALA EVALUATION

Background

1. The Council of Ministers of the European Union (EU) decided, on June 1, 1995, to evaluate theEuropean Commission (EC) aid programmes3 to Asian and Latin American (ALA) countries, as well asto the Africa, Caribbean and Pacific region (ACP) and Mediterranean (MED) countries. The purpose ofthe evaluation is to help shape future policies and programmes for EC aid as a result of the lessons learntfrom the evaluation; improve the effectiveness and efficiency of aid procedures and instruments; andprovide information to Member States (Member States) about the implementation the EC’sdevelopment aid policy.

2. The Inception report for the ALA Evaluation, as well as the Terms of Reference, were approved by theEvaluators’ Working Group (EWG), comprising representatives of Member States and staff of the EC.The evaluation studies have been done in three stages: a Desk Study, Field Study and the present report,which contains a synthesis of the findings of the Desk and Field evaluations of EC aid to ALAcountries.

The Data Constraints

3. The choice of countries and projects to be evaluated was affected from the outset by the relative scarcityof documents about EC activities in ALA countries. Even in mid-July 1997, when the evaluation exercisefor ALA countries began “no centralised archive or systematic mapping of relevant documents and dataon policy, procedure and programme implementation processes was available at DG1B” (ALA DeskStudy, page 56). A special effort was made within DG1B (the Directorate responsible for ALA aid) tocollect documents and the beginnings of a filing system for legal and technical documents and anevaluation library was created. Nevertheless, the ALA consultants have the impression “that someinformation gaps referring to past programmes, projects, management and procedures may never beeliminated”. The Overseas Development Institute (ODI) inventory of projects, which preceded the ECevaluation exercise, was extremely helpful to the evaluators.

4. In mid-July 1997, the EC was able to provide the Desk Study evaluators with less than 100 evaluationstudies for the ALA countries. Yet EC aid to ALA countries had begun over twenty years before,comprised the financing of approximately 2000 projects on two continents and in 33 countries. Theconsultants found that only 45 of the 100 evaluations available contained sufficient information forevaluation in the Field Study and noted that “recent evaluations give better quality information than oldones, benefiting largely from the progressive adoption of the LogFrame approach.” The evaluatorsnoted that the quality of information on crosscutting issues, such as gender, poverty alleviation, humanrights, etc., was generally poor and records about co-ordination with Member States were widelyscattered as to be practically unavailable. Country studies were only just becoming available in 1997. Ingeneral, the availability of evaluations and other information about EC aid programmes in Latin Americawas greater than for Asia - for reasons that the evaluators were unable to determine.

Country and Project Selection

5. In the Desk Study the consultants analysed the policies and instruments of the EC in all 15 Asian and 18Latin American countries in the period 1986-1995. The Field Study scope was decided according tocriteria discussed with the Evaluators Working Group (EWG). Peru and The Philippines were identifiedas countries with the capacity to act as full-fledged partners of the EC because of their relatively strongpublic administration and local authorities, and the availability of non-profit organisations with thecapacity to reach final aid recipients. By contrast, Honduras and Vietnam were chosen as examples ofcountries with more limited capability to absorb and manage external assistance. The selection of onlytwo countries on each of the two continents out of the thirty-three countries that comprise the ALAregion, seriously limits the possibility of generalising about major issues, such as the quality of countrystrategy papers, from the country findings to the region as a whole.

3 Although the EC is the aid-giving instrument of the EU, the term “EC aid programme” rather than“EU aid programme” is used in this report in order to be able to differentiate when needed betweenthe Community and the Commission.

2

6. Project coverage was limited to those initiated or completed between 1986 and 1995. The consultantsanalysed fourteen projects in all: five in Honduras, and three each in Peru, The Philippines and Vietnam.Projects were chosen to represent as wide a range as possible of management and other performanceissues in development co-operation (DC), as well as a balance between revenue earning and socialdevelopment. An essential requirement was that enough documents should be available for an evaluationto be feasible. No weight was given to performance outcomes. The DC projects reviewed included socialservice and emergency humanitarian initiatives in Honduras and Vietnam managed by non-governmentalorganisations (NGOs). The criteria were designed to identify a diversified portfolio of projects, butcould not ensure that the projects are a representative sample of EC aid projects in ALA countries. Theconsultants are conscious of the small size of their project sample and, therefore, have been careful toselect only those findings that are common to most of the projects reviewed to generalise about the ECALA portfolio as a whole.

7. Given the relatively small number of Economic Co-operation (Ec) projects initiated or concluded inALA countries in 1986-1995, as well as the limited amount of information available for these, only oneinitiative in The Philippines was included in the portfolio prepared jointly by the EWG and theconsultants. Similarly, European Investment Bank (EIB) projects were not included in the evaluationportfolio.

3

The Desk Study

8. The Desk Study comprised a review of documents at EU headquarters in Brussels in June and July andin September and October 1997. Annex VI contains a summary of the Desk Study findings updatedafter the Field Study. The Desk Study was distributed and approved in December 1997.

The Field Study

9. The Field Study was undertaken during the first six months of 1998 with visits to Costa Rica (theregional headquarters for EC aid to Central America), Honduras, Peru, The Philippines, Thailand (wherethe EC delegation used to manage aid to Vietnam until 1996), and Vietnam. After the fieldwork, theconsultants continued to receive material requested during their visits about the aid programmes fromthe Governments and Delegations in the countries concerned and from the EC in Brussels. The FieldStudy Report was published in two volumes in November 1998, consisting of a Final Report andAnnexes.

Methodology for the Desk and Field Studies

10. The Desk Study evaluated the data available for ALA programmes using the criteria provided by theCouncil of Ministers. The chief ones are: relevance, effectiveness, efficiency, impact, and viability orsustainability.

11. The Field Study analysed each step of the project cycle for each of the projects reviewed. This reviewresulted in overall conclusions about the EC’s aid management and the improvements that are requiredto make that aid more effective, efficient and, particularly, more sustainable.

12. The consultants assessed the extent to which EC aid programmes during the period under reviewcomplied with the Council of Ministers’ requirement for Co-ordination, Coherence andComplementarity - “the three Cs”. The concepts are considered individually in Chapter III of thisreport.

13. In addition to the country analyses, the consultants undertook a review of the EC’s Rules andRegulations (R&Rs) to understand better some of the underlying issues in project implementation. Theconsultants’ full recommendations about the R&Rs are in Annex 2 of the Field Study.

The Synthesis Report

14. The Synthesis report is essentially a fusion of the findings of the Desk and Field studies. The reportassesses the main strengths and weaknesses of the EC’s programmes and instruments, analyses theirunderlying processes, and highlights the key measures needed to improve the future effectiveness of ECaid to ALA countries. This report also includes new work done subsequent to the Field Study.

15. The consultants interviewed, via a written questionnaire, the members of the ALA Committee todetermine their role in quality control for individual projects and the impact of country strategies onpolicy formulation and project selection. The results from these questionnaires are tabulated andanalysed in this report.

16. The consultants reviewed the EC role in regional co-operation. A chapter on that subject is included inthis report.

17. The changes in internal EC structures resulting from the current reorganisation are too recent to be ableto undertake an evaluation of outputs. Nevertheless, the consultants attempted to assess the likely impacton EC aid management of the creation of the External Relations Common Service (SCR) and thesimultaneous process of decentralisation and strengthening of EC Delegations that was begun in 1998.

18. In summary, this Synthesis report presents the Consultant’s major findings about EC aid managementand the decision-making process, based on data published for the two regions and those that areavailable within the EC, and on an evaluation of selected projects initiated or completed between 1986and 1995. The report also attempts to assess the extent to which changes within the EC and in itsproject management procedures since 1995 have had an impact on the aid management issues identifiedby the consultants during the evaluation period.

4

II. THE EVOLUTION OF EC RELATIONS WITHTHE ALA COUNTRIES

1. The EC designation of Asian and Latin American countries as “ALA”, is a terminology unique to theEC that serves to classify its diverse aid recipients in Asia and Latin America into one category. Noprovision was made in the Treaty of Rome for tariff preferences or financial support for ALA countries,unlike the ACP countries. The Treaty of Rome did not make any specific provisions for theMediterranean countries either, but the EU4

has gradually expanded its economic and political sphere ofinfluence in the Mediterranean, so that the MED countries now have a thirty-year history of formalinstitutional links to the EC.

2. EC aid to ALA States was initiated in July 1974 with a Council resolution that confirmed “the principleof Community financial and technical aid to non-associated developing countries5”. The fundamentalcharacteristics of aid to ALA countries, as distinct from that to the associated states, were: annualprogramming and ad hoc policy formulation, financing exclusively from the EC budget, and an emphasison humanitarian and mostly emergency aid. With exceptions, the programme reflected the EC’spreference to concentrate its limited resources on the poorest countries. By 1976, a modest programmeof financial and technical co-operation had begun to benefit ALA countries, which are listed below byregion.

ALA countries by sub-regionsCENTRAL & EAST ASIA SOUTH-EAST ASIA CENTRAL AMERICA SOUTH AMERICAChina Cambodia Costa Rica ArgentinaMongolia Indonesia Cuba Bolivia

Laos El Salvador BrazilSOUTH ASIA Philippines Guatemala ChileBangladesh Thailand Honduras ColombiaIndia Vietnam Mexico EcuadorNepal Nicaragua ParaguayPakistan MIDDLE EAST Panama PeruSri Lanka Yemen UruguayBhutan Venezuela

3. In 1981, Council Regulation 442/81 defined the purpose of development aid to non-associatedcountries, including ALA. The five objectives of EC aid were: i) to assist the poorest countries, ii) toimprove the living standards of the most marginalised strata of the population, iii) to promote ruraldevelopment and agricultural production, iv) to promote a regional approach to development, and v) tomeet humanitarian needs in cases of natural disasters. The Commission set the budgets for the initial‘first’ and ‘second’ generation bilateral agreements with Latin American countries annually and there wasno contractual commitment as under the Lome agreements. During the 1980s, the EC began tonegotiate framework agreements with individual Asian countries to promote trading relations, but also toidentify and meet specific development needs in a sustained rather than ad hoc manner. Between 1981and 1992, the EC expanded its ALA aid in volume, instruments used and beneficiary sectors. Whengovernments could not be counted upon to share EU values, the EC began to reach populations directlyvia NGOs. As discussed in Chapter IV, the EC also supported regional bodies and supra-nationalorganisations.

4. The Maastricht Treaty had important implications for EC policies for the ALA countries. The Treatyemphasised the need for greater co-ordination of aid between the Member States and the EC and thisled to the specification that there should be greater Coordination, Coherence and Complementarity (thethree Cs) between the countries and the Commission. Article 130u of the Treaty defined the policy goalsfor DC with developing countries as: sustainable economic and social development, smooth and gradualintegration of the recipients into the world economy, and reduction of poverty.

5. The 1992 Council Regulation (443/92) defined the types and terms of financial and technical assistanceto the ALA countries. Annex IV contains a comparison in tabular form of the differences between CR442/81 and CR 443/92, which illustrates visually the changes and expansion that occurred. CR 443/92affirmed that human development is the aim of development co-operation and required that financialand technical aid should be directed principally to the poorer levels of the population, as well as for thepoorest countries. Among the provisions of the Regulation are: promotion of the role of women,conservation of the environment, support for efficient public management, development of the ruralsector and improvement of food safety, and improvement of the legal and social framework for theprivate sector, including for micro and small- and medium-scale enterprises. The Regulationrecommended that five-year programming should be established for each objective, country or region,

4 The term EU is used throughout to avoid differentiating historically between the EU and EEC.5 The associated developing countries are the ACP countries.

5

where possible. As a consequence, the EC expanded its aid programmes from a narrow focus ongovernments and public bodies to a broader array of institutions and the private sector.

6. CR 443/92 specifically added Economic Co-operation (Ec) to its development promotion instruments.Financial and technical assistance would be given to the poorest countries and neediest populationswithin countries, while Ec would be provided to those countries or regions with high economic growthrates. The Regulation also broadened aid objectives and related policies to embrace other key issues inaddition to poverty, i.e., human rights, gender, democratisation and good governance, environment andculture. The 1992 Regulation set aside 10% of the budget for the 1991-95 period for environmentalinitiatives. In the 1990s, more ambitious “third generation” agreements were signed with most LatinAmerican countries, which included clauses designed to safeguard democratic principles and providescope for expansion of DC.

7. In 1995, EC aid commitments to Asia and Latin America were at a record level of 1.2 billion ECU, or 17% of total allocable EC aid, as compared with under 300 million ECU in 1986, as shown in the table thatfollows.

1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 86-95ASIA 140 257 226 426 317 383 470 504 451 696 3870LATIN AMERICA 160 156 159 210 222 286 338 401 390 486 2808ALA TOTAL 300 413 385 636 539 669 808 905 841 1182 6678% annual change 38% - 7% 65% - 15% 24% 21% 12% - 7% 41%

By comparison, aid to ACP countries in 1995 amounted to 2.6 billion ECU and to 869 million ECU forMED countries. Despite the substantial increase in amount of aid, ALA’s share of total EC aid, declinedfrom 15.3% in 1986-90 to 13.6 % in 1991-1995, but this reflects the growing absorption of EC aid since1990-91 by the Central and East European countries (CEECs) and the Newly Independent States (NIS).From 1991 to 1995, EC aid to Latin America increased by 68% and that to Asia by 82%. The increasereflects the enlargement of the range of aid, the use of new instruments and the incorporation of newcountries, including those, like Vietnam, that abandoned practices, such as violations of human rights,which had previously prevented EC assistance.

8. EC “programme aid” includes assistance for structural adjustment, which is provided for imports inkind or in foreign currencies to support the central budget of recipient countries. Most of theseconcessionary funds benefit the ACP countries, as initiated under Lomé III and have been financedfrom the European Development Fund (EDF), though a small amount of structural adjustment supportwas allocated to Mediterranean countries from the EC budget in recent years. Structural adjustment isnot part of the EC mandate for ALA countries.

9. The evaluation of ALA projects covers a period of evolution in EC policies towards these countries.From 1986 to 1992, EC programming was done annually, there was not necessarily any continuity infunding of programmes and they were largely limited to poverty and humanitarian aid issues, with afocus on agriculture and rural production. Since 1992, the objectives of EC assistance have broadened,Ec has been added to policy instruments and multi-year programming, with more participation by thegovernments and stakeholders, is being introduced in all countries. A stronger body of EC policytowards ALA countries has, therefore, only begun to evolve since 1992. Given the time lag betweenpolicy formulation and its implementation, few projects evaluated for this report reflect the impact ofCR 443/92. This evolution in EC policies has occurred simultaneously with the growing internationalinfluence of the EU as a financial leader on the world stage and, as a consequence, the Community’sincreasing share of development assistance and trade.

6

III. THE FINDINGS OF THE ALA EVALUATION

1. The ALA evaluation comprised a review of the policies and instruments of the EC in 15 Asian and 18Latin American countries in the period 1986 to 1995 and a review of fourteen projects that were initiatedor completed in the same period in four countries. The assignment faced several constraints that need tobe highlighted at the outset.

2. Without formal protocol agreements for the ALA countries only broad definitions of the objectives ofEC lending for these countries, which mainly relate to poverty, are available from 1986 to 1992. TheCouncil of Ministers formulated a more detailed statement of EC lending objectives for the ALAcountries in CR 443/92. Therefore, an assessment of whether the EC attained the objectives for its aidprogrammes as a whole in the ALA region has to be measured against very broad baseline policyobjectives for the period 1986 to 1992. Furthermore, the consultants reviewed a relatively small slice ofthe EC’s aid programme in each country visited and, therefore, an insignificant portion of the EC’s aidfor the region as a whole. Also, the country and project selection process could not guarantee that thesewere representative of the whole. The lack of availability of basic documents and specifically projectevaluation studies seriously handicapped the analysis. From 1992, the EC issued more specific policyobjectives relating to environmental and gender issues among others, and began a process of improvingaid management procedures, which, given the delay between policy formulation and implementation, arelargely not reflected in the projects evaluated.

3. Despite the evaluation difficulties that each faced, the Desk Study and the Field Study reached similarconclusions. With the caveats noted above, the consultants evaluated EC decision making and aidmanagement based on the criteria of: Relevance, Effectiveness, Efficiency, Impact and Sustainability andthese criteria are discussed below.

Relevance

4. The criteria of relevance refers both to the conformity of the aid with the policy objectives of the EC and to its consistencywith the development aims of the recipient states.

5. Relevance to EC Policy Objectives. The distribution of EC commitments as between Asia and Latin Americafor the period amounts to ECU 3871 million to Asia and ECU 2808 million to Latin America, a roughly60:40 split. The informal EC strategy has always been to deliver two-thirds of EC aid to Asian countriesand the balance to Latin American countries within ALA. If population size in the two regions werefactored in, such a distribution would result in a more equal per capita distribution. In fact, LatinAmerica benefits more than Asia from EC aid on a per capita basis, so this is one target that has notbeen attained.

6. Annex I and II show the distribution of EC aid to the ALA region by country and objective. Both theAsia and Latin America regions were major beneficiaries of poverty-related assistance for food aid,humanitarian projects and assistance to and through NGOs, as well as aid for the social sectors. Theseare the major areas targeted by the EC in its policy statements throughout the evaluation period for theALA region. In 9 out of the 13 projects reviewed for the Field Study, DC was specifically directed toextremely poor or socially excluded populations within countries, but, as previously stated, theconsultants may not have reviewed a representative sample of projects for the ALA countries as a whole.

7. The EC has also advocated through its aid programmes: human rights, good governance, politicalpluralism, gender parity and environmental protection, as well as poverty. From 1991, the EC requiredthat environmental considerations be specifically addressed in all aid activities and under CR 443/92 thata target of 10% of Community aid should be allocated to environmental activities, including forestry.While aid for both directly environmental and forestry and natural resource initiatives grew significantlyin the 1990s from a low base in both Asia and Latin America, the actual allocations are difficult todisaggregate. According to estimates made for the Desk Study, the environmental allocation just reached10% of total aid in the 1990-1995 period, with a higher proportion going to Asia than Latin America.

8. The projects reviewed provided scant information on EC concerns for environmental issues. The mostecologically-sensitive initiatives, the irrigation projects in Peru, presented the same challenge as anyproject in very arid lands, which require provision of continuing water flows and not just temporary,selective or occasional flows as in temperate areas. Appropriate engineering aspects were considered inthe project design; some soil salt saturation was to be expected and it was taken care of when it occurred.The consultants believe that environmental issues were not prominent per se in the design of ECprojects until CR 443/92, but note that provisions for environment protection are included in currentEC requests for tenders.

7

9. By contrast with a number of other technical agencies, the EC seems to have lagged behind inhighlighting environmental issues in the early nineties. One UN agency, UNEP, is totally devoted topolicy and assistance in this field and predated this decade’s initiatives. The UNDP and certaininternational and regional lending institutions, as well as many of the Member States, have beenoperationally committed to environment protection since the early ‘90s or even before. All their aidoperations have to be screened for their environmental components and a significant portion oftechnical assistance is dedicated to environmental programmes, with EIAs a sine qua non for projectpreparation and appraisal.

10. Similarly, the EC allocations for gender are also difficult to disaggregate, though specific projects, manyrelated to helping rural women improve their welfare, can be identified mainly in Asia. The only projectamong those evaluated in which gender issues were clearly prominent and were properly considered wasthe Preventive Health Project in Honduras. Preparation of the Logical Framework included a specificgender analysis and a gender specialist was included in the technical assistance team for the Project.Consideration of gender conditions were not apparent in other projects, such as the two irrigationprojects in Peru or the “street children” projects and the water supply (health) in Honduras or theagricultural development projects in The Philippines and the Repatriation Project in Vietnam. Whileimproved conditions in rural areas, resettlement, and job creation have an impact on women and onhouseholds and some improvement in the quality of life must have taken place, no analysis of genderimpact was done.

11. The consultants conclude that gender issues were not a required consideration in project analysis at thetime that many projects reviewed in this report were initiated. Nevertheless, some analysis could havebeen performed even while implementation was in progress, but was not done. When the analysis wasdone, as in the Honduras project cited above, it was done with proper depth and detail. Fromimpressions gathered in field interviews, together with the now explicit requirement that gender issuesare an essential part of EC project preparation and financing, the consultants assume that gender is nowtreated with greater attention than in the past, but this has still to be documented.

12. The Desk Study concluded that aid commitments have been in conformity with general EC policyobjectives, but that there was insufficient information on the consistency of the EC aid with thedevelopment policies of recipient states. The finding on conformity with EC aid objectives wasconfirmed by the Field Study, which noted, however, that the EC’s policy guidelines in CR 443/92 areso broad that conformity is not difficult to achieve. The analysis suggests that the EC has only givenprominence to issues such as gender and environment in recent years, seemingly somewhat later thanother aid agencies, including bilateral.

13. Consistency of EC Aid with Country Development Objectives. Development (DC) and Economic (Ec) Co-operation between the EC and an aid recipient country are governed by an initial Framework Agreement(FA), which establishes the broad areas of agreement for co-operation. In the case of both Honduras,Peru and The Philippines during the period under review, the country FAs were part of wider regionalagreements. From 1976 to 1992, the specifics of EC aid to each ALA country were agreed through anannual programme. Consequently, annual programming predominated during the period under review.Council Regulation 443\92, as noted above, recommended multi-year programming for EC aid to ALAcountries and in the last three years of the evaluation period (1992-1995), the EC started three yearplanning exercises, which were integrated with an annual programme review. The three-year planningexercise, which began in most countries in 1993, is undertaken as a collaborative exercise between ECstaff at Headquarters and in the Delegation and the Government. The degree of participation byGovernments and by the Delegation varies depending on the time, technical competence and resourcesavailable in each.

14. Following on 443/92, the EC introduced “Country Strategy Papers (CSP)” for ALA recipient countries,which were intended to create the framework for decisions related to the three-year planning exercise.The government, the EU Member States (Member States) and the EC must approve the CSPs. TheseCSPs are compiled from published sources and/or information provided by Member States and thegovernment, rather than from original macro-economic and sector studies undertaken by the EC itself.The CSPs identify the major areas of economic and social activity that are considered by the country andthe EC to be priority areas for EC aid intervention. Like the three-year planning exercise, discussedabove, the degree of participation by Governments depends on the time, technical competence andresources available. There are no standard methodological formats for these CSPs and, as yet, no recordof their effectiveness in making EC aid more consistent with the country’s development objectives.

15. The first CSP in Peru was done in 1993 and is currently being updated. EC co-operation with Peru wasconcentrated first on food aid and then on rural development. At present the focus is on poverty and onquestions of human rights, with reform of the justice system seen as a way to create an environment ofconfidence for investors and thus generate employment opportunities.

8

16. The first CSP was prepared in The Philippines in 1998, and, until 1998, EC projects had to be inconformity with the priorities established under the country’s Medium-term Development Plan. In ThePhilippines all the DC projects respond to the EC and government goal for poverty alleviation andimprovement of economic conditions through a micro-project approach, i.e., a number of small projectstied to villages, enterprises or a sector.

17. EC relations with Honduras are governed by the framework agreement between the EC and the CentralAmerican countries that are members of the Economic Integration Treaty. The difficulties of operatingwithout a separate country framework agreement are discussed in the Field Study report on Honduras.Honduras’ limited absorptive capacity for technical and financial assistance and the frequent staffturnover in the public sector compound these difficulties. As a consequence, the second CSP forHonduras is currently being prepared largely by the Delegation and at EC headquarters. Honduras was apriority country for food aid and rural development in the early years of EC co-operation. More recently,EC aid has focused on democratisation and modernisation of the public administration, health andeducation and food security and sustainable development and natural resources management.

18. EC development aid for Vietnam started in 1977, was suspended after Vietnam’s military intervention inCambodia and was confined to humanitarian aid from 1986 to 1990, when diplomatic relations werereinstated between the EU and Vietnam. Even when the country dialogue was non-existent, the ECsupported NGO activity for humanitarian assistance, principally refugees, through a special budget lineapproved by the European Parliament. The first longer-term planning exercise between Vietnam and theEC began in 1996 for the period 1996-2000. The EC also participates in the Consultative Groupmeetings for Vietnam, chaired by the World Bank, which requires donors to make long-term aidcommitments for a development programme produced, whether nominally or otherwise, by theGovernment. EU co-operation in Vietnam in the period 1986-1990 was characterised by itsconcentration on humanitarian and food aid and aid to NGOs with DC added more recently. Ec isemerging as an important aspect of the ECs policy agenda, together with support for democratisation.The Joint Commission has also acknowledged that the Vietnamese entrance into ASEAN enlarges thescope for co-operation and potential benefits for Vietnam from EC regional aid to ASEAN.

19. Because of annual programming and the lack of dialogue about country strategy, until 1992 EC aidoperations were open to the criticism that the programmes lacked a sense of ownership by thegovernments and that the projects where designed in an ad hoc manner. While the degree ofparticipation in decision-making about EC programmes by governments depends on their capacity andtime availability, the EC has progressively introduced measures since 1992 to make the country policydialogue participatory and the decision-making process transparent and collaborative. In the two mostdeveloped countries of the four that they visited; Peru and The Philippines; the consultants had theimpression that Governments now (i.e., in 1998) consider themselves partners of the EC in the decisionsthat are taken about the priority sectors and the stakeholders that will be recipients of EC aid.

Effectiveness

20. The question addresses the extent to which the objectives of the aid programmes and instruments have been achieved.

21. The evaluations of the individual projects suggest that, despite the many omissions that characterised theplanning and conceptual stages of project preparation and implementation, the projects reviewed did, intheir majority, attain their targeted physical outputs. While this result is somewhat surprising given theseveral constraints to optimum performance, a major caveat is that the performance targets identified inthe appraisal reports are limited to physical targets and do not address the wider issue of country strategyor the sustainability of the project. As a consequence, EC project managers will have to focus more oneffectiveness of aid in a broader sense in future.

22. In the absence of a more comprehensive evaluation of the impact of EC aid in an entire country, theconsultants conclude that, while individual projects largely attained their limited objectives, the ECprojects appear to have had a localised impact and to have benefited only a small number of people.This situation should be mitigated through the institutionalisation of CSPs, which will enable broadersocial and economic objectives to be identified and targeted, and will use project identification andfinancing as a tool to attain these broader objectives. The issue of whether the effectiveness of EC aidwould improve if the EC diversified its strategies for development aid for different types of countriesand whether one option might be greater targeting of specific objectives is raised in the final chapter ofthis report.

Efficiency

23. The question asked is whether the implementation of EC development aid policy has been handled efficiently.

9

24. The consultants examined the projects at various stages of the project cycle in order to evaluate the EC’smanagement of the project cycle and the decision-making process from identification to evaluation. Theconclusions reached for each stage of the project cycle follow, as does a diagram showing ECmanagement of the project cycle.

25. Project Identification. A potential weakness in project identification was brought to the fore by the ALACommittee (see below). The Committee recently requested that EC staff discuss alternative aid initiativesto the one finally selected, in their presentations to the Committee, to ensure that the highest priorityprojects are indeed being selected for EC financing. During fieldwork, the consultants also queried theextent to which project selection followed from a thorough examination of options. There is littleevidence, even after 1992, that the EC has reviewed potential alternatives within each sector, but, theALA Committee requirement for such a review will now ensure that this work is done prior to projectselection. This step should improve both the quality and priority allocation of EC aid. The Delegations,which have first-hand exposure to developments in country and to the work of other donors, should beheavily involved in screening project selection. With the increasing emphasis on longer termprogramming, but, above all, on CSPs, the identification of preferred sectors for EC intervention shouldbecome more refined.

26. Micro projects in the Philippines that are directed at villages and the relatively small NGO projects inHonduras financed under special budget lines suggest that they must have been selected withconsiderable understanding of the needs of the beneficiaries. Sometimes EC projects are a reaction todirectives such as that from the European Parliament after the Rio Conference, which led to therelatively quick provision of financing for the Street Children project in Honduras. Furthermore, theprogramme for returning refugees to Vietnam was a humanitarian decision by the EC, which was notinitiated by Government, although it was eventually well received.

27. The EC has occasionally financed aid activities for humanitarian and social objectives, which conform toEU principles and which did not always emanate from the host government. The EC should reserve theright to finance projects that may not necessarily reflect a government’s political priority, if such aproject furthers EU objectives related to environmental, humanitarian, gender, poverty, etc., principles.Because the EC offers grants rather than loans, governments will be less likely to refuse EC aid, evenwhen they do not fully support its purposes. In cases where the government is not totally committed tothe project, the EC will need to invest more effort into ensuring that the project can be implementedand will be sustainable.

28. Project Preparation and Appraisal. The EC tends to compact identification, pre- and feasibility studies,project formulation and ex ante evaluation into a single stage called preparation and appraisal. A draftFinancing Agreement (FA) is also prepared at this stage. The EC contracts out the preparation andappraisal of projects to external consultants and these are generally allocated some 2 to 3 weeks for workin the country. The Delegations interact with these consultants. The EC instructions for appraisaldocuments emphasise the objective of a least-cost project option. The documents do not contain ananalysis of the conditions required for successful implementation, risk analysis or implementationschedules. They also exclude cost-benefit analysis.

29. Once the appraisal phase is completed, the documents are reviewed by the EC staff at Headquarters andin Brussels and then, if the project costs exceed ECU 1 million, are presented to the ALA Committee forapproval. The ALA Committee is made up of representatives from the Member States and the membersfrequently submit written questions to the EC staff about the project, in addition to the review inCommittee.

30. The consultants’ own reviews of EC appraisal reports and of ALA Committee files and their wide-ranging discussions in Brussels and in the aid recipient countries, lead to the conclusion that this crucialpreparation and appraisal stage of the project cycle has often not been well done in the past. Theappraisals suffer from a lack of time for more thoughtful analysis and data collection and preparation, aswell as the absence of discussion of the preliminary results with a peer group. The frequent use ofconsultants has prevented a build-up of technical expertise in the EC itself for the critical review ofconsultants’ proposals.

31. The consultants conclude that with better appraisal, significant difficulties could have been avoided inthe Majes irrigation project in Peru, in the Street Children project in Honduras and in the Ec project inThe Philippines. Inadequate project preparation and appraisal throws a heavy burden on theimplementation stage of the project cycle, where unforeseen difficulties can disrupt plannedarrangements and challenge the project manager’s capacity to implement the project on schedule andwith the original costs.

32. The FAs used by the EC are too sketchy for activity planning and too rigid for a proactive management.In the first instance, they leave many aspects to be defined in the execution phase, thereby opening theway for differences in interpretation and time-consuming compromises, as well as requiring substantial

10

additional work for the formulation of the Overall Working Plan, OWP, and, later, for project closureand transfer. In the second instance, FAs are very formal instruments, whose modification isburdensome even when the rationale for modification is irrefutable and the need to proceed is urgent.Difficulties in project implementation identified in this evaluation could often be traced back to the FA.

33. The increasing introduction of the LogFrame6 approach in EC work since 1993 should enhance the

participation of relevant stakeholders in the project preparation process. The use of a LogFrame hasbecome mandatory only recently. In the Preventive Health project in Honduras and in the Programmefor Reintegration of Vietnamese Returnees (ECIP) where the LogFrame was used, the consultants notedthat the clarity of objectives, definition of performance indicators and evaluation of possible risksconsiderably improved management performance and monitoring of implementation.

34. NGOs present their co-financing proposals directly to EC headquarters and negotiate directly withBrussels. Appraisal is limited to a review of the co-financing proposal by the Delegation beforesubmission to Brussels. Although the small size of these projects limits the extent of appraisal that isfeasible, the consultants found evidence that some of the projects reviewed could have been betterprepared. However NGOs may be the executor of large projects financed by the EU entirely on its own.

35. A majority of the EC’s projects that were evaluated for this report are directed to improving humanwelfare and generally require notoriously difficult changes in human behaviour to be successful. Bycontrast, implementation of power or road projects is generally easier, but, with the exception of twoirrigation projects, infrastructure projects were not included in the consultants’ portfolio. Because theEC’s focus has been and appears likely to continue to be focused on the poorest groups in societythrough projects similar to those evaluated, project preparation in particular will have to be improved aswill monitoring, if the projects are to be successful. These improvements will require a greaterinvestment of time and resources on these aspects of the project cycle.

36. Implementation. Ultimately, project management can determine the successful implementation orotherwise of an aid initiative. Often for such management to be effective, considerable support isrequired from the country and the local Delegation. The former requires a sense of ownership by thecountry, which, as discussed, was not always present prior to 1992. The local Delegations are not staffedto provide intensive support to project management in ALA countries and only recently did the move todecentralisation give more operational authority to Delegations. In addition to the difficulties inheritedfrom the lack of comprehensiveness and depth of appraisal reports, EC projects were characterised bytwo main problems during implementation - delays and poor project management.

37. Delays. Practically all projects have a very slow beginning, owing to a slow interaction betweenHeadquarters, Delegation and country. The procedures for project approval, the organisation of thetendering process and the arrival of the technical assistance team to implement the project, has takenfrom one to three years in EC projects in ALA countries. With such delays, project circumstances canhave changed drastically once implementation begins. All the projects reviewed, except the Rural WaterSupply in Honduras and Majes II in Peru, which were evaluated by the consultants after they had beenextended, and the NGO projects, are known to have suffered delays in initiation of implementation.There are financial and operational costs attached to delays, which can have a serious impact on projectimplementation.

38. A recurring problem is that EC disbursements cannot start until an Overall Work Plan (OWP) andAnnual Work Plan have been drawn up, but these can only be done after the consultants are in place andhave assessed the work to be done. Creative solutions have been used in selected cases by localDelegations, such as the use of bridge loans for one project from the budget line of an existing project.For example, in The Philippines, the Earthquake Reconstruction Project (ERP) obtained a bridge loanfrom the Central Cordillera Agricultural Project (CECAP) to cover basic operating expenses. The EChas frowned on such practices, but they reflect an entrepreneurial attempt to resolve administrativeproblems that cause unacceptable project delays. Various solutions are currently under discussion in theEC, such as the creation of a revolving fund for Delegations from which to initiate projectimplementation, which could lead to faster disbursement of funds for implementation.

39. The consultants also noted that ever stricter R&Rs designed in Brussels with the laudatory aim ofpreventing irregularities or pursuing EU goals, tend to increase implementation difficulties. One suchexample is the rules on tenders that require equipment purchases and materials to be of European orlocal origin, when frequently such are not available.

6 The logical framework method consists of an analytical process and a way of presenting the results of this process, whichmakes it possible to set out systematically and logically the project/programme's objectives and the causal relationships betweenthem. The logframe method also indicates how to check whether these objectives have been achieved and to establish whatassumptions outside the scope of the project/programme may influence its success.

11

40. Delays that are less susceptible to flexible solutions have arisen, as in Peru, because of discrepanciesbetween the EC FA and national legislation about the agency legally able to receive EC funding.Similarly, in The Philippines, funds for the ERP did not become available until January 1993, althoughthe FA was signed in March 1992, because of the complexity of the Government’s budget laws and ofEC procedures to approve and release funds. In these cases, either the national legislation will need to bechanged - which is normally a long process that governments are reluctant to undertake to accommodatea single donor agency, or the EC FAs will need to be better adapted to local circumstances. Theconsultants recommend the latter, with all the due safeguards that can be introduced in the FA.

41. Whilst the EC has revised many of its procedures in recent years, there is no evidence that some of themost persistent causes of delay and difficulty in project implementation that arise from EC procedureshave as yet been addressed satisfactorily. A systematic audit of these issues needs to be undertaken and isprobably best done by a mix of country analyses as is being done in The Philippines, “stripe reviews”across sectors and/or countries, and an instruction to Delegations to maintain specific records of delaysin project implementation.

42. Project management is usually delegated to two project co-directors - one a national appointed by theGovernment and the other internationally recruited. The frequency of tensions between the two suggeststhat the system is seriously flawed. The CECAP project in The Philippines provides an example ofproject delays provoked by tensions between the project managers. The consultants suggest that, in mostcases, project management will be more efficient when the director is a national and the expatriate isappointed as an Adviser.

43. The administrative procedures for tendering for consultants and for equipment are generally well done.The main concern is that contracts are usually awarded to the lowest bidder, without regard to thequality of the proposal. Given the complexity of the projects and the absolute need for sustainability ofeach project, this policy needs to be reviewed to ensure that the EC is getting the best quality consultingadvice. The final steps of project completion and transfer to local beneficiaries have often been difficult,largely because the issue of the long-term sustainability of the project is not addressed at the appraisalstage and is left until very late in the implementation stage. Both the Tumbes and Majes projects in Peruwere beset with problems relating to their transfer that should have been addressed at the time of theOWP.

44. Despite the many omissions that characterised the planning and conceptual stages of project preparationand the difficulties inherent in and specific to EC implementation, the projects reviewed did, in theirmajority, attain the targeted physical outputs. While this result is somewhat surprising given the severalconstraints to optimum performance, a major caveat is that the performance targets identified in theappraisal reports are limited to physical targets and do not address the wider issue of the sustainability ofthe project. As a consequence, EC project managers will have to focus more on project sustainability,which is discussed below.

45. Monitoring is as much a management tool during implementation as it is an essential foundation forevaluation and has been the main lacuna in the project cycle. Monitoring consists of internal monitoringof a management information system for the project, which includes performance indicators that gobeyond physical targets and financial inputs and outputs and that ensures that project implementationmeets the broader objectives of the project. The weak monitoring in EC projects has hindered mid-termadjustments in project implementation and has also limited the quality of ex post evaluations. This inturn limits the lessons that could have been learnt from previous aid projects.

46. The root cause of weak monitoring in the projects reviewed is the lack of baseline data against whichproject activities can be monitored. The telescoping of the identification, preparation and appraisal stepsin the project cycle is identified as one reason for the absence of collection of baseline data, which takestime. These baseline data should provide the targets for project implementation, thus leading to thecreation of an internal monitoring system during preparation of the LogFrame. But EC staff, internalreviewers and the ALA Committee will need to ensure that high standards are set and met for thepreparation of the LogFrames.

47. Supervision of project implementation within the EC has been the responsibility of the Technical Unit inthe respective geographical DG, except for projects managed autonomously under dedicated budgetlines, such as the budget line for NGO activities. Without a proper internal project monitoring system,EC headquarters staff have relied for project supervision on the mandated quarterly progress reports onactivities and on financial performance prepared by project directors. Unfortunately, there is no standardformat for these reports, so they vary widely in coverage and quality. In general, the reports reviewed bythe consultants are descriptive and omit critical discussions about issues and options in projectimplementation.

48. Furthermore, the Headquarters staff (Technical Officers) who review the quarterly reports tend to haveheavy workloads and a relatively rapid turnover because many have fixed-term contracts. The attention

12

given to the quarterly reports is not consistent but intensifies as project risks become visible.Compounding the supervision weaknesses are the complex MIS system at EC headquarters and the lackof appropriate filing systems for project reports. The consultants question whether these administrativeproject issues have been fully addressed during the recent reorganisation of the Service Commun Relex(see Chapter V).

49. The recent transfer of selected functions from headquarters staff to Delegations, as discussed in ChapterV below, has given some Delegations more authority over monitoring and supervision. Decentralisationhas taken place in Peru and is in its first stage with the Costa Rican Regional Delegation. The advantagesof a greater transfer of responsibilities to the Delegation is that it brings about a quicker resolution oflocal problems and a greater presence of the Delegation in project implementation and, therefore,greater contact with project managers, the Government and the beneficiaries. The transfer of moreauthority for project management to Delegations should improve project implementation and,consequently, foster the visibility and credibility of the EC in the ALA countries.

50. Monitoring of NGO projects. EC projects implemented by NGOs receive relatively less supervision than doothers. The justifications are that: smaller amounts of aid are at risk, the EC tries to allow NGOsdiscretion and independence in their activities, and the costs of full-fledged supervision would be out ofline with project costs for these smaller projects. Nevertheless, the consultants report the desire ofNGOs in Honduras for greater supervision as an aid to project management and for a final externalreview in order to internalise the lessons learnt from the project. While the reasons for lightersupervision are understandable, the lesser availability of technical experts within NGOs justifies creationof a system that would provide periodic supervision with a light touch and that is actually a peer input oftechnical assistance.

51. Evaluation. The paucity and poor quality of evaluation studies for ALA projects handicaps judgementsabout project effectiveness and efficiency, sustainability and impact of most of the projects that wereevaluated by the consultants. Today, evaluation studies are mandatory for EC projects and theconsultants found that the quality of recent evaluations in their small sample had improved.

52. Interviews with project co-directors and with EC staff and delegations indicate that the mid-term projectreviews that were introduced in the nineties have proved invaluable for adjusting the course of existingprojects and shaping the design of an extension to or a follow-up of an ongoing project. In the projectsreviewed by the consultants, mid-term reviews or sector evaluations were prepared for the Tumbes,Majes and Micro-enterprises projects in Peru and the Rural Water Supply project in Honduras. No suchreviews were done for projects being implemented by NGOs.

Impact

53. This question relates to the direct and indirect consequences of the EC’s aid policy.

54. An in-depth evaluation of the impact of EC aid in the four countries visited by the consultants is notpossible because the sample of projects reviewed represents too small a share of the total EC portfolioin each country. The projects were purposely selected from as wide a range of sectors as possible, but, asnoted earlier, cannot be regarded as a representative sample of EC projects in one country or in the ALAregion. The consultants further conclude that EC interventions may have been too scattered in terms ofsector and region to have an impact at the national level. Nevertheless, based on the data available tothem, the consultants consider that four out of the fourteen EC aid initiatives they evaluated had asubstantial actual or potential localised impact.

55. The Preventive Health project in Honduras appears likely to become a model. While implementation isstill under way, the design appears very suitable for benefiting a large sector of the young population in akey area. The Project is providing training and training of trainers in the prevention of diseases to whichteenagers are typically exposed, particularly in developing countries. A very positive feature is that anetwork of NGOs supported and promoted by the Project will ensure continuity of objectives.

56. The ERP project in The Philippines had a positive impact on a poor, socially excluded and quitetroubled area of the country. Aside from direct social and economic benefits, the Project established abridge between the population and municipal and provincial governments, on the one hand, and thecentral government on the other - a long overdue first step to decentralised and good governance.Notably, the project was equipped with a systematic monitoring system.

57. In Vietnam, the ECIP had a positive impact on the country and a high political profile. The EC andseveral Member States were shown working together quickly and effectively with large numbers of the“boat people” to address a problem that had shocked the world. The project helped to establish a sounddiplomatic relationship between the EU and Vietnam. Furthermore, ECIP was a test case for future aidco-operation with the subsequent Returnees Assistance Programme and a source of experience for the

13

1997 EU directive on EC assistance to uprooted populations - a reference source now usedinternationally.

58. The Maternal and Child Health Project, also in Vietnam, provided a replicable model of the structuresrequired for this kind of endeavour. It created a health hub: a central clinic and satellite units, in districtstotally lacking such services. The result was a reduction of maternal and child mortality, as well as offertility rates, because of the introduction of the use of contraceptives and primary heath care practices.

Sustainability

59. How sustainable are EC projects after funding ceases?

60. To a considerable extent, the most critical aspect of a project’s success or failure is whether it issustainable once the external financing source has been removed. Given the weaknesses in EC aidmanagement and decision-making identified during the ALA evaluation, a review of criteria forsustainability helps to explain some of the failures. EC aid, more evidently than it has in the past, shouldfinance projects that meet targeted objectives, have impact, are sustainable and act as catalysts to attractor remove constraints for Member States aid.

61. In the EC’s Project Cycle Management Manual, sustainability is defined as: “A project is sustainablewhen it can provide an acceptable amount of benefits to the target group during a sufficiently longperiod after the donor’s financial and technical assistance ceases.” To achieve such an outcome requiresa mix of inputs, most of which are essential for all projects, and may also require others that are specificto a local context. The principal conditions for sustainability in any project are discussed below, withillustrations of violations of these principles from the ALA projects that were reviewed.

• Project Context. In the absence of country strategies, the EC has financed ad hoc projects, without fullyevaluating the context in which the project is placed. The EC has begun to institutionalise thepreparation of 3-5 year country strategies. The intention is to write such strategies based on the work ofother agencies and research institutions, including commercial ones. Such research provides a usefulpolitical and economic context and helps to determine the content, size and targets for EC aid. Of equalimportance, however, is an understanding of the project context, i.e., an analysis of the reasons for thesuccesses and failures of past projects in the same country, as well as a better understanding of thecountry’s overall strategy for a particular sector in which an aid initiative is located. Some of thecountries visited by the consultants were beginning to undertake project reviews. As noted in an earlydraft of the Philippines Country Strategy paper for 1998-2000: “To eliminate future difficultiesencountered during the implementation of certain projects, it was agreed to review the wholedevelopment co-operation logistics by a team of independent experts in the course of 1998.” A newstrategy was to be elaborated based on this country evaluation. The consultants recommend that the ECshould undertake more sector reviews, as well as analytical reviews of the main reasons for the success orfailure of a basket of aid initiatives.

• Technical Design. Projects may be sustainable if a number of criteria are ignored or misread, but if thetechnical design of a project is flawed the risk of failure is considerably higher. In the case of most of theALA projects reviewed the quality of the technical preparation is difficult to judge because the relevantfeasibility studies and appraisal reports were not done, but can be presumed for that very reason to berather poor. Consequently, the lack of attention given to the technical design of projects has been amajor factor in the weakness of project implementation. This fact became evident to the EC itself andhas led to the adoption of a series of measures during the nineties to improve project quality andmanagement, which are discussed in Section V.

• Financial Viability. As with technical design, financial viability may be the most common reason whyprojects are not sustainable. In revenue-earning projects, any underestimate of the timing of the break-even point or of longer-term returns to the investment will inevitably threaten a project’s viability, ashappened with the agriculture projects in Peru and the water supply project in Honduras. In the case ofsocial investment projects that will be managed by national, state or local governments, an unrealisticassessment of the government agency’s capacity to take over the recurrent costs is a frequent cause oflack of sustainability. If this constraint to sustainability is frequent, the EC may wish to review itstreatment of recurrent costs in project financing, particularly in lower-income aid recipient countries.

• Institutional Sustainability. Institutions whether public or private in lower-income developing countriesare likely to be weak. If the country has large numbers of well-trained people (like the Philippines) theseare unlikely to want to remain in lower-paid public sector institutions if alternatives open up in the localprivate sector or of emigration. Consequently there tends to be staff turnover. In countries that have ascarcity of skilled people, as in Honduras, in particular, institutions are inevitably weak. Institutionalstrengthening presents one of the great challenges for developing countries and for the international aid

14

community. The EC has adopted the practice of creating Project Management Units (PMUs) in ALAcountries, but these skirt the problem and postpone the institution-building aspect of the project untiltowards the end of implementation. Too often, as in the micro-enterprises project in Peru, insufficientlead-time is allowed before project completion to integrate the PMU into an appropriate public sectorinstitution. The hard choice between ensuring efficient project implementation through a PMU orattempting to strengthen from the outset the institution that will eventually manage the project, has notbeen resolved by the EC - as it has not by most aid agencies. The EC may wish to begin to record BestPractices for institutional management and ensure that these are easily accessible to staff.

• Project Management. In all the projects reviewed except for NGO projects, the EC has used thetechnique of creating two project director posts - one local and the other a foreign technical expert. Thelocal expert is usually a political rather than technical appointee. This can lead to a conflict between thebest technical and the best political solutions to issues in project management. The ALA review indicatesthat this management sharing worked relatively well in the ERP project but not in the CECAP project inThe Philippines, and tends to depend for its success on the individuals who are appointed. The systemneeds to be further examined because the quality of management is critical for successfulimplementation and for sustainability.

• Stakeholder Participation. In the early draft of the Philippines Country Strategy, dated March 1988, theEC states: “On the cross-cutting issues affecting development projects, the EC stressed the importanceof strengthening the co-operation in the framework of Partners in Development, with clear localownership and breaking with the traditional concept of ‘donor-receiver’ relations”. In the ALA projectsreviewed by the consultants, the experience with participation of governments and affected stakeholdersin project preparation was mixed, mainly with reference to projects prepared prior to 1992. Stakeholderparticipation was rarely sought at the design stage of the project, but the projects became more inclusiveduring implementation. The early participation of stakeholders in the design of projects that will affecttheir welfare is an essential condition for project sustainability. The incorporation of gender issues, i.e.,participation of and assessment of impact on women in project preparation for all types of projects ispossibly more important than the direct financing of projects that exclusively benefit women.

• Environmental Sustainability. The EC has been relatively slow to incorporate environmentalconsiderations into project preparation, at a time when most donor agencies recognise thatenvironmental sustainability is an essential condition for financial and economic sustainability. The EConly began to require environmental impact assessments (EIAs) for EC projects in the nineties. NowEIAs are an essential part of project preparation throughout the EC and are a sine qua non for projectreview by the ALA Committee.

Dissemination of Information

62. The Council asks whether the EC’s aid activities are generally well known and understood by the range of players engagedand recipient countries, in particular, as well as by European public opinion.

63. The EC Delegation in Costa Rica, which has a regional mandate that includes Honduras, has establisheda well-staffed and proactive public information programme for the area. EU events, news andpublications are widely distributed throughout an ad hoc network of correspondents, embassy contacts,sympathisers, interest groups and government bureaux. The programme can be considered a model forinformation about the EU, as is the initiative of the Philippines Delegation to set up a web site about EUoperations that are of interest locally. In general, however, the generation of information about ECactivities does not seem to have been effective. Whereas, Europeans have traditional ties to the ACPcountries and familiarity with Lome conventions and a growing recognition of their geographic ties tonon-EU countries with which they share the Mediterranean, there is no documentary evidence of theirawareness of the EC’s programmes for the ALA countries. Once the EC has a successful aid programmein the ALA countries, it will need to inform its Member States and the wider public, as a means to enlistsupport for this programme.

21

Co-ordination, Complementarity and Coherence

64. The Council asks about the extent to which the EC’s development aid activities helped to achieve the objectives of the “threeCs”. These are defined as:

Co-ordination of their development co-operation policies by the EC and the Member States, and mutual consultation ontheir aid programmes. Co-ordination also with other donors, notably the Bretton Woods Institutions, should also be takeninto account. Complementarity of EC development policy with those pursued by the Member States to foster:

The sustainable economic and social development of the developing countries and more particularly the most disadvantagedamong them;

• The smooth and gradual integration of developing countries into the world economy;

• The campaign against poverty in the developing countries.

• (agriculture, transport, trade,etc.).

Coherence between EC policies and actions in development co-operation and in other areas

65. The Maastricht Treaty principles of Co-ordination, Complementarity and Coherence of purpose in aidgiving by the Member States and the EC requires actions (by the Member States) in response toinitiatives (by the EC). In the course of their visits to Member States Embassies and EC Delegations forthe Field Study, the consultants found little evidence of interest in co-ordination and complementarity ofaid efforts. The one exception was the Pilot Co-ordination Project (PCP) in Peru. The PCP hasdeveloped a systematic way for exchanging information for co-ordination purposes among the EC andMember States country representatives. The PCP group meets periodically and has established workinggroups for different sectors. Member States representatives chair the different groups. The most tangibleresults have been achieved in the health sector, but noticeable effects have been obtained in othersectors as well. In addition to improving co-ordination, the PCP has enhanced EU visibility andrecognition in Peru.

66. A long-standing policy issue in Asia for the EC has been the attempt to co-ordinate EC aid withMember States’ aid as a means of enhancing the impact of European aid in Asia over and above Asia’scurrent share of 10% of EC aid. More recently and with similar intentions, the EC recommended thatmanagers of its Latin American programme identify the priority areas for co-ordination with MemberStates’ aid. The prime responsibility for ensuring that the Member States are aware of the EC’sprogrammes rests with the EC, whether through communication at headquarters or through itsdelegations. The ALA Committee enhances this flow of information by ensuring that the variousagencies within the Member States’ aid structure are informed about EC programmes through its reviewfunction of these programmes, but some Members state that they come in too late to the process to beeffective.

67. Judgements about the internal coherence of EC policies that use different instruments to achieve thedefined goal are outside the realm of this evaluation. All the projects evaluated by the consultantsentailed DC, with the single exception of an Ec project in the Philippines. Furthermore, there has beenless diversification in the EC instruments used in the ALA countries than in other regions, such as theACP countries. Proposals about ways in which the three Cs could be better pursued for regionalintegration are discussed in Chapter IV.

68. The relative lack of success in implementation of the three Cs policy may reflect lack of political will.Otherwise, it should not be too difficult to find ways to co-ordinate and complement EC and MemberStates activities. The responsibility for implementation of the three Cs is a collective one by the MemberStates and the EC. Given the lack of success so far in implementation, the Ministers may wish to takethe matter up again not just with the Commission but also with the Member States.

22

IV. REGIONAL CO-OPERATION AND CO-ORDINATION

1. In operational terms regional co-operation may be defined as projects that (a) affect several countries byconnecting them through transport and communications, (b) design management systems for naturalresources that straddle international borders and require the co-operation of more than one country fortheir sustainable management and utilisation, such as rivers and water basins, and (c) strengthen regionalinstitutions. The Field Study included the Tumbes irrigation project in Peru, which had originally beendesigned for bi-national co-operation and also examined individual projects in Honduras, Peru and ThePhilippines, which are all members of regional groupings, but did not specifically examine regionalissues.

2. Some 9% of EC aid commitments to Latin America directly supported regional programmes and 3.4 %of EC aid to Asia directly supported regional programmes during the 1986-95 period. EC assistance toregional programmes moved in opposite directions in the two ALA regions. In LA regional co-operationrepresented a high share of total aid commitments in the 1980s, which coincided with periods of politicalinstability in the region. Such assistance reached a peak of 25% of total aid commitments in 1988, butdeclined since. Since the debut of CR 443/92, EC aid for regional co-operation in Asia grew steadily to3.8% in 1994 and jumped suddenly to 6.9% in 1995.

3. Support for regional co-operation programmes was advocated in CR 442/81, but the Maastricht Treatyand CR 443/92 gave greater emphasis to regional co-operation. Financial and technical assistance(F&TA) could be directed “as a priority area” to regional cooperation under CR 443/92 and had thepotential to divert such assistance to more advanced ALA countries, when the focus for F&TA underCR 443/92 was almost exclusively on the poorest countries.

4. The EC has operated with specific regional framework agreements in both Asia and Latin Americaduring the period of the evaluation. More recently, as discussed below, the EC has again emphasised theimportance of regional co-operation in ALA countries. The principal EC regional agreements in LatinAmerica are:

• Co-operation Agreement with the Andean Pact and its member countries - Bolivia, Colombia,Ecuador, Peru and Venezuela - dated 17 December 1983

• Co-operation Agreement with the partner countries of the General Central American EconomicIntegration Treaty of 12 November 1985

• Inter-Institutional Co-operation Agreement with the Council of the Common Market of the South(MERCOSUR) dated 29 May 1992

• Framework agreement of Co-operation with the Republic of Costa Rica, El Salvador, Guatemala,Honduras, Nicaragua and Panama, dated 23 February 1993

• Framework Agreement of Co-operation with the Andean Pact of 23 April 1993

• Inter-regional Framework Agreement of co-operation with MERCOSUR dated 15 December 1995.

5. The second-stage regional agreements tend to be more comprehensive as well as more specific than theoriginals and follow the evolving policy themes elaborated in Council Regulations, described in Section I.For example, the 1993 agreement with Central America is more specific about objectives, areas ofcoverage and programmes than the 1985 agreement, with multi-annual programming required for DC.Specific actions to combat extreme poverty include mitigating the social impact of structural adjustment.Areas of co-operation on the environment are detailed, as well as aid for refugees and strengthening thedemocratic process. There is greater emphasis on Ec and a broad range of target sectors is listed.

6. The second agreement with the Andean Pact countries (1993) is almost identical to the 1993 CentralAmerican Agreement. There is, however, more emphasis on the transfer of technology and the targetsfor Ec are more specific, which the political and economic situation in the Andean Pact countrieswarrant. In both the Central American and Andean Pact old and new agreements, the Joint Commissionestablished for the Agreements supervises implementation of EC programmes.

7. The 1995 agreement with the MERCOSUR countries is designed to help the region strengthen politicalco-operation and to undertake a progressive and reciprocal liberalisation of trade. Ec is covered in a“partners” agreement with a sweeping scope of possible initiatives, including small and medium-sized

23

businesses and a broad range of sectors and services, as well as protection of the environment. TheMERCOSUR agreement is implemented under the supervision of an ad hoc Co-operation Council, atministerial level, and assisted by a Joint Commission of Co-operation.

8. In Asia, the EC and The Association of South East Asia Nations (ASEAN) signed a frameworkagreement for commercial, economic and development co-operation in March 1980. The membercountries are: Brunei, Indonesia, Malaysia, The Philippines, Singapore, Thailand and Vietnam7. Totalcommitments to ASEAN were 66.47 million ECU in the period under evaluation. Technical andfinancial co-operation is mostly concentrated in the early years of the evaluation exercise. Ec, at some 48million ECU, is the major component of EC assistance to the ASEAN group, as might be expected withso many countries having high rates of economic growth during the 1986-1995 period.

9. As noted in the Desk Study, the EC has tried to strengthen links between countries through intra-regional co-operation in trade, such as in the Programa Andino para la Exportacion and in thedevelopment of the commercial fisheries sectors in Colombia, Ecuador, Peru and Venezuela. Similarly,the EC has promoted intra-regional trade promotion projects in the Central American countries. TheEC regional programmes for ALA States clearly respect and reinforce the policy objectives of regionalintegration.

10. The consultants attempted to identify in those national projects they reviewed during the Field Studycomponents that clearly support regional objectives and regional projects directly or indirectly. Withspecific exceptions, such as maternal and child care in Central America, it is not at all obvious that theEC does take regional objectives into account in designing national aid programmes, unless such projectsare specifically designed as regional projects. (By implication, this suggests that Governments also tendto ignore opportunities for promoting regional objectives in national programmes.) While not allnational programmes necessarily can address regional objectives, the EC should consider emphasisingthe need to take account of regional objectives when designing national programmes. The ALACommittee can play a significant role in ensuring that regional considerations permeate the EC’s countrystrategies. For such a policy to be successful, of course, regional objectives will need to be clarified andperiodically updated and the EC staff should continue to assist countries to keep current their regionalco-operation and/or integration plans. Perhaps, most important, regional co-operation goals will need tobe continually and diplomatically promoted by each EC Delegation to Governments and to MemberStates’ Delegations in country.

11. The EC in Brussels and the EC Delegations in aid recipient countries have a major responsibility forhelping to design programmes and projects that further the objectives of countries that have formalisedtheir intentions for regional co-operation. EC project identification and formulation should necessarilybe done in close co-ordination with those countries and their regional agencies. The Member States ofthe EU have an equal responsibility for reviewing EC proposals for regional policies and, onceapproved, for co-ordinating their aid policies with the EC. The EC and Member States may wish toreview the processes for integration of effort in greater depth. It may well be that the processes that arelikely to work in one region may not be the right ones to adopt in another. The size and technicalcomposition of Delegations in the different countries of a particular region will to a considerable extentdictate the capacity of the EC and the Member States to help develop and promote regional policies insitu. There would seem to be a good case for holding a major consultative group meeting prior to ECsupport for a regional programme to plan the sectors or local regions where each Member States willfocus its aid efforts to meet national targets but also to further regional objectives.

12. No organisation is better qualified than the EC to share its technical expertise in regional integrationwith incipient or evolving regional groups throughout the world. The ALA Evaluation team was not ableto evaluate the EC’s regional programmes in depth, but is aware that the EC is using a variety of policiesand instruments, both direct and indirect, to encourage regional integration. In the process the EC iscreating a sphere of influence in a region with which it has fewer historic ties than with ACP or MEDcountries and that should help to establish economic and trading relationships between the ALA regionand the EC. Naturally, the EU’s sphere of influence would be much stronger if the resources of theMember States could be welded together with those of the EC.

7 Vietnam joined ASEAN in 1995.

24

V. RECENT DEVELOPMENTS THAT AFFECT THEQUALITY OF EC AID MANAGEMENT.

1. The following sections identify the major changes that have taken and still are taking place in the ECthat are likely to affect the quality of aid management. The consultants also attempt to assess whetherrecent changes in the EC’s structures and processes address the critical weaknesses in EC aidmanagement discussed in Chapter III. The task is large given the numerous changes and theircomplexity. The consultants’ assessment should be viewed in the context that most of thesedevelopments within the EC are very recent, in fact ongoing, and, therefore, only preliminaryjudgements about their impacts and efficacy are possible.

SEM 2000

2. An internal drive for improved quality in EC operations began in mid-1995. It was called the drive forsound and efficient management for the Year 2000, known simply as SEM 2000. (A review of SEM2000 is contained in the Field Study Annexes.) In discussions about the changes that were needed, theCommission consulted widely within the EC but also involved Member States, on a voluntary basis. Thefindings of SEM 2000 are reflected in the current reorganisation of the technical and financial staff ofDG1, DG1A, DG1B and DG VIII, which is discussed below, and are also being implemented in otherparts of the Commission. Whether or not induced by external pressures, the existence of an internal“change team” that keeps the institution’s objectives, policies and administration under continuingreview normally creates a healthy organisation.

The External Relations Common Service (SCR)

3. The current reorganisation of the technical and financial staff of the geographical Directorate Generals(DGs) into a “common structure for the implementation of Community aid to third countries”,generally referred to by its acronym (SCR) from the French: "Service Commun Relex”, is virtuallyaccomplished. The guiding principles for the creation of the common structure are8:• simplification and rationalisation in programme implementation, leading whenever possible, to

economies of scale• a balance between the administrative autonomy of the SCR necessary for its efficient functioning

and the need of the geographical DGs to ensure, via an appropriate control on the implementationphase, that their respective aid programmes are in line with their original objectives and take intoaccount their political nature

• a high degree of “responsibilisation” of (i.e., attributing greater responsibility to) all the actorsintervening in the management of Community aid programmes.

4. The new SCR results from the merger of technical and financial staff previously responsible for suchwork in DG 1, DG 1A, DG 1B, and DG VIII. The technical staff, however, is still dividedgeographically in the new SCR between the EC’s traditional regional groupings, e.g., ALA and MED inDirection B and ACP in Direction C. Certain cross-sector aid themes, such as NGOs, environment,gender issues, human rights, drugs, etc. are concentrated in one or other Direction, which then hasglobal responsibility for these themes.

5. In adopting the second phase of SEM 2000 (July 1995), the Commission recommended that “systematicevaluation is introduced for all Community programmes and actions” and acknowledged that“evaluation is a task that falls primarily to the operational DGs”. By placing evaluation capacity in theSCR, the Commission stated it had reconciled the desire for a decentralised approach for evaluation withthe need to create a critical mass of evaluation capacity. This would encourage the use of commonstandards and methodologies of evaluation.

6. Through defined processes the geographical DGs and the SCR will alternate lead roles throughout theproject cycle and are required to inform, co-operate and/or consult with the other at different stages ofthe cycle. Essentially, the geographical DGs are the leaders starting from programming and ending withthe signature of the financing agreement. The SCR prepares terms of reference for evaluation incollaboration with the geographical DGs. The evaluation itself is managed by the SCR.

7. The Evaluation Unit. The considerable emphasis on evaluation during this restructuring addresses a majorweakness in the EC’s management of the project cycle. A separate Evaluation Unit has been created thathas global rather than regional responsibility for evaluation. Currently, the Unit has nine evaluators,

8 Source: Service Commun de Gestion de L’Aide Communitaire aux Pays Tiers: Rapport Conjoint – Groupes de Travail No.1 etNo.2. O/97/424. Secretariat General SEC (97) 2305.

25

which includes three external experts on short-term contracts, in addition to the head of the EvaluationUnit. This compares with ten evaluators and three heads of units in the previous DG IA, IB and VIIIevaluation structure. While the concept of integrating evaluation resources is good, it is difficult to seehow the Unit can effectively evaluate the EC’s development aid programmes world-wide or help toensure that common standards and methodologies are applied at all stages of the project cycle with thislevel of staffing. Therefore, recommendations in this report for improved monitoring and evaluation, inparticular, will be difficult for the Evaluation Unit to implement. The Unit will have to ensure thatappropriate monitoring systems are included in project preparation to facilitate later work on evaluation.The Unit will also need to expand its internal resources by using experienced consultants frequently. In awelcome development, the SCR plans to post on a web site to be constructed by April 1999 thesummaries and full texts of all completed evaluations.

8. The independence of the Evaluation Unit is protected because it will present its reports jointly to thegeographical DG and to the head of Direction F and also has the authority to schedule evaluations of itsown accord. Ideally, the importance of evaluation would be highlighted and the independence of theUnit guaranteed, if it were taken out of its current Direction and reported directly to the DirectorGeneral of the SCR. The latter would then transmit the evaluation findings to the geographical DGs,which would give those findings greater visibility.

9. A review of the evaluation work for EC programmes could reveal a number of technical problems thatrepeatedly characterise these programmes. If it were to undertake “stripe” reviews of the same topicsacross sectors and countries, the Evaluation Unit could contribute to a better understanding of thecauses of such problems. These could evaluate, for example, financing of recurrent costs, counterpartfunding, monitoring, project management, etc., which would address serious gaps in the EC’s evaluationprocesses. But, any suggestions for an increase in the responsibilities of the Evaluation Unit eitherrequire an increase in staff or a redesign of the Unit’s traditional way of doing evaluations.

10. Issues. A potential weakness of the new SCR structure is that financial and technical staffs are still inrelatively small pools and still have narrow geographic responsibilities. The Evaluation Unit discussedabove is an exception - it is small but has world-wide responsibilities. There appear to be no formalvertical links between the horizontal geographic technical departments and, therefore, no obvious meansof cross-fertilisation of ideas. To ensure that experiences are shared and identified, brief informalelectronic summaries of best practices, in addition to the normal project databases, can be developed.

11. A practical question is whether topic specialists in specific geographical units, e.g. NGO specialists in theACP country Directorate (C), will respond with equal priority to demands for their services from othercountry Directorates, i.e., MED and ALA (which are in Directorate B)? There could be a danger ofneglect of such issues in other regions when cross-sector activities are placed in specific geographic SCRdepartments.

12. The integration of the results of evaluations into programme design and country macro and sectoranalyses is the responsibility of the geographical DGs. The restructuring has rightly emphasised theimportance of integrating evaluation findings into new programme design. To ensure that evaluationresults achieve their objective of improving the quality of future projects, there will need to be widedissemination of the lessons learnt. An additional advantage of this dissemination is that it will enhancethe EC’s reputation for transparency. In a welcome development, the SCR plans to post on a web site tobe constructed by April 1999 the summaries and full texts of all completed evaluations.

13. The reorganisation of the SCR is too recent to be able to draw conclusions about its operational viabilityand whether it will successfully address the Commission’s objectives described above. A major questionis whether the sharing of responsibilities during the project cycle between the geographic DGs and theSCR will work, or whether it will lead to overlaps and/or underlaps. As with relations with theDelegations, discussed below, the sharing of responsibilities during the project cycle will work smoothlyonly if the managers of the geographic DGs and the SCR provide leadership and create incentives thatencourage co-operation.

14. The continued separation of project specialists into geographic units when some of the units have a verysmall staff raises the question of whether economies of scale have indeed been achieved. Would it bemore effective to group, for example, all the infrastructure specialists in one department and give themresponsibilities for all the EC’s programmes world-wide? That in turn would require that a mechanismbe established to ensure that each geographic DG had access to infrastructure specialists on an equitablebasis. In such a case, the benefits of geographic specialisation would be lost but replaced by more variedgeographic experience and greater cross-fertilisation of ideas.

15. The impact of the reorganisation of the SCR on the ongoing EC decentralisation process cannot as yetbe measured because the former is too recent and there is virtually no experience of interaction between

26

the new SCR and the EC Delegations9. If the decentralisation process effectively strengthens thetechnical capacity of the Delegations, the current small technical units in the SCR could devote moretime to quality enhancement and to ensuring that common standards and methodologies are applied atall stages of the project cycle. The potential for useful interactions do not appear to be prescribed andwill depend on the initiative of individual staff and the leadership provided by individual managers.

9 The field work for the ALA study ended before the SCR was established.

27

EC Delegations

16. The consultants had contact with three different kinds of Delegations in the course of the fieldwork:• long-established Delegations with both diplomatic and operational functions in Peru and the

Philippines• a Delegation initiating its operations in Vietnam, where responsibility for country operations was

handled by the Delegation in Thailand until 1996, and,• a Delegation with regional responsibilities for Central America in Costa Rica, which the consultants

visited to discuss the projects they were evaluating in Honduras.

17. As in all organisations with a large Headquarters staff and country Delegations, creative tensions tend toarise between the two in the course of pursuing the organisation’s mandate. The functions of theDelegations and relations between the Delegations and Headquarters were described in the Field Studyreport. The conclusions reached by the consultants were that: “Headquarters relies heavily ondelegations for technical and administrative tasks, without delegating enough power to resolve situationslocally. On the other hand, the Delegation’s country knowledge and insight may be under-utilised infunctions such as country strategy formulation, project appraisal, tailoring the Framework Agreement tocountry conditions”. But, the consultants did single out Peru as an example of a Delegation that issuccessfully involved in project management and has enhanced the image of the EC/EU in the process.

18. In addition to the changes discussed in the previous section, certain changes have occurred since the endof the ALA project evaluations. The EC 1998 Guidelines on “Les Relations entre le Siege et lesDelegations” are now the definitive source of information and guidance about the roles andresponsibilities of Delegations. Moreover an important segment of the Guidelines deals with thesensitive area of “Managing Co-operation Programmes: Respective Responsibilities of Delegations andHeadquarters”. The sections dealing with implementation and monitoring of the project cycle are likelyto be invaluable to Delegations. The issuance of the Guidelines is combined with Commission supportfor decentralisation of its country management operations to Delegation staff and the increasingmovement of Headquarters staff to Delegations.

19. The Field Study found that, on the whole, the pilot decentralisation project in Peru had downloadedconsiderable purely administrative activities to the Delegation. There was also a sense that theDelegations and the Costa Rican Regional Delegation, in particular, was “drowning in papers”. But, theconsultants field visits coincided only with pilot projects for decentralisation rather than theimplementation of the 1998 guidelines. One positive effect of the pilot decentralisation projects, whichare likely to become a characteristic of the Delegations in future, is the closer relations that were beingforged with the local project directors, as a result of the Delegations’ growing involvement in projectwork.

20. An area that does not appear to be affected by the Guidelines and that may require some attention bythe Commission is the Delegation’s role in the preparation of a country strategy. The ideal combinationof perspectives for the country strategies are those of the Government and its National DevelopmentPlan, the Delegations’ insights into national policies and the local context, combined with theirknowledge of EC policies and R&Rs, and the headquarters’ country officer, who is most familiar withEC-wide policies for aid giving. Many country officers in Brussels will invite the Delegation’sparticipation in country strategy formulation, but the Commission may need to consider requiring someevidence of close consultation, such as joint signatures on the final strategy document that is submittedto the ALA Committee.

21. Decentralisation is an accepted good management principal. There are likely to be numerous initialproblems in ensuring that delegation of real responsibilities occurs and that it works properly. As withthe sharing of responsibilities between the SCR and the geographic DGs, there are dangers of overlapsand underlaps. Again, this is a situation that will require careful review by the respective EC managers.

The ALA Committee

22. The ALA Committee was set up in 1981, essentially as a project review committee, but it has metregularly with more formal structures and a clearer mandate only in the past few years. The Committee’scurrent legal base stems from Council Regulation including (CR) 443/92. The Committee is guided bypolicies set by Council working groups, such as the Development Working Group, which establishesareas of focus of EC aid for environment, gender, and related issues. The Committee reviews allprogrammes requiring a commitment exceeding ECU 2 million and approves such programmes througha system of qualified majority votes.

23. The fifteen members of the Committee – each delegation is usually represented by 2 or 3 representatives- are managers and officers from the Member States aid ministries and representatives of the permanent

28

country delegation to the EC. The meetings have been chaired by the Deputy Director-General ofDG1B and are attended by relevant EC staff. Meetings are currently held once a month and last one dayor, exceptionally, two days. This entails a serious time commitment from the country participants.Nevertheless, the meetings are well attended and continuity of participation does not seem to be a majorissue.

24. The ALA Committee has several functions. One of these is the critical review of project proposals, withsuggestions for changes to improve quality. Committee members circulate the proposals received priorto the meeting to the various agencies within their country’s aid structures and, consequently, frequentlysubmit written comments on the proposals ahead of time to EC staff. In only some 5-10% of cases arethe issues not resolved ahead of time, in which case modifications are made through a voting procedureat the meeting, or, rarely, after re-submission to another ALA Committee meeting.

25. Another important function of the Committee is to inform the Member States aid agencies about theprojects that the Commission is financing and, therefore, to provide opportunities for the joint actionrequired in the “3 Cs”. With the insights that the Committee develops into the workings of the EC, theCommittee can be an important element for change in the EC’s procedures and even policies. This rolewill be further enhanced if it also leads to the dissemination of information about Member Statesprogrammes to the EC.

26. All written material presented to or by the Committee is archived by the Committee’s small staff. Mayorverbal comments are reflected in the minutes. The archived material has a limited utilisation as ananalytical resource for change. As a result of work done for this evaluation, the consultants recommendthat the documents related to the Committee’s meetings be designed to record major decisions or issuesthat arise in the discussion. The minutes should be analysed periodically and systematically to identifyrecurring themes about EC policies and procedures.

27. As a result of its deliberations over time, the ALA Committee had requested that alternative aidinitiatives be discussed in each proposal presented for financing, to ensure that the project selected istruly the highest available priority. Such a review of options should be standard practice for EC staff,although it is still very rare. The ALA Committee also noted the lack of regularity in presentation ofMid-term Reviews of project implementation. This coincides with the general concern identified in boththe Desk and Field studies for the ALA Evaluation about the serious absence of monitoring of projectactivities and of formal evaluation of EC aid initiatives in ALA countries.

28. A major change in the Committee’s work is currently being institutionalised. The Committee identifiedthe lack of country and sector strategy papers as a major gap in its ability to assess the context in whichprojects are prepared and financed. Country strategy papers are now required covering a 3-5 year period,which the Committee is starting to review. In tandem, the Commission will prepare longer-termindicative budgets and programmes for each country or regional programme, which the Committee willalso review. The change in emphasis to a more upstream analysis of EC operations should start theCommittee’s technical assistance contributions earlier and at a stage where it can influence strategicpolicy decisions and project selection. The major benefits of this change are that the Committee and theCommission should now be able to:• develop a strategic framework that will help to keep EC assistance within its target priorities for

each ALA country, as opposed to funding ad hoc projects• concentrate more on a discussion of the relative roles of financial and technical assistance and of Ec

in ALA countries, and• create even more transparency about the EC’s aid operations through an earlier understanding of its

priorities, which could enhance the potential for greater co-ordination, coherence andcomplementarity in aid giving between the EC and the Member States.

29. In an attempt to understand better the work of the ALA Committee, the consultants designed aquestionnaire about the ALA Committee and its work, which it sent to thirty-three Committee membersin fifteen countries. To date, the consultants have received a total of twelve replies from ten countries.The replies are consolidated in the table in Annex VIII. (The consultants issued the questionnaire withthe understanding that individual replies would not be published and the authors would not beidentified.)

30. An evident positive factor is that attendance is regular and all countries submit written queries about theproposals to the Commission. The Committee’s willingness to accept the additional responsibilities andthe workload of country strategy approval is unanimous. The critical part of the questionnaire concernsthe Commission’s responsiveness to the Committee’s decisions. With one exception, all the Membersexpressed satisfaction with the Commission’s participation in the Committee, but a minority stated thatthe Commission’s follow up to the comments is not always satisfactory. Other comments were that it isdifficult to change project proposals at the Committee stage because the meetings occur too close to

29

implementation. Another Member stated that the Committee should concentrate on country strategiesand methodological issues, rather than on project review.

31. The issues contained in the responses provided by the ALA Committee members to the consultant’squestionnaire need to be discussed between the ALA Committee and the respective Commissionmanagers. Given the obvious dedication of the Members and the important changes in procedures andin aid management that the Members have provoked, any opportunities to enhance their contributionsshould be welcomed by Commission managers. Furthermore, the ALA Committee remains a potentiallyeffective force in promoting the three Cs, which is a matter of concern to the Council of Ministers.

The Quality Support Group (QSG)

32. The Quality Support Group was set up in 1996 to improve the management of the project cycle for DCin DGVIII. The initial objective of the QSG was to ensure that project financing proposals andappraisals reflect the EC’s policy and operational guidelines. The QSG was comprised of 10 members ofDGVIII staff to review all projects exceeding either ECU 1 million or ECU 2 million, depending on thesource of funding. The Group meets once a month, has to reach consensus and must record itsconclusions. Gradually, the QSG has operated as a permanent quality review board for aid proposalsbefore they are presented to the finance decision committee, i.e., the European Development Fund(EDF) for DGVIII.

33. To address the concerns about project quality raised in both the Desk and Field studies for the ALAEvaluation, the consultants recommend that an equivalent QSG be created for the ALA countries. Thiswould help to improve the quality of the proposals presented to the ALA Committee at a time when thatCommittee is shouldering new responsibilities for country strategy papers. Peer review is an effectiveway of raising standards within an organisation and the model used in DG VIII could usefully betransferred to the ALA countries within the EC.

34. With the ALA Committee more engaged in the upstream review of aid strategies and options and with a“QSG” for the ALA countries within the EC reviewing project operations more consistently, some ofthe major concerns identified in this ALA evaluation would be addressed. Furthermore, the carefulrecording of decisions by these two groups and their periodic review and analysis, should lead to theproduction every 3 to 5 years of a Lessons Learnt report, which could be widely disseminated to ECstaff and to Member States. Such a process would help to improve management of EC aid programmes,could be a useful source of technical information for Member States, and would lead to much greatertransparency about the EC’s aid programmes for the European public and for aid recipients. Suitablyedited and published such reports could add considerable value to the literature available on the projectcycle and, in the process, provide useful publicity for the EC.

30

Internal Manuals

35. The EC has developed a series of manuals and guides to aid staff with project management and withoperations from within a complex organisation like the EC. Notable among these are the:• July 1992 Guide: Management of Financial and Technical Assistance and Economic Co-operation

with the ALA and MED developing countries• Feb. 1993, Integrated Project Cycle Manual, which standardised the use of the LogFrame in EC

project management and spurred adoption of the LogFrame by bilateral aid agencies• March 1995 (seventh edition) Manual of Operational Procedures• Nov. 1997, Self-Monitoring Manual, which complements the Project Cycle Manual and essentially is

designed to assist managers to monitor projects that were designed under a LogFrame approach.This manual is used only by DG1B/B and has not been officially adopted by the EC.

• 1997 Guide–Ex-post and Intermediate Evaluation Manual, which defines procedures for thedifferent evaluation stages.

• 1998 Guideline by Mr. Cioffi on: “Les Relations entre le Siege et les Delegations”.In addition the EC has a body of Rules and Regulations that govern the administrative aspects of itsoperations. All of these are critically evaluated by the consultants in Annex 2 to the Annexes volume ofthe Phase II- Field Study for the ALA countries.

36. While some of these operating manuals and rules of procedure require updating or leave room forimprovement, the total body of guidance that is now available to EC staff is impressive. Furthermore,the Integrated Project Cycle and the Evaluation Manuals, plus the Evaluation Format, have set standardsfor several other aid agencies.

37. The three EC manuals of 1993 and 1997 specifically address previously identified knowledge gaps andare designed specifically for EC projects. By themselves, these manuals cannot resolve weakness inproject management and will require that adequate numbers of staff be trained in their use. Such traininghas already taken place since 1993 for DGVIII and since 1996 for DG1B. A question is whether theongoing reorganisation has clustered or dispersed to other responsibilities those who were trained.

31

VI. ISSUES AND RECOMMENDATIONS

1. Important issues related to the ALA countries are their heterogeneity, the breadth of the policyframework for ALA aid and the lack of diversified strategies for aid delivery among countries at verydifferent income levels and with different patterns of income distribution. An analysis of the countryprogrammes and projects evaluated by the consultants has identified a number of issues that affect thequality of the aid delivered and its impact and the efficiency of aid management. The analysis of recentstructural changes within the EC highlights the need for a more efficient use of scarce resources. Theseissues are discussed below and specific recommendations are formulated for the Council of Ministersand the Commission Managers and, in one instance, for the Member States. The absence of protocolsfor the ALA region does not lead to cohesion among the countries of the region in their relationshipwith the EC, so that no recommendations have been formulated for the beneficiary countries.

Recommendations to the Council of Ministers

A Diversified Strategy for ALA Countries

2. It is now over six years since the EC developed specific policies for the ALA region, which areembodied in CR 443/92. The EU redefined its relations with the MED countries in 1995 and hasrecently held in-depth negotiations with the ACP countries for a new Lome agreement. Given alsorecent changes in the economic dynamics in Asia, the timing is appropriate for a thorough review of theEC’s policies towards and relations with the ALA countries. Specifically, the EC should re-evaluate theobjectives, delivery systems and expected outcomes from EC assistance to the ALA countries.

3. These are after all a heterogeneous mix of large and small, poor and not so poor, densely populated andnot, with some moving more resolutely to regional trading agreements and economic integration. Withinthe ALA countries, India and China contain some 62% of the world’s poorest peoples, while Argentinais ranked 29th among the countries with the highest GNPs per capita. The EC may wish to recommendstrategies that differentiate between ALA countries at different income levels or with large segments ofpoor people, or that are in the process of regional integration. As a result, the ALA countries might beclassified into two or three different categories for purposes of EC assistance.

4. The EC provides DC assistance exclusively through grants no matter what the income level of thecountry or the type of project for which assistance is being provided. CR 443/92 instructs the EC tofocus its DC on the poorest countries and regions and its Ec on those with the highest growth rates andper capita incomes. Yet richer countries can still attract DC for the Council’s priority assistance goals ofregional co-operation, gender, democratisation and good government, and the environment. At a time ofgeneral scarcity of development assistance -and it is probably true that in relation to demanddevelopment assistance is always scarce - the EC may wish to review the rationale for its blanket grantassistance even for priority objectives to all ALA countries.

5. The consultants reviewed only one Ec project in the course of the Field Study and are unable, therefore,to analyse the impact of Ec aid in the ALA countries from first-hand experience. Logically though, asone tool for diversifying aid strategies, the EC might require that DC should be combined moreeffectively with Ec to meet economic growth objectives. In middle to higher income countries DC couldthen be partially allocated to enhance the regulatory, policy, institutional and infrastructure framework inorder to attract Ec and other private investment. Once a sustained pattern of growth is established theEC could eventually reduce or eliminate allocations of DC to higher-income ALA countries. Such astrategy though is controversial. One MS made the consultants aware, but this is unlikely to be a loneopinion, that it regards Ec more as a means to propagate individual MS trade and investment objectives,rather than a development tool. Given the relative scarcity of DC funding, more thought may need to begiven to ways in which Ec can effectively support a diversification of aid strategies within ALA countriesand in the best interests of those countries. One possibility could be to decide on a case by case basisuntil a body of experience has been built up and monitored and evaluated.

6. The consultants noted apparent differences in treatment by the EC of Asia and Latin America asindividual regions, with an apparent bias of assistance, calculated on a per capita basis, towards LatinAmerica. The consultants visited the limited sample of only two countries in each continent so wereunable to explain the situation. Analysis is also constrained by the lack of any work within the EC aboutthe difference in treatment between the regions and the lack of institutional memory in the EC. Theconsultants recommend that such an analysis should be done by the EC to assess the reasons for andpossibly correct the bias in assistance to Latin America.

Clarification of Rationale and Conditions for Assistance to Priority Objectives

32

7. An overriding objective of EC aid, incorporated in every Council Regulation, is the reduction of poverty.Over the years the Council has added other objectives: gender, NGOs, environment, food andhumanitarian aid, regional cooperation, democratisation and governance, trade, etc.). The number ofobjectives and the lack of clear guidance about the priorities among them now require that there be areexamination of the targets of EC assistance in ALA countries. More clearly defined objectives for ECaid in stratified groups of countries within ALA, should lead to greater aid effectiveness, and, above all,should ensure that the impact of the aid is more visible at the national level. Possibly, as is now the casewith environmental aid, quantitative targets may need to be established over a period of years for thepriority objectives.

8. As a corollary, once the Council has established new policies for EC aid to ALA countries, EC managersshould ensure that clear guidelines are issued promptly to staff to ensure that the Council regulations canbe implemented efficiently.

Recommendation to the Council, the Member States and the Commission

1. Co-ordination, Complementarity and Coherence of Aid

9. In the course of their visits to MS Embassies and EC Delegations for the Field Study, the consultantsfound little evidence of interest in co-ordination and complementarity of aid efforts. The one exceptionwas the Pilot Co-ordination Project (PCP) in Peru. The relative lack of success in implementation of“the three Cs” may reflect lack of political will. Otherwise, it should not be too difficult to find ways toco-ordinate and complement EC and Member States activities. The responsibility for implementation of“the three Cs” is a collective one by the Member States and the EC. Given the lack of success so far inits implementation, the Ministers may wish to take the matter up again among themselves and within thecountries they represent. As a first step, the Ministers could ensure that the concepts are clearly definedand that implementation guidelines are drawn up and widely circulated to national aid agencies and theirrepresentatives in beneficiary countries, as well as to EC Delegations and staff.

2. Regional Co-operation

10. The EC in Brussels and the EC Delegations in aid recipient countries have a major responsibility forhelping to design programmes and projects that further the objectives of countries that have formalisedtheir intentions for regional co-operation. The Member States of the EU are responsible for reviewingEC proposals for regional policies and, once approved, for co-ordinating their aid policies with the EC.The EC and Member States may wish to review in greater depth the processes for co-ordinating and/orcomplementing efforts. It may well be that the processes that are likely to work in one region may not bethe right ones to adopt in another. The size and technical composition of Delegations in the differentcountries of a particular region will to a considerable extent dictate the capacity of the EC and theMember States to help develop and promote regional policies in situ. There would seem to be a goodcase for holding a major consultative group meeting prior to EC support for a regional programme toplan the sectors or local regions where each Member States will focus its aid efforts to meet nationaltargets but also to further regional objectives.

Recommendation to the Commission Managers

11. In this section, the consultants identify the major issues that affect EC aid management and that are theresponsibility of the Commission. The recommendations are therefore addressed to the CommissionManagers.

The Information Base

12. The serious absence of information about ALA aid initiatives not only constrained the quality of thisALA evaluation but also has prevented and probably still prevents internal analysis of the effectivenessof EC policies and programmes for ALA countries. A filing system for legal and technical documentsand the creation of an evaluation library was being set up in mid-1997, but the consultants believe thatsome possibly substantial information gaps for past programmes will never be filled. While it may not bepossible to recreate the past, it is essential that the EC improve systematically the generation, collectionand collation, and regular dissemination of information. This will require specific targeted efforts atheadquarters and in the delegations. Appropriate information technology systems that capture datagenerated by EC activities immediately and disseminate it for information, analytical, monitoring andevaluation purposes, as well as the creation of an on line handbook of best practices, should be createdwithin the EC as a major priority.

13. The specially commissioned work by ODI done prior to the initiation of the three consultants’evaluations of the EC’s programmes, which presented statistical information about the EC’sprogrammes from 1986 to 1995, created a very useful framework for the evaluations. It is surprising,

33

however, that the data could not have been generated automatically from the EC’s internal informationsystems. The ODI study is already out of date, but the consultants are not aware that the EC has put inplace systems to continue the flow of such essential information and to ensure the updating of the ODIdatabase.

14. The consultants found that the files with the decisions and recorded comments of the ALA Committee,which meets regularly, provided valuable insights into operational issues. The information had merelybeen filed until this evaluation. The consultants recommend that, in future, such data be recorded in adatabase, categorised and periodically analysed by EC staff, as a continuous input into improved designof country strategy papers and projects.

15. Institutional memory within the EC is weak. Within EC headquarters the staff is probably more mobilethan in most agencies. Personnel are frequently assigned to country delegations. The assignment oftechnical experts by Member States to the EC for periods of about three years, the overall dependencyon consultants and temporary staff, plus rather heavier turnover than in many aid agencies (probably as adirect result of European integration which facilitates mobility), as well as periodic reorganisations, allweaken institutional memory within the EC10. Consequently, it behoves EC managers to compensate forthe lack of institutional memory by creating information systems that regularly record ongoingprogramme and project information. These systems should be made easily accessible to staff andbecome a reference source for lessons learnt and best practices in aid management.

16. A major conclusion of the ALA evaluation exercise is that the EC should ensure that the informationbase about its operations in ALA countries is complete, current and easily accessible. By comparisonwith several other aid-giving organisations, whether multilateral, regional or bilateral, the informationbase at the EC - at least in 1997 when this evaluation began - is significantly worse. Given the size of itsaid flows and its global importance, the EC needs to improve its databases urgently. Much of theinformation that the EC requires to improve its operations should be produced routinely and as a by-product when policies, programmes and projects are developed, approved, implemented and, in the caseof programmes and projects, completed and evaluated. Without such an information base, the ECcannot begin to monitor the effectiveness of its decision-making processes and aid management, nordoes it have a sound basis for evaluating the appropriateness of its policies for the ALA countries.

The Project Cycle

17. The extensive analysis of projects at each step of the project cycle undertaken in this evaluation indicatesthe weaknesses frequently apparent in EC aid management for ALA countries. In the following sections,the issue is identified followed by a specific recommendation (in italics).

i) The absence of country strategy papers (CSPs) and of long-term programming until the earlynineties led to ad hoc financing of projects. To date no analysis of the quality and relevance of theseCSPs has been done.

The consultants suggest that the comments of the ALA Committee about such strategy papers should be collated forperiodic review by EC evaluators or consultants. In addition, the Delegations and EC staff should provide, in aformat that can supplement the ALA Committee’s initial review, regular monitoring of the CSPs during the course ofproject implementation.

ii) Project selection has been criticised as not targeting the highest priority projects.

As a result of a recent request by the ALA Committee, the EC will evaluate options before project selection. Duringproject selection, there should be greater emphasis on the project context, which is partly resolved by the now requiredcountry strategy papers, but also needs a better understanding of the sector context and the reasons for existing projects’failure or success,

iii) The EC tends to compact project preparation, pre- and feasibility studies, project formulation andex ante evaluation into a single and overly brief stage called preparation and appraisal. This crucialpreparation and appraisal stage of the project cycle has often not been well done in the past. Theappraisals suffer from a lack of time for more thoughtful analysis and data collection andpreparation, as well as the absence of discussion of the preliminary results with a peer group. Theweak monitoring in EC projects has hindered mid-term adjustments in project implementation andhas also limited the quality of ex post evaluations. This in turn limits the lessons that could havebeen learnt from previous aid projects. A main cause of weak monitoring is the lack of baseline dataagainst which project activities are measured and the collection of which should be started duringproject preparation.

10 From another perspective, there are advantages to some mobility within an institution; when staff do not rotate and stay forlong periods in one assignment they tend to have good institutional memories but also tend to be change averse.

34

Project preparation and management budgets and project cost coefficients must be adapted to accommodate more carefulproject preparation, including stakeholder participation, appraisal and monitoring. At the same time, managers of theearliest projects that are based on a LogFrame approach to project management should be asked to record constraintson monitoring caused by the EC’s administrative or budgetary procedures. The Commission could compile a checklistof changes that are needed to improve project preparation and appraisal and that need to happen before monitoring canbecome an unquestioned and integral part of the project cycle. These records should be forwarded to an appropriate unitwithin the EC for analysis and follow-up recommendations and guidelines.

iv) Headquarters and delegation staff have perceived a tendency to select the lowest biddersystematically in EC project contracts. The immediate savings are obtained at the risk of loweringperformance standards and sometimes even the modification of the project to accommodate alower consultants’ budget.

Given the complexity of the EC projects and the need for their sustainability after implementation, the policy forawarding contracts needs to be reviewed.

v) The FAs used by the EC are too sketchy for activity planning and too rigid for a proactivemanagement. In the first instance, they leave many aspects to be defined in the execution phase,thereby opening the way for differences in interpretation and time-consuming compromises, as wellas requiring substantial additional work for the formulation of the Overall Working Plan, OWP,and, later, for project closure and transfer. EC Financing Agreements (FA) are formal instruments,whose modification is burdensome even when the rationale for modification is irrefutable and theneed to proceed is urgent. Difficulties and delays in project implementation could often be tracedback to the FA and to complex and unwieldy administrative Rules and Regulations (R&Rs). Arecurring problem is that EC disbursements cannot start until an Overall Work Plan (OWP) andAnnual Work Plan have been drawn up, but these can only be done after the consultants are inplace and have assessed the work to be done.

To avoid exogenous constraints on project implementation, the FAs and R&Rs that govern project administrationmust be adapted to facilitate and not hamstring procedures for project implementation. Various solutions are currentlyunder discussion in the EC to speed up disbursements, such as the creation of a revolving fund for Delegations fromwhich to initiate project implementation, which could lead to faster disbursement of funds for implementation. It isurgent that a solution be found quickly to this recurring problem.

vi) EC projects frequently suffered from poor project management attributable to the EC tradition thattwo directors are appointed as joint project managers—one a national and the other a Europeanexpert.

A solution that would please the countries where the consultants identified project management as a problem, would bethe appointment of the national expert as the Director and the European expert as an Advisor.

vii) For project management to be effective considerable support is required from the country, whichrequires a sense of ownership that was not always present prior to 1992.

The EC will have to continue to improve its dialogue with Governments about priority projects and programmesthrough joint preparation of the CSPs, which should conform to the country’s development priorities.

viii) The paucity and poor quality of evaluation studies for ALA projects handicaps judgements aboutproject effectiveness and efficiency, sustainability and impact of most of the projects that wereevaluated by the consultants.

The Commission made evaluations mandatory for all projects in 1995. The recent reorganisation of EC staff hascreated a single team with exclusive responsibility for evaluation of all the EC’s external programmes. Ideally, theimportance of evaluation would be highlighted and the independence of the Unit guaranteed, if it were taken out of itscurrent Direction and reported directly to the Director General of the SCR. The latter would then transmit theevaluation findings to the geographical DGs, which would give those findings greater visibility.

To ensure that evaluation results achieve their objective of improving the quality of future projects, there will need to bewide dissemination of the lessons learnt. An additional advantage of this dissemination is that it will enhance theEC’s reputation for transparency11.

The Commission’s objective of decentralising evaluation has been achieved but the critical mass of expertise createdmay be too small to be effective and to implement recommendations for extending the role of evaluation within the EC.For example, the consultants believe that the EC should be undertaking more cross-sector analysis that targets issues

11 In a welcome late development, the SCR plans to post on a web site to be constructed by April 1999 the summaries and fulltexts of all completed evaluations.

35

that are common to projects in a variety of sectors, e.g., financing of recurrent cost. Another area of interest is theobjectives that cut across individual sectors, like gender and poverty. The objective would be to build up a body ofexperience with these project elements and to track compliance with EC objectives. But, any suggestions for an increasein responsibilities of the Evaluation Unit either require an increase in staff or a redesign of the Unit’s traditional wayof doing business.

36

Aid Management within the EC

18. Given the pervasive weaknesses in the project cycle, the consultants recommend that a Quality SupportGroup similar to that already established for DGVIII and consisting of peer reviewers from within theEC should be created to review project proposals for the ALA countries. This would help to improvethe quality of the proposals presented to the ALA Committee at a time when that Committee isshouldering new responsibilities for country strategy papers and some members believe their inputs aremade too late in the process to effect change. With Delegations more involved in support toimplementation, these combined efforts should result in more relevant projects that are more efficientlymanaged and that result in an improvement of the cost-effectiveness of EC aid. The consultant’srecommendations for improved communication within the EC and for collation and dissemination ofevaluation data should help to improve the knowledge base within the EC about best practices.

19. Changes in aid management can be recommended and even adopted but will require leadership by ECmanagers to ensure that the reforms are institutionalised. The creation of the Common Service and thedecentralisation of responsibilities to the delegations should have a positive impact on projectmanagement. There are likely to be numerous initial problems in ensuring that delegation of realresponsibilities occurs and that it works. As with the sharing of responsibilities between the SCR and thegeographic DGs, there are dangers of overlaps and underlaps. This is a situation that will requireleadership by the respective EC managers and the creation by them of incentives to staff that encourageco-operation and the sharing of responsibilities for project management.

Recommendation to the ALA Committee and Commission Managers

20. The concerns expressed by a number of ALA Committee members in their responses to the consultant’squestionnaire about their responsibilities need to be discussed by the ALA Committee with therespective Commission managers. Given the obvious dedication of the Members and the importantchanges in procedures and in aid management that the Members have already provoked, anyopportunities to enhance the contributions of the ALA Committee will assuredly be welcomed byCommission managers. Furthermore, the ALA Committee remains a potentially effective force inpromoting the three Cs, which is a matter of concern to the Council of Ministers. Therefore, the ALACommittee and Commission managers should establish a dialogue to find ways to enhance thecontributions of the Committee to the EC’s aid management.

ANNEX I.

DISTRIBUTION OF EC AID TO ALA COUNTRIES BY COUNTRY

ANNEX II:

DISTRIBUTION OF EC AID BY OBJECTIVE BY REGION

ANNEX III:

ORGANIGRAMME OF THE RELEX COMMON SERVICE

ANNEX IV:

COMPARATIVE OUTLINE OF CR 442/81 AND CR 443/92

ANNEX V:

TERMS OF REFERENCE FOR THE ALA SYNTHESIS REPORT

ANNEX VI:

SUMMARY OF MAIN STUDIES OF DESK STUDY

SUMMARY OF MAIN FINDINGS OF DESK STUDY

Generally satisfactory

Relevance

Much of the aid conforms to the specific policy objectives progressively laid down in the main EU textrelevant to the subject: Regulations 441/81, and 443/92. Some exceptions: relatively small amounts of aid aredirected to the poorest; the target of 10% for actions concerning environment had not been met in LA by1995; gaps of documentation prevent evaluating programmes in certain areas (e.g. human resourcesdevelopment). Coherence between action and policy appears generally good.

Overall consistency between Recipient Country, needs and EU actions

While documentation is scarce also on this point, it is noted that EU actions are the result of negotiation withthe Recipient Country - the latter having a paramount role in ensuring consistency of EU development co-operation with its national development policies. Effectiveness

The achievement of the expected outputs was highest in programmes directed to immediate needs. Lowerdegrees of effectiveness were perceived in hard-to-quantify areas, such as structural reform and institution-building. However, the documentation pointed out several important obstacles to effectiveness. A group ofdifficulties concerned the performance of the Recipient Countries. A second group represented externalfactors and, more specifically, trade conditions. A third was constituted by a number of budget constraints,rigidity of procedures or operative weaknesses affecting the EU management of aid programmes. Althoughthe rating is «satisfactory», greater attention to this aspect is recommended.

Occasionally satisfactory

Complementarity

Evidence indicates that complementarity between EU programmes and programmes financed by MemberStates or multilateral institutions occurs in an occasional, rather than systematic way. However, a fullassessment of complementarity would require detailed knowledge of all undertakings enacted or supported byall donors. A master table of external assistance is not readily available. Hence, the judgement on the issueremains largely intuitive.

Not assessable

Efficiency

Documentation is lacking in respect to efficiency considerations. Dearth of the quantitative elements (cost-benefits) impedes the assessment of the degree of efficiency attained by the development co-operationfinanced by the EU. This is a serious sore spot. At present a judgement about efficiency appears to bepossible only by direct, ad hoc study of individual cases.

Impact

While in the last few years there were significant improvements in defining long-term impact, none of thesubsequent more recent programmes has yet been submitted to evaluation. At the present stage ofdocumentation it is not possible to express a judgement on the impact of programmes financed by the EU.Also in this case, such a judgement would be possible only on the basis of direct, ad hoc study of individualcases.

Sustainability

Lacking efficiency measurements and impact assessment, evaluators have no objective basis for judgingsustainability. Here again, such a judgement would require study of individual cases.

Coherence

Internal coherence of EU development co-operation programmes is rarely discussed in the EUdocumentation. This would suggest that this is not considered a particularly important requirement. There areno operational directives on maintaining coherence among different programmes and between theprogrammes, on one side, and the principles and policies by which the programmes were inspired, on theother side.

Disbursement record

The disbursement commitment (D/C) ratio for ALA over the period 1986-1995 was 62%. This compares toa 78% D/C ratio for MED. There was a slight deterioration in the ALA D/C ratio for 1991-1995 to 60%. Asto ACP, its performance would be influenced by the weight of programme aid on total commitments –25%v. ALA’s 0.4% (STABEX). The D/C figures for Asia and Latin America are broadly similar.

ANNEX VII:

EXECUTIVE SUMMARY FOR THE FIELD STUDY

1. EXECUTIVE SUMMARY

1.1 The mandate

1.1.1 The Field Study of the management of EC-financed aid projects is part of an overall aid evaluationrequested by the EC Council. It was preceded by a Desk Review of the same subject matter and will befollowed by a synthesis of the desk and field conclusions. The Study’s Inception Report and Terms ofReference have been cleared by the Evaluators Working Group, EWG, composed of representatives ofMember States, MSs, and of the EC.

1.1.2 The subject of the Study were 14 projects in development co-operation, Dc, and economic co-operation, Ec, located in Honduras (5), Peru (3), Philippines (3) and Vietnam (3) completed or initiated in theperiod 1986-1995. The selection was made as follows:(a) Development co-operation projects were chosen to form a package of income-producing initiatives

(prevailing in Peru and in The Philippines) and social service initiatives, including humanitarianoperations (prevailing in Honduras and Vietnam). Only one Ec project was included (in The Philippines)as such projects are far less numerous than Dc’s;

(b) Countries were chosen to represent different degrees of aid absorptive capacity: strong in Peru and ThePhilippines, and weak in Honduras and Vietnam. The choice reflects the fact that recipient country’saction is as relevant to the results as the financing agency’s.

1.1.3 A significant part of the research concerned the EC Rules and Regulations, RRs, relevant to projectmanagement from initial policy to final evaluation. The research was conducted at the EC in Brusselsconcurrently with a review of the ALA Committee’s activity.

1.2 Making the evaluation

1.2.1 Disciplines represented in the five-member evaluation team, headed by Mr. V. Masoni, were economics,sociology, political science and engineering. The Field work took place in the first semester of 1998. Afterreturning from the field, further exchanges were made with the EC Headquarters, and EC offices inTegucigalpa (Honduras), Lima (Peru), San Jose (Costa Rica), Bangkok (Thailand), Hanoi (Vietnam), andManila (The Philippines).

1.2.2 This Report presents the Study results. An Aide Mémoire for each country is annexed. While drawinglargely from EC staff, MSs representatives and recipient country sources, as well as from EWG’s extensivecomments, the Report reflects solely the evaluators’ views.

1.3 Systemic aspects

1.3.1 The selected country/project package do not form a statistical sample of the far larger and diversifiedEC aid portfolio in the Asia and Latin America regions, ALA. However, it includes such a wide variety ofsituations as to expose central aid management issues.

1.3.2 To widen the base for systemic conclusions beyond the chosen projects/countries, the evaluators: (a)discussed with EC staff, recipient country and MSs representatives in the field their experience with any ECproject, (b) examined EC RRs, which impart directions to the system, and (c) reviewed the ALA Committeeactivity, including the records of the exchanges between the EC and MSs representatives on projects and onselected management issues, (d) reviewed (i) aid co-ordination, (ii) EU in-country presence, and (iii) Ec issues.

Overall findings

1.4.1 Rules & Regulations (RRs) and Management-by-Objective (MBO). On the whole, the Studyfound that the EC managed the projects with proper administrative care, conforming to its own RRs.However, reliance on RRs is too heavy; a more entrepreneurial approach should be considered. RRssimplification and modification could incorporate an MBO approach, and encourage managers to applythemselves to the fullest of their professional competence and direct knowledge of the case at hand. (See1.5.1-13).

1.4.2 Country Strategy. In the four countries the EC and the government have agreed on a three-yearactivity program. However, none has yet an aid strategy based on full-fledged analysis of priorities andoptions. All projects reviewed were unquestionably in line with the Council Resolution, CR 443. In particular,on different occasions, EC had exhorted the recipient country to act on human rights, good governance,political pluralism, gender parity, and environmental protection; and Dc was directed to groups that wereextremely poor and/or socially excluded. Still, that has not been enough to give cohesion to the aid activities.

1.4.3 The decision to institute the requirement of: (a) country strategy, and (b) analysis of project alternativesis commendable, although considerably belated in respect to other aid organisations. The decision of theALA Committee to add the review of country strategies to that of individual projects should spur theapplication of the aforementioned requirements, although it would add considerably to the Committee’sworkload.

1.4.4 The quality of the country’s project management was varied, largely depending on the strength ofinstitutions and on circumstances. Changes of governments and turnover of country’s civil servants were veryinfluential; the Delegations’ performances were not significantly different from one another. Following arethe judgements on the composite EC-country performance:

• Initiative (Preparation): sufficient as to general planning. Although lacking strategy formulation, theprojects were in line with CR 443 and aimed at significant development objectives (One “streetchildren” project in Honduras was an exception; its very design indicated that it could have had only aminimal impact);

• Appraisal and Decision-Making: insufficient in regard to ex ante evaluation, which needs more timeand greater depth, as does the analysis of sector and project alternatives. Financial Agreements, FA, aretoo sketchy for activity planning and too rigid for a proactive management. In the first instance, theyleave many aspects to be defined in the execution phase, thereby opening the way for differences ininterpretation and time-consuming compromises, as well as requiring substantial additional work for theformulation of the Overall Working Plan, OWP, and, later, for project closure and transfer. In thesecond instance, FA are very formal instruments, whose modification is burdensome even when therationale for modification is irrefutable and the need to proceed is urgent.

• Implementation: good. However, preparation and approval of OWP and Annual Working Plans,AWPs, are slow, with a negative impact on the disbursement of funds;

• Progress Reporting and Monitoring: good in complying with formal requirements. Howeverreporting is generally descriptive, and weak on the discussion of implementation issue, and ofoptimisation alternatives. Many projects lacked a full-fledged internal Monitoring and Evaluation, M&E,system, although this has changed in recent years. There are now countries in which all projects areequipped with M&E: this was gathered from

• reliable interviews and verified in The Philippines (See 1.5.11 Better project documentation);• Ex-post Evaluation: unsystematic; ex post evaluations are still infrequent;• Management Information: The information provided by the project reports is generally insufficient.

Serious thought may be given to overhauling the Management Information System, MIS, relevant to aidactivities. Part of that would be the reorganisation of the filing and archives system at Headquarters.

Quality, Timeliness and Recipient Country Factors

1.4.5 A review of performance in regard to these factors does not lead to sharp generalisations, either positiveor negative. Performance has appeared strongly dependent on external circumstances, on local factors, andon the inherent complexity of projects with marked social character and calling for broad local participation.

1.4.6 In respect to the “3 Cs” requirement, the Study has focused on Co-ordination and Complementarity inrespect to aid projects. It has not been able to consider Coherence, which refers to harmony of EC and MSsactions in all endeavours - a subject totally outside the inference potential of the project package. All actorson the EC and the MSs side appeared to consider Co-ordination and Complementarity important thoughproblematic to pursue. In effect, each party strives for them with disparate results, depending oncircumstances, the project nature, and the staff available (generally minimal). In the end, the paramount factoris the inclination of the recipient country to support (or to oppose) donor co-ordination.

1.4.7 While performing as an MSs device for project quality control, the ALA Committee provides a mass ofinformation on EC aid activities that can be used by the MSs for program co-ordination purposes. However,as to quality control, it is noted that by the time Financing Proposals, FPs, are brought to the Committee, theproject is already so firm as to be difficult to modify. As to Co-ordination, each MS has its own planningcycle, and cannot adjust its project pipeline in lock-step with the EC project submissions to the Committee.In practice, Co-ordination is easier at the country level. (e.g., the Co-ordination Pilot Project in Peru) becausespecific project issues are evident to MSs representatives, EC staff and country counterparts alike. However,at country level, professional manpower constraints are severe for both Delegations and MSs Embassies orequivalent representative entities.

1.4.8 The “3Cs” appears objectively hardest to pursue in Ec, since each MSs endeavours to support theinterests of national companies: business is inherently competitive. Acknowledging that, it may be decidedthat Ec:• Improve equal access mechanisms and transparency of the allocation of its benefits to private firms

(weak in the Ec project in The Philippines), or• Concentrate on areas of common EU interests: trade fairs, institutional advertising for Europe,

information about industrial, technological, and research developments in Europe and alike (e.g., the EUInternet site in The Philippines).

1.4.9 Practically all projects have a very slow beginning, owing to a slow interaction between Headquarters,Delegation and country. The suggested RRs simplification and greater discretionary power to Delegationswould substantially speed up operations while keeping performance risks within reasonable bounds (See1.5.3).

1.4.10 In the cases included in the Study, the formulation of country plans and projects had been made withclose Government participation. The depth of participation varied greatly with the country’s: (a) political will,and (b) technical capacity and commitment to development of central and local institutions, as well as theweight of social intermediaries representing final users. The degree of co-operation in implementation wasgood in all cases, although delays occurred very frequently owing to country’s budget stringency and/or ECRRs on contracting, initial financing, disbursement processing and project hand-over.

1.4.11 The Study noted the EC drive for aid management rationalisation, beginning with the adoption of theintegrated project cycle concept and of the Logframe (1993). The drive has led to increasing clarity ofdirectives, staff information and training, internal re-organisation and cost-cutting. Simultaneously, though,economy drives on personnel costs may have affected some operations as well as the EC institutionalmemory. On the whole, there are still significant needs, and opportunities, for improving managementquality, as well as for reinforcing the EU presence in recipient countries and in international aid circles.

1.5 Areas for improvement

Concern with development results.

1.5.1 The relevance and effectiveness of aid programmes are increasingly questioned, world-wide, and MSsresources are the target of many other national or international claims. The EC may be expected to achieve asuperior performance in aid activities. Should additional discretionary powers be granted to managers, thelatter would also have the responsibility for demonstrating a quality performance.

1.5.2 An issue to be considered is the passage from “administrative care” to an “entrepreneurial approach”,such as a full-fledged MBO, aiming at well defined country, sector and project objectives, at the lowest totalcost and with quality and time specifications. The approach is suitable to any organisation with competentstaff: that is the EC case.

1.5.3 The risk of leaving more space to offices’ and officers’ judgement may be considered acceptable ifmanagers could be assisted by:

(a) A strengthened system of Monitoring and Evaluation during execution;(b) Reinforcement of informal, expeditious consultations, within EC, among peers of different disciplines,

as well as peers with similar tasks in different offices;(c) Selection of staff with significant managerial experience, experience that cannot be borrowed from

consultants or short-term appointees.

Tightening timing

1.5.4 By all indications, a drastic acceleration of project operations is both needed and possible. Processsimplification and greater delegation should be accompanied by detailed analysis of the project cycle toidentify the specific actions in which delays can be avoided.

Country strategy

1.5.5 The decision of the ALA Committee to add the review of Country Strategies to that of individualprojects should be implemented substantively and sustained through time, even at the expense of a lessdetailed project review by the Committee.

1.5.6 It appears that the EC will continue to draw on country reports, sector studies, thematic papers, anddevelopment statistics issued by other development institutions. While this represents a considerable savingfor the EC, a stronger presence in developing countries and in international circles may suggest theproduction of materials stemming from European thinking and values, not necessarily coincident with thoseof non-EU sources.

The “3Cs”-- a challenge

1.5.7 At the country level the main avenue to foster Co-ordination and Complementarity among all externalaid programs is the EU (EC and MSs) participation in international consultative groups and consortia. Themove to country strategy would reinforce the EU unitary presence, now feeble on that front. Still, achievingcohesion in aid programs has proved to depend on many hard-to-manage factors. That task is likely torepresent a permanent challenge for all parties, EU and non-EU.

Revamping the aid management system

1.5.8. The revision of the aid management system under way at the EC is aimed at (a) better responding tothe policy directives (i.e., CR 443 directives on poverty alleviation, gender parity, etc., and to evolvingsituations in developing countries), and (b) improving efficiency and effectiveness of the EC aid system.Following are the evaluators’ comments:

Rules & Regulations. An extensive restructuring and streamlining drive of the aid-relevant RRs startedseveral years ago. Although most of the rules are now operational, a proper evaluation of their impactrequires a longer observation time. Meanwhile, further improvements could be considered in the sense of:• Granting more leeway to staff initiative and judgement,• Indicating time-spans and estimated workloads for major actions RRs prescribe,• Re-grouping RRs more tightly by subject matters, and placing them on electronic support to facilitate the

search of topical information.

Strategic planning. In the last 3-5 years, increasing emphasis has been given to country strategic thinking.The momentum of this change is augmented by the fact that the ALA Committee will review the countrystrategy papers. The next steps to consider would be to: (a) further develop the present three-years activityprograms into full-fledged medium and long term strategies, and (b) extend the requirement to all countries.

Reorganisation. In 1998, a major re-organisation of aid activities has been made through:• The creation of the Service Commun Relex, a Commission service resulting from the merger of technical

and financial services of DG I, DG VIII, DG IA and DG IB.• A clearer definition of relations between Headquarters and Delegations.If a more entrepreneurial approach is adopted, a dedicated research is likely to suggest further changes. Oneissue is whether Delegations should act as EU representative offices or EC operational outposts - or a welldefined blend of the two. At present, workloads are heavy and far too miscellaneous. Unlike in ACPDelegations, the aid professionals of the Delegations visited consist of 2-3 units with general assignmentsrather than aid sector roles. Since staff have substantial Dc experience, generally acquired working on ACPsbefore transferring to ALA regions, they are able to handle project matters, case by case. But, detrimentally toefficiency, they are neither project nor project portfolio managers, acting essentially as all-round “fire-fighters” on representational and operational fronts.

Better project documentation. The quality of the project analytical work has improved in the periodrelevant to the projects under review. Among others, the project Logframe has been applied for several years.Substantial further advances should be considered in three areas:• Ex ante evaluation (including the discussion of alternatives and, overall, increasing the depth of

evaluation);• Monitoring and evaluation (instituting M&E as a permanent project feature, as it was done in The

Philippines and as it is under way in the three other Delegations);• Ex post evaluation (increasing frequency).The unification and re-organisation of records would facilitate staff work and make the process moretransparent. To speed up retrieval, major reports and decision papers may be placed on electronic support,like the RRs, so as to be scanned through the “search” function.

Economic co-operation’s prospects

1.5.9 At the moment, financial and staff commitments on Dc are far greater than the correspondingresources invested in Ec. The Study has not traced the criteria used by the EC to decide the Dc-Ec mix in agiven country. The formulation of country strategy will necessarily focus also on that question. In so doing, itought to consider that:• Europe’s presence is weak in Asia, while stronger to a degree in LA;• Certain countries offer more substantial business opportunities than others (e.g., The Philippines and

Peru);• Private enterprise participation would: (a) contribute to filling the resource gap of the recipient countries;

(b) bring local entities to the fore thereby widening empowerment, a prerequisite for greater politicalpluralism in the country, (c) increase awareness of the global interdependence between donor andrecipient countries, and

• Going concerns - i.e., investment, trade and cultural EU-country partnerships -establish a more lastingin-country presence than most traditional aid projects. In respect to in-country presence, due recognitionshould also be given to information and public affairs activities, such as those carried out by the CostaRica Delegation for the entire Central America.

1.5.10 Comments gathered in broad interviews with EC staff and MSs representatives in the field, as well asgeneral considerations, would suggest that a good case could be made for enhancing Ec in ALA countries.

Errata-Corrige: On page 39 of the Report, please read M&E rather than M&V

ANNEX VIII:

ALA COMMITTEE QUESTIONNAIRE