Quality Assurance Group - documents.worldbank.orgdocuments.worldbank.org/curated/en/180811468313530171/pdf/445710WP0BOX...SAP user friendliness, insufficient guidance/support to TTLs)

Quality Assurance GroupAssessment of the Quality of “Other

Economic and Sector Work”March 15, 2005

44571

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Quality Assurance Group 2

TABLE OF CONTENTSTABLE OF CONTENTS

I. Executive Summary 3

II. Introduction 12

III. Findings 18

IV. Recommendations 35


I. EXECUTIVE SUMMARYI. EXECUTIVE SUMMARY


WHY OESW?

FBoard concern over rapidly growing share of OESW in AAA program, more than doubling in dollar terms (from $11m to $26m) and in number (from 103 tasks to 247 tasks) between FY01 and FY04.

FOESW never systematically reviewed by QAG.1/

1/ However, Policy Notes were included in the standard QAG ESW Assessment.


WHAT IS “OESW” 1 / ?

Policy Note: An informal report providing “just-in-time”advice to a client on a range of development issues.

Consultations/Country Dialogue: Analytic support provided to clients over a period of time to facilitate their decision making process.

Conference/Workshop: A forum used as a vehicle for the Bank to communicate its analytic findings with the purpose of influencing the client’s policies.

1/ OESW is a convenient, informal term adopted by the assessment team to refer jointly to the three above specific output types within the ESW product line. This definition of OESW applies to the FY03-04 review period. As of 07/01/2004 OPCS issued new instructions to the Regions, stating that Consultations/Country Dialogue and Conference/Workshop are no longer valid outputs for ESW.


THE SAMPLEFThe FY03-FY04 OESW universe included 5621/ tasks for a

total cost of $48.91/ million.

FOESW sample includes 110 randomly selected tasks for a total cost of $16.3 million, representing 20% of all tasks by number and 33% by cost.

FTasks costing less than US$ 20,000 were excluded from the sample.

FSample yields robust results at the Bank-wide level and for large Regional/Network cohorts.

1/ These figures are now 530 tasks for a total cost of $47.9 million as a number of tasks have been recoded during this assessment.


MAIN FINDINGS

F 52% of tasks sampled (57 out of 110), accounting for 44% of the sample cost, were improperly recorded as OESW.

FAbout 70% of Conferences/Workshops and Consultation/ Country Dialogue tasks were improperly recorded as OESW.

F Improper recording is primarily caused by a weak overall governance framework (unclear product definition, low SAP user friendliness, insufficient guidance/support to TTLs).


FOnly 53 of the 110 sampled tasks1/, amounting to $9.1 million, were assessed and rated using the OESW guidance questionnaire.

FAt 96% overall SAT2/ for the number of tasks and 95% by value, ratings are slightly above those for FY02 ESW, and exceed the 90% SAT target for ESW.

FHowever, about a fifth of the tasks were only rated moderately satisfactory, revealing scope for further improvements.

FAll Regions and Networks recorded strong ratings but reveal missed opportunities to maximize impact due to insufficient attention to dissemination arrangements (planning and funding), and insufficient focus on quality enhancement, particularly peer reviews.

MAIN FINDINGS …Continued

1/ Ten of these 53 tasks, which should have been coded Formal ESW reports, were assessed. Ratings for these ten tasks are similar to those of the 43 OESW tasks.

2/ SAT = Moderately Satisfactory or Better.


F ECA and HDN results show missed opportunities to increase impactthrough more succinct conclusions and recommendations and betterclient engagement and participation.

F Ratings for Strategic Relevance and Timeliness (100% SAT) indicate that all tasks assessed were implemented within a well defined strategic framework.

v A solid result (with 47% of tasks rated Highly Satisfactory, 43%rated Satisfactory and only 10% rated Moderately Satisfactory).

v However, many of the tasks would have benefited from the introduction of success indicators.



FRatings for Bank Inputs and Processes (81% SAT– but with 31% of tasks being rated Moderately Satisfactory) reflect missed opportunities for greater impact due to inadequate managerial attention to quality and lack of peer review. The assessment also found weaknesses in the application of the Bank-wide ESW governance framework, including:

vSignificant understatement of OESW costs as several cost items not accounted for (TF, cross support, WBI and seconded staff);

vUnclear product definition (policy notes), obsolete regional guidance and low SAP user friendliness.



KEY RECOMMENDATIONS 1/

OPCSF Issue new guidance to TTLs for coding Consultation/Country

Dialogue and Conference/Workshop tasks outside the ESW product line – Recommendation implemented early FY05.

F Improve SAP’s user friendliness and interactivity and establish an ESW portal to provide guidance to users – Recommendation well under implementation.

F Promote the identification and funding of dissemination strategies at concept stage – Recommendation well under implementation.

F Explore, jointly with SFR, ways to ensure full accounting of tasks’costs (to include TF, cross support, WBI and seconded staff).

1/ The full list of recommendations can be found in Chapter IV.


II. INTRODUCTIONII. INTRODUCTION


F Promote accountability for quality by providing indicators on quality of OESW work:

• Bank-wide

• By Region; and

• By Network/Sector

FEncourage systemic change through:

• Improved understanding of key determinants of OESW quality

• Disseminating assessment findings to appropriate Bank units

OBJECTIVES


F Methodology and Guidance Questionnaire were developed following a pilot OESW assessment conducted under the FY02 ESW assessment.

F Tasks costing between US$ 20,000-$50,000 were reviewed by a “triage panel”and were rated using a simplified version of the Guidance Questionnaire 1/. All other tasks were rated by a customized panel.

F The overall quality of each task was assessed on four quality dimensions (Strategic Relevance and Timeliness, Internal Quality, Dialogue and Dissemination and Likely Impact). Bank Inputs and Processes were also assessed but were not take taken into account in the overall quality rating.

F A Stage 2 assessment was available to all tasks not rated SAT in the first review. This was requested for only 2 tasks.

F 12 Bank staff participated in the assessments as observers.

F 39 panelists conducted the assessment.

1/ The simplified questionnaire provides and overall rating and a summary rating for each quality dimension but does not include ratings for individual questions.

METHODOLOGY


FPanelists used QAG’s new six point scale:

vHighly Satisfactory (1): Best practice in most respects and no significant deficiencies

vSatisfactory (2): Satisfactory or better on all aspects.

vModerately Satisfactory (3): Satisfactory on all key aspects but significant missed opportunities

vModerately Unsatisfactory (4): Significant deficiencies in a few key aspects

vUnsatisfactory (5): Significant deficiencies in several key aspects

vHighly Unsatisfactory (6): A broad pattern of deficiencies

METHODOLOGY…Continued


FThe formal trigger for entry into the universe was the standard SAP milestone “Delivered to the Client” at the time of sampling for each batch.1/

FThe FY03-F04 OESW task universe includes 562 tasks for a total cost of $48.9 million.

FAll tasks with a cost of less than $20,000 were excluded from the universe.

FThe sample was stratified by cost ($200k+, $100-$200k, $50-$99k and $20,-$50k) and type of output (Policy Notes, Consultations/ Country Dialogue and Conferences/ Workshops).

1/ Tasks later reclassified remained in the universe.

SAMPLE


FThe sample included 110 randomly selected tasks with a total cost of $16.3 million, representing 20% of the universe and 33% of its cost.

FThe sample was selected in 5 batches.

F Statistical Robustness:

v Bank-wide, at 95% confidence level with a 7% sampling margin of error;

v For Regions and Networks, at 90% confidence level with a 5-25% sampling margin of error depending on cohort size;1/

FResults in this report are presented on a weighted basis to adjust for over/under representation in the sample because of stratification and sample design (the weighting is explained in Annex 5).

SAMPLE ROBUSTNESS

1/ Results are less robust for smaller cohorts (PSD, LCR, AFR, ESSD and INF).


III. FINDINGSIII. FINDINGS


F 52% of tasks sampled were miscoded (571/ tasks out of 110 sampled) representing 44% of the $ amount ($7.2 million out of $16.3 million).

F An equal percentage of tasks were miscoded in each of the two years covered by the review (FY03 and FY04).

F Miscoding was found across all Regions and Networks.

F There is no significant correlation between tasks’ cost and the likelihood of miscoding. The average cost of miscoded tasks is $127,000 while the average cost of all tasks is $149,000.

1/ In addition, another 10 tasks were coded OESW but should have been coded as ESW Reports.

CODING


FOf the 57 Miscoded tasks:v 23 should have been coded as internal or external relations products.

v 21should have been coded as Non Lending Technical Assistance (NLTA).

v 9 should not have been created as separate tasks since their only objective was to disseminate the findings and conclusions of a main ESW task.

v 4 were prematurely recorded as delivered to the client.

CODING …Continued

Type of Task No. of TasksNo Percent

Conference / Workshops 29 20 69Consultation / Country Dialogue 30 21 70Policy Notes 51 16 31

Total 110 57 52

Miscoded

Miscoded Tasks


FOnly 53 tasks were assessed and rated (the balance of 57 tasks could not be assessed and rated as they were not ESW).

FLike FY02 ESW, the most frequently stated development objective of OESW is Structural and Sector Policy Reform.

FOverall quality of the FY03/04 OESW sample (96% SAT by number of tasks and 95% SAT by value) is slightly above the FY02 ESW assessment (92% SAT by number and 94% SAT by value) and exceeds the 90% SAT target for ESW. SAT ratings were similar for each of the two FYs covered by the review.

RATINGS


F Distribution of Ratings:

10010053Total

121N/A 1 /

000Highly Unsatisfactory

000Unsatisfactory

342Moderately Unsatisfactory

242212Moderately Satisfactory

455328Satisfactory

271910Highly Satisfactory

% of $ Amount%No. of Tasks

Rating

1 / One task was not rated as it had been an input into a previously Qagged ESW task.

RATINGS …Continued


HNPAccelerating progress towards HNP MDGs

World

Social ProtectionWBG Pensions ReformWest Bank & Gaza

Economic PolicyPY Policy OptionsParaguay

Economic PolicyPK- Trade Policy NotesPakistan

Economic PolicyLAC Meets the Market Latin America

Urban DevelopmentJO – Urban Management StrategyJordan

HNPIran Environmental Health (Hospital Waste)

Iran

Social ProtectionREG ROMA CONFECA

Economic PolicyChile New EconomyChile

HNPHealth Financing OptionsAfrica

OPERATIONS RATED HS


F 10 tasks or 18% of the number of tasks assessed and 17% of total $ amount were rated HS.

F The percentage of HS tasks in OESW (18%) is slightly below the FY02 ESW level of 20%.

F Tasks above $100,000 are as likely to be rated HS as those below $100,000.

F Of the 10 HS tasks, 9 were clustered in two Networks (HDN & PREM) and 6 were clustered in two Regions (LCR & MNA).

OPERATIONS RATED HS


FOutstanding aspects of HS tasks include:v Team skill mix;v Strategic relevance, particularly related to CAS and to

client programs and objectives;v Timeliness in terms of client capacity and Bank’s

programming agenda;v Alignment of coverage/scope with objectives;v Clear identification of critical issues;v Quality of analysis;v Quality of partnership arrangements with other

donors;v Appropriate dissemination arrangements.

OPERATIONS RATED HS …Continued


REGIONAL/NETWORK RESULTSFOverall ratings show strong results for all Regions and

Networks but some missed opportunities to maximize impact caused by insufficient focus on dissemination of findings and weak managerial attention to quality enhancement, particularly peer reviews.

F Potential weaknesses were identified in ECA and HDN (internal quality, particularly the quality of conclusions and recommendations, the extent of client engagement and participation, the degree of consultation with other stake-holders) that could also limit impact.


RESULTS BY QUALITY DIMENSION

Quality Dimension HS S MS SAT

Overall Assessment 18 55 23 96Strategic Relevance & Timeliness 47 43 10 100Internal Quality 17 57 19 93Dialogue and Dissemination 26 46 19 91Likely Impact 10 57 27 95Bank Inputs and Processes 14 36 31 81

<----------------Percent--------------->

Percent of Projects in Each Rating Category


STRATEGIC REL. & TIMELINESS(100% SAT)

F Results for this dimension are particularly strong; 47% of the tasks were rated Highly Satisfactory, 43% Satisfactory, and only 10% were rated Moderately Satisfactory.

F Assessed tasks were implemented within a well defined strategic framework.

F Strong timeliness of the tasks in terms of client capacity and Bank programming agenda.

F Clarity of objectives at entry would have benefited in several instances from the use of a Concept Note or more extensive internal discussions.

F A number of tasks lacked explicit indicators of success.


INTERNAL QUALITY (93% SAT)

F Task’s scope and coverage are well aligned with objectives.

F Timely identification of critical issues and the quality of empirical evidence and analysis are strong aspects.

F But a few tasks exhibited missed opportunities as conclusions were not properly articulated and recommendations were not bold enough.


DIALOGUE & DISSEMINATION(91% SAT)

FWith 23% of tasks being rated Moderately Satisfactory, there are significant missed opportunities mainly due to inadequate dissemination arrangements:

ØConcept notes are generally silent on dissemination arrangements and these efforts are rarely adequately budgeted;ØOESW tasks could have greater impact if dissemination

efforts were to specify the instruments to be used and the target audiences and actors to focus on;ØThere is considerable scope for better dissemination of

findings in the Bank.


LIKELY IMPACT (95% SAT)

F Tasks are expected to have a strong impact on the development community and the Bank.

F But 27% of the tasks received a Moderately Satisfactory rating for this dimension because of missed opportunities to maximize impact:

ØConcept notes often did not discuss the need to build ownership and coalitions for change;

Ø Insufficient focus on dissemination activities.


BANK INPUTS & PROCESSES(81% SAT)

F Task team members generally have appropriate skills.

F Managerial attention (CMU & SMU) both at entry and during implementation is weak, particularly for tasks below $200,000.

F Peer review process is sometimes lacking, particularly for tasks below $100,000.


BANK INPUTS & PROCESSES …Continued

F Significant understatement of OESW costs as several types of expenditures not recorded:

ØTF contribution often not accounted for;

ØCross support is not properly captured;

ØContributions from WBI as well as from staff seconded to the Bank are frequently not accounted for.

F SAP not sufficiently interactive and user friendly as it does not provided adequate guidance to users to prevent task miscoding.


F There are wide variation in Regional and Network guidance to staff regarding OESW.

F Some Regional/Network guidance are out of step with corporate level recommended good practice as presented on the OPCS website (need for concept paper, peer review, etc).

F Bunching is significant with about 40% of tasks delivered to the client in June.

F Many TTLs are unclear about the various product definitions.

BANK INPUTS & PROCESSES …Continued


IV.IV. RECOMMENDATIONSRECOMMENDATIONS


FOPCS should:

vIssue new guidance to TTLs for coding Consultation/Country Dialogue and Conference/Workshop tasks outside ESW product line –Recommendation implemented early FY05;

vImprove SAP’s user friendliness and interactivity and provide guidance to users by establishing an ESW portal – Recommendation well under implementation.

OPCS


OPCS …Continued

F Promote the identification and funding of dissemination strategies at concept stage – Recommendation well under implementation.

F Explore, jointly with SFR, ways to ensure full accounting of all task costs (including TF, cross support, WBI and seconded staff).

F Conduct a review of Regional and Network ESW guidelines to ensure appropriateness and consistency –Recommendation well under implementation.


REGIONS/NETWORKS

F Regions and Networks should identify ways to reduce task bunching at FY’s end.

F Sector managers should pay more attention to OESW quality enhancement activities, and emphasize the importance of proper SAP data input, including selection of result indicators at the concept note stage, and rating the extent to which the results have been achieved at task completion.

F ECA and HDN should consider measures to strengthen arrangements for client engagement and participation, consultation with other stake-holders, and internal quality enhancement, particularly in regard to conclusions and recommendations.

OTHER ECONOMIC SECTOR WORK ASSESSMENT

ANNEXES

March 15, 2005

2

ACRONYMS AND ABBREVIATIONS

AAA Analytical and Advisory Services ARPP Annual Review of Portfolio Performance AFR Africa Region CAS Country Assistance Strategy EAP East Asia and Pacific ESW Economic and Sector Work ECA Europe and Central Asia LCR Latin America and the Caribbean OESW Other Economic and Sector Work MNA Middle East and North Africa Region QER Quality Enhancement Review QEA Quality at Entry Assessment NGO Non-Governmental Organization PAD Project Appraisal Document PIP Project Implementation Plan OPCS Operations Policy & Country Services PRSP Poverty Reduction Strategy Paper QAG Quality Assurance Group QEA Quality-at-Entry Assessment SAL Structural Adjustment Loan SAR South Asia Region TA Technical Assistance VP Vice President TTL Task Team Leader WBI World Bank Institute

Acknowledgement

A team managed by Xavier Legrain, which included Saeed Rana and Amnon Golan, prepared this report. Data processing and analysis was done by Melvin Vaz. Statistical analysis and sampling was performed by Jason Mayfield. Assessment logistics and documentation were managed by Vinod Ghosh. QAG wishes to thank the panelists, observers and task teams who participated in the exercise, as well as the quality assurance staff in the regions for their support and valuable contributions. Mr. Prem Garg, Director, Quality Assurance Group, guided the overall effort.

TABLE OF CONTENTS

Annexes

1. Lists of Tasks ...........................................................................................................4

2. List of Main Contributors ........................................................................................6

3. Approach Paper........................................................................................................7

4. Guidance Questionnaire .........................................................................................17

5. Methodology for Sampling, Stratification and Weighting ....................................24

6. Statistical Tables ....................................................................................................26

5

Annex 1 List of Tasks

Africa (AFR) East Asia and Pacific (EAP)Africa SP-Africa Dutch TF MADAGAS Poverty&ShockCambodia KH Country Gender AssessmentAfrica Uganda Procurement Hub Workshops China CN-Road Traffic SafetyAfrica RP: LKD - Dis. of Pov. Dynamics China CN Tax Policies to Promote NGOsAfrica Gender-GPG funds China Rural Finance and DevelopmentAfrica Econ. Act/Pov. Assess. China PSD Strategy in Lagging RegionsAfrica Customs Administration Issues China Subregional Fiscal and Poverty StudiesAfrica ESSD:Agric.Tech-Sustainable Systems for China CN-HOUSING MAINTENANCE & OPSAfrica ESSD:Reg.Tree crops Policy Note EAP Asset-backed SecuritizationAfrica Records Management Capacity Framework EAP 4E-Reg. HIV/AIDS (FY04)Africa Health Financing Options Indonesia ID - Feeding IndonesiaAfrica Regional study-Sec. Edu.(SEIA) Indonesia Financial Sector StrategyAfrica Regional Health Strategic Options Paper Pacific IslandsPI-Regulatory AssistanceGuinea-Bissau GW: Sustainable Fisheries Mgt Strategy Philippines PH-Urban InfrastructureLesotho Lesotho Financial Sector Update PNG Rural Development ForumMalawi MW: Agriculture Policy Options Thailand Thailand FY04 Country Dialogue MonitorNigeria LKD NG ANTI-CORRUPTION THAILAND TH ICT/Knowledge Economy PartnershipNigeria Informal Macro Update Note Vietnam VN - Rural Inputs to PRSC II and IIINigeria Nigeria: Fiscal Federalism/Decentralization Vietnam Vietnam WTO AccessionNigeria Nigeria Policy Notes Vietnam VN-Transport & Trade FacilitationNigeria NG Power Sector Policy Reform DialogueSouth Africa S. Africa - PSD/ICATanzania Study of Post-Primary Education and TraiUganda Trade Export Competitiveness

ECA Latin America and the Caribbean (LCR)Azerbaijan EDUC SECT NOTE Brazil BR Brown Agenda Sector DialogueAzerbaijan TRADE DIAGN STUDY Brazil BR-Policy Note Follow-Up/PublicationBosnia-HerzegovinaLOC GOVT FIN STUDY Brazil BR Fiscal Policy for Investment GradeBosnia-HerzegovinaBANK SECT RVW Chile CHILE NEW ECONOMYCentral Asia SCALE UP CDD IN CA Colombia CO GENDER PORTFOLIO REVCentral Asia WATER-ENERGY NEXUS IN CENTRAL ASIAColombia CO Afro-descendants, Soc. Incl. , MDGsCzech RepublicPPP CONFERENCE Latin AmericaGROWTH IN LAC CONFERENCE 2003ECA ADMIN & MGMT OF MULTI PENSION LCR LAC MEETS THE MARKET ENCOUNTER (2003)ECA HOUSING & LAND MRKTS OECS CountriesOECS - CPAR FOLLOW UPECA Transport and Lbr Mob in ECA Paraguay PY POLICY OPTIONSEurope and Central AsiaREG ROMA CONFGeorgia HEALTH NOTESERBIA POL NOTES (SERBIA)Serbia and MontenegroENV DIAL SSDSouth Eastern Europe and BalkansSEE KNOW ECON INITUkraine TELECOM ICT (UA)

6

Middle East and North Africa (MNA) South Asia Region (SAR)Djibouti DJ Water Sector Note Afghanistan Comprehensive Needs AssessmentIRAN Transport Sector Review Bangladesh BD-SOE WorkshopIran, Islamic Republic ofIran Environmental Health(Hospital Waste Bangladesh BD Dissemination of Country Frmwrk RepJordan JO-Urban Management Strategy India Workshop on EducationMNA HIV/AIDs Action Plans/Surveil. Sys. India Social & Env. Issues in ForestryMNA Workshops on Pension Reform India IN Capital Markets Policy Notes IMNA MNA-Knowledge/ICT Regional Strategy India AP EDUCATION STATUSMNA MNA-Knowledge for Dev. Strategy India IN Banking Sector Policy NotesMorocco Morocco - Govern./Civil Society India State Rural DecentralizationWest Bank and GazaWBG Pensions Reform India Urban Health IssuesYemen, Republic ofRY Gender Mainstreaming India Private Health Services for the PoorYemen, Republic ofYemen - Gender Note India Tamil Nadu Development Agenda Phase IYemen, Republic ofRY-Environment Safeguard India Karnataka Policy Notes 2

Maldives PER Dissemination WorkshopOther (OTH) Nepal Measuring Empowerment and Social InclusWorld Solvency for Developing Markets Pakistan Power Policy NotesWorld UNCSD side event on local leadership Pakistan PK-Poverty Assessment DisseminationWorld Accelerating progress towards HNP MDGs Pakistan FSE (PK)ECON & POL ASPECTS OF FIN SEC WKWorld Reaching the Poor Conference Pakistan PK -Trade Policy NotesWorld SSN/esw - Safety Nets Primer Launch South Asia SAR - Workshop on HIV/AIDSWorld CICMA: DB 2004 Dissemination Sri Lanka Governance Policy NoteWorld Children & Youth

Annex 2

STAFF DIRECTOR: Legrain, XavierMODERATORS: Legrain, Xavier Rana, Saeed Golan, Amnon

LOGISTICS: Ghosh, Vinod

DATA ANALYSIS: Vaz, Melvin Mayfield, Jason

Agueh, Florent Harris, Bruce Shetty, SudhirAlikhan, Mohsin Hasan, Parvez Siraj, KhalidAl-Khafaji, Amir Hayward, John Skolnik RichardBakker, Marie Hume, Ian Socknat, JamesBalkind, Jeffrey Janovsky, Katja Sokol, JoseBerryman, Sue Jimmenez, Emmanuel Steeds, DavidBlaxall, Martha Kharas, Homi Stout, SusanBornemann, Edwin King, Elizabeth Swartzendruber, FredByam, Gerard Ludwig, Peter Varon, BenDevarajan, Shanta Marsden, David Winkler, DonaldFisher, Fritz McCarthy, Eugene Yazbeck, AbdoGopal, Gita Mitton, Giselle Zachau, UlrichHalperin, Ricardo Schiavo-Campo, Salvatore Khawaja, M. A.

Goodland, Andrew Braedt, Oliver Fruman, CecileLaporte, Bruno Huq, Wahida Ivins, IngridStrelkova, Lada Kaoues, Nora Olive, MarshaWilder, Susan Keller, Jennifer Search-Zalmai, Leila

PANELISTS

OBSERVERS

MAIN CONTRIBUTORS

QAG TEAM

Annex 3 ASSESSING THE QUALITY OF OTHER ESW IN FY03-FY04

APPROACH PAPER

1. Other ESW (OESW), like regular ESW, is defined as analytical effort, owned by a specific Bank unit and designed to influence policies and programs of an external client. However, unlike regular ESW, it usually does not have a formal report as its output. OESW includes a diverse range of activities from conferences and workshops, consultations and country dialogue to more structured policy notes. The share of OESW in the Analytical and Advisory Services (AAA) program has increased significantly both in terms of the number of tasks and costs. (Table 1) In FY03, QAG set up a Panel to conduct a pilot review of a sample of 12 OESW tasks delivered during FY02, and costing more than $100,000. 2. After reviewing the results of the pilot assessment, Management has decided that review of Other ESW (OESW) tasks should be broadened. This Approach Paper presents the modalities of implementing that decision. The proposed FY03-FY04 Assessment of OESW will draw and build on the completed pilot. 3. The OESW Program of Quality Assessment is designed to:

• Promote accountability by providing management and staff with robust quality indicators based on credible, independent assessments;

• Identify systemic strengths and weaknesses in OESW and promote appropriate changes in policies and procedures relating to OESW, including its management and review; and

• Learn from experience by disseminating findings and conclusions, including examples of best practice.

Background 4. Given the significant scale of these activities, the quality of OESW, like the rest of ESW, is important to the Bank, to its clients and to the wider donor community. A number of Executive Directors had questioned the value/effectiveness of the growing scale of OESW activities. In FY03, Management had decided that a pilot review would be carried out to assess the quality of FY02 OESW products. At the same time, QAG launched the assessment of Country AAA, initially with 3 pilots in FY02, then 6 pilots in FY03. 12 Country AAA Assessments are underway in FY04. While the Country AAA assessments have the country AAA programs over a CAS period as unit of account, they cover OESW tasks as part of the overall program. A number of OESW tasks have thus already been through the rapid assessment of sampled tasks under the Country AAA exercise.

9

Table 1. Growth of the Other ESW, FY01-FY03 Deliveries (#) Delivery Costs (m) Output Type FY01 FY02 FY03 FY01 FY02 FY03 Reports 222 238 397 39 41 67 Core Diagnostic 55 85 110 9 16 24 Country Advisory 142 93 154 26 15 22 Other Diagnostic 16 38 79 3 7 12 Regional Studies 9 22 37 - 3 8 Other ESW 103 183 273 11 18 21 Policy Notes 54 107 147 5 10 13 Other 49 76 126 6 8 8 Non-Lending TA** 137 169 283 16 26 38 AAA Total 462 590 953 66 85 126

** Non-lending TA, despite its significant and increasing scale is not covered in this exercise. The database for such activities requires considerable improvement, and the governance framework needs further definition. These issues are currently being addressed. A pilot assessment of non-lending TA will be considered when that is completed.

5. OESW, which doubled from FY01 to FY03 in dollar terms and almost tripled in numbers accounted for about a quarter of ESW expenditures in FY03. Policy notes accounted for about two thirds of the OESW costs in FY03. (Table 2) Table 2. Type of FY03 OESW Tasks Average Task # Cost $K Cost $K

Conferences and Workshops 66 4,840 73 Consultations and Country Dialogue 60 2,812 47 Policy Notes 147 13,032 89 Total 273 20,694 76

Pilot Assessment for FY02 OESW 6. The pilot “Other ESW’ assessment showed that 80 percent of such tasks were Satisfactory overall compared with the 94 percent Satisfactory for “regular” ESW in FY02.

• The pilot review concluded that all the “Other ESW” tasks reviewed were rated Satisfactory for Strategic Relevance, because they were integrally linked to country and regional strategies.

10

• On Internal Quality, the ratings were lower than found in the regular FY02 ESW sample. The results were similar to those of the regular ESW for the adequate coverage of appropriate areas and identification of critical issues as well as for tapping into the knowledge available in the Bank and elsewhere. However, weaknesses in this dimension were the relatively low scores for selectivity and prioritization of conclusions as well as quality of proposed follow-up actions.

• The strongly participatory nature of most of the “Other ESW” tasks contributed to the strength of these activities in the ratings for Dialogue and Dissemination.

• The results were more mixed for Likely Impact, largely due to the uneven attention by sector/country management at the implementation/dissemination stage.

• The assessment of Bank Inputs and Processes revealed substantial weaknesses resulting from the looser internal review processes for “Other ESW”. Management attention at entry and during implementation was considered to be Marginal in a number of cases and quite variable in intensity across different tasks. The two tasks rated Marginal overall also showed Less than Satisfactory management attention at entry and during implementation. Budgetary management of “Other ESW” showed need for improvement. Peer review processes, particularly at the implementation stage, were considered to be Marginal in many cases.

7. The Panel recognized that OESW is a very diverse group of tasks and that a number of these products had only recently been reclassified as ESW, some of which were originally designed against less precise requirements. The Panel concluded that

• the review procedures used for OESW were ad hoc and varied considerably across regions and type of task. The Panel recommended that there should be a minimum set of mandated procedures, which apply to all OESW tasks that cost over $100,000 (Bank budget and other funding) to ensure quality standards but allow flexibility to regional management to adjust them to the needs of the varied tasks.1 There should be for all larger OESW tasks a concept paper/note or an initiating memorandum, with the review processes similar to those used for regular ESW. In the case of tasks with multiple and diverse outputs, the concept note should also clearly indicate what type of integrative document (e.g. summary or synthesis note) will be prepared and what mechanism would be used to ensure consistency of analysis and recommendations across different outputs.

• the review processes followed during and before task completion should be flexibly managed by each region to allow variation to accommodate different tasks

1 In mid FY03, OPCS asked each Region to review its ESW guidelines with a view to including the Concept

Review date as a required field in SAP for both ESW reports and "other ESW" tasks (i.e., policy notes, conferences, workshops and consultations) costing more than $50,000.

11

• the focus on follow up activities and their budgeting needs to be considerably increased. This would increase the likely impact of OESW. At the completion of conference activities, the Panel recommended the use of appropriately designed participant feedback questionnaires (based on WBI experience). In conferences/workshops, where Ministers and senior officials are the main participants, simplified questionnaires or informal instruments may be more appropriate.

8. In terms of process, the pilot assessment concluded that tasks could be assessed for their quality in the same manner as regular ESW, albeit with some modifications to accommodate the varied nature of OESW tasks. These modifications were based on the experience gained during the pilot and input received from task teams. Specifically,

• the panels should be encouraged to weight the sub-questions differently to take account of the widely different nature of the tasks or score the sub-questions as not applicable as appropriate. Particularly in the case of conferences and workshops, where the quality of the participants and the level of their interaction are critical, the questionnaire has been adjusted to take account of these factors. Some questions were adjusted to enable greater attention to be paid by QAG panels to follow up activities and actions when the task, such as a workshop or conference, is completed.

• the dialogue and dissemination aspects should receive greater attention in cases where the OESW tasks were strongly participatory, with greater client- government involvement in task preparation and implementation and involvement of other donors and civil society more broadly.

• the assessment should take account of: (i) use of client funds to partly fund the task, (ii) review processes being used other than peer reviews; and (iii) regional management’s active involvement at task inception or during implementation, under Bank Inputs and Processes.

9. While the Panel was of the view that the limited sample covered was sufficiently robust to provide a basis for useful conclusions to be drawn, it was not possible to draw conclusions about the quality of smaller tasks since tasks costing less than $100,000 were not part of the sample. The Panel recommended that it would be desirable to evaluate also a sample of FY03 and FY04 tasks costing less than $100,000.

FY03-FY04 Assessment

10. The OESW in FY03 consisted of 273 tasks and a total cost of $21 million, which is almost a fourth of total FY03 ESW costs. The cost of individual tasks ranged from $1,000 to $625,000 with the median at $45,000. The average cost of an OESW task was $76,000 with significant variations across regions. Average costs ranged from $39,000 in Africa to $94,000 in SAR. This compares with the Bank-wide average cost of $168,000 for regular ESW tasks.

12

11. As of end-December 2003, 72 FY04 OESW tasks costing $3.4 million were recorded as deliveries in the system. An analysis of the posting of OESW tasks showed that the majority of them were posted in the system for the first time only in the last quarter of the fiscal year. For instance, of the 273 FY03 OESW tasks, 171 were posted in the fourth quarter for the first time. It is therefore expected that the universe from which the FY04 tasks to be sampled will expand through the fiscal year. The pipeline data, however, shows 397 tasks with an estimated total cost of $34 million.

12. Coverage. The formal trigger for entry into the sample is the standard SAP milestone, delivered to the client. The assessment will be based on a stratified random sample of OESW tasks "completed" in FY03 and FY04 in two batches. The first batch will cover all FY03 tasks and FY04 tasks completed in the first half of the fiscal year, and the second batch will cover the FY04 tasks completed in the second half of the fiscal year. The sampling universe will exclude tasks under $20,000 in total cost. The sample will be stratified by cost and type of output. The sampling percentages by cost strata for the FY03 and FY04 tasks will be:

Table 3: FY03-FY04 Sample 2

13. If the actual outcome of the FY04 program is assumed to be roughly the same as FY03 in terms of the number of tasks and size distribution, by the time the exercise is completed, about 100 tasks delivered in FY03 and FY04 will have gone through the assessment. This sample size should permit a robust assessment of how well OESW dollars are being spent Bank-wide (with less than 5% sampling margin of error at 95% confidence level). The sampling process will ensure that at least 10 tasks from each region are assessed, even if it may mean over-sampling for some regions.

14. Tasks managed by Task Team Leaders (TTL) who have been subject to a QAG evaluation in the previous 12 months (i.e. QEA6 or C-AAA) will also be excluded and replaced by other randomly selected tasks.

15. Quality Dimensions . The assessment methodology for OESW will be broadly similar to that employed in the regular QAG assessments of ESW. Rating the Overall Quality of each task will be guided by the same broad criteria used in assessing ESW covering four quality dimensions i.e., (i) Scope and Strategic Relevance; (ii) Internal Quality; (iii) Dialogue and Dissemination; and (iv) Likely Impact. Bank Processes and Inputs will also be assessed.

2 FY04 figures were estimated.

Task # of

Tasks*Sampling

%Sample

Size# of

TasksSampling

%Sample

Size# of

TasksSampling

%Sample

Size# of

TasksSampling

%Sample

Size>200K 28 40% 11 12 60% 7 4 60% 2 44 45% 20100-199 66 20% 13 14 40% 5 4 40% 2 84 24% 2050-99 70 15% 10 36 25% 9 26 25% 6 132 19% 2520-49 70 15% 11 29 25% 7 71 25% 18 170 21% 36

TOTAL 234 45 91 28 105 28 430 23% 101

Consultations TotalPolicy Notes Conferences

13

16. Ratings. The relevance and relative importance of the above dimens ions varies with the type of OESW analyzed and the country context. Based on their understanding of the country context and the task-specific goals, panelists would exercise judgment in deciding on the relative weights to be given, first to the individual sub-ratings, then among the four assessment dimensions. Such a balanced set of judgments is a central responsibility of panelists. The questionnaire has been adjusted and some questions added to allow panelists to take account of the varied outputs covered by OESW. Policy notes are likely to be closest for evaluation purposes to regular ESW tasks but panelists would take particular care to adjust their weightings of sub-questions for tasks such as conferences, workshops and country dialogue. Where questions do not apply for particular outputs, panelists are encouraged to use “not applicable”. As indicated above in para. 8, conferences and workshops are likely to pose a particular challenge and the questionnaire has been adjusted to take account of these variations. In some cases, the task being evaluated may not meet the core criteria to be classified as ESW. Panelists would treat such cases as a miss-classification and exclude the task from the OESW sample. The role of the QAG moderator will be particularly important to ensure consistent criteria and standards across OESW tasks.

17. Quality will be rated on a six-point scale similar to the scale used by OED in its evaluations. This departure from QAG’s four-point scale is to see whether the majority of the tasks rated Satisfactory under the four-point scale could be better differentiated. Panelists are instructed to use the full range afforded by the scale in accordance with the standards described below. Within each question, panelists are encouraged to use the comments section for clarifying the basis of their ratings.

Highly Satisfactory (1) Best practice in most respects and no significant

deficiencies Satisfactory (2) Satisfactory or better on all aspects.

Moderately Satisfactory (3) Satisfactory on all key aspects but significant missed opportunities

Moderately Unsatisfactory (4) Significant deficiencies in a few key aspects

Unsatisfactory (5) Significant deficiencies in several key aspects

Highly Unsatisfactory (6) A broad pattern of deficiencies 18. Assessment Process. A two-prong approach will be followed for the OESW assessment. A panel of typically two members will assess tasks costing above $50,000. The panels, selected from experienced Bank staff or consultants will review each sampled task, using the guidance questionnaire as the framework. Their work will start with a review of the task file, a set of background documents assembled by QAG and the TTL. The task documentation will be expected to cover key documents from concept paper through to final report, dissemination activities and follow up, including comments by peer reviewers, minutes of decision/review meetings and summaries of discussions with the client. The task team will also be requested to provide a short contextual note. Based upon this background material review, the panel will prepare a list of issues for

14

discussion during a meeting with the Task Team. The questions will be made available to the task team at least two working days in advance. Managers may participate in this discussion if they so desire. Unless agreed otherwise with the Task Team, the panel interviews will only be set up four months after the delivery date. The panels may also contact appropriate counterparts in the client countries in order to strengthen the panel’s judgment on dialogue aspects and Likely Impact. (All panel contacts with clients would be arranged in close consultation with the responsible task team and country director/country team.) 19. Following the interview and related discussions, the panels will prepare consensus ratings and a narrative statement. These will be shared in draft with the region. Where panelists intend to give a lower than satisfactory rating to managerial performance, they will be expected to consult the managers concerned. The Region will be given five days to comment. The panel will also be available to discuss this draft report with the TTL and/or their managers. After the panel finalizes its report, incorporating comments from the TTL, the report will be transmitted to the TTL and the regional and sector anchor managers by QAG. The TTLs or their managers may request a second stage review for those tasks rated less than Moderately Satisfactory. In this case, QAG will convene a new panel, which will also hold consultations with sector/country managers and peer/QER reviewers. The draft report of the second stage panel will also be provided to the regional staff for their comments before being finalized. In cases of disagreement, the Region will be given the opportunity to attach a one-page rejoinder to the final assessment report, focusing on areas of divergent views (and reasons why) as well as on any proposed follow-up. 20. Tasks costing between $20,000 and $50,000 will go through an initial desk ‘triage’ by a panel of two selected from a pool of specialists. The $20,000 cut off was selected because of cost effectiveness concerns. These tasks will be also rated on four quality dimensions and the Bank Processes and Inputs by using a simplified questionnaire. Only those tasks, which are rated Marginally Satisfactory or lower, will go through the assessment process described in paras 18-19 above. For those tasks which are rated Moderately Satisfactory or better, the TTL and regional managers will be provided the ratings of the panel and only a brief summary narrative. Based on this initial review the panels will determine whether they have an adequate understanding for rating the tasks based on a desk review alone or whether they need a brief discussion with the task manager or in some cases even a feedback from selected stakeholders. The extent of interaction needed in these smaller tasks is expected to be much less than the larger OESW tasks costing more than $50,000. 21. The unit of assessment remains the OESW task with its own SAP ID. But the assessment is being conducted in the light of broader programmatic and country considerations. Individual OESW tasks may not be stand-alone pieces; they may be sub-tasks in support of a larger exercise such as a PRSP or a diagnostic report or else is one of a series of small tasks delivered over a period of time. Also, the approach to and depth of analysis may reasonably be expected to differ with country context/circumstances, across different types (sectors, themes) of OESW, and with different task budgets. For these

15

reasons, it is important that panelists understand, from CAS/PRSP or other sources, the larger Bank programming framework and relevant country context. As part of the preparation for the assessment process, QAG will ask task teams to respond to a short list of contextual questions on the key objectives of the task, the broader programmatic context, including related Bank activities/reports, i.e. previous ESW, as well as special country circumstances and conditions that may have a significant bearing on the quality and management of the task. Task teams will also be asked to help characterize the nature of client participation in the task. For their part, panelists are requested to summarize their views of the likelihood of achieving task objectives as well as comment on salient country contextual and programmatic factors in the introductory section of the questionnaire. 22. The Panel Composition. The panels, and its members, lie at the heart of the QAG assessment approach. Experienced, credible, senior professionals, with relevant technical and/or geographic expertise, will carry out assessments. Panelists would normally be active Bank staff level GH or above, retired Bank staff or consultants of comparable experience. As was done in the case of previous ESW and QEA Assessments, QAG will try to use external panelists from the academia, NGO and donor community. QAG also seeks the active support of the Network and Regions in identifying and selecting panelists. To avoid any potential conflict of interest, an individual who has been involved in a particular OESW task, as peer reviewer, manager, or otherwise, or who is currently a staff member of the same Region, may not be selected to serve as a panelist on that task. Panelists are provided guidance and orientation as part of the start-up process prior to panel duty. Equally, QAG provides a moderator for each assessment to ensure consistency of approach, including firmness of ratings, among panels, across sectors and Regions. With a view to support staff development and disseminate QAG methodology in the Regions and Networks, they may propose an observer (a relatively less experienced staff member) to attend a panel session. 23. Reports. QAG issues two types of reports: Task-specific assessments, and a Synthesis Report. Assessment Reports are distributed to the TTL, plus the relevant Country Director and Sector Manager as well as the Chief Economist, Regional Quality Director and the relevant Sector Board Heads. 24. Once the task assessments are completed, a Synthesis Report for OESW, in a power-point format, will be prepared to provide an overview/analysis of assessment findings, examine systemic issues, highlights patterns of strengths and weaknesses of OESW, and makes recommendations for further improvements in quality. The overall OESW Synthesis report in draft form will be reviewed with participants and other interested observers of the QAG process, then with senior Regional/Network staff involved with quality issues. It will then be sent to Regional VPs, Network VPs for review and to the Managing Director. Major findings and the conclusions of the OESW will be included in the ARPP.

16

25. Dissemination Channels and Learning. For the purpose of promoting staff learning through sharing of findings, including best practices, QAG plans to increase activities devoted to dissemination and learning through a variety of channels, including:

• publishing best practice examples on the QAG website.

• providing summaries of lessons and conclusions on the QAG website.

• publicizing best practice examples through interaction with Regions and Networks and Sector Boards.

• working in partnership with WBI and Regions/Networks, participating in targeted staff training workshops, drawing on findings accumulated over the several years of ESW assessments.

Cost of the Assessment 26. Panels for each assessment of an OESW task are estimated to require an average of 5 staff-days for tasks costing above $50,000 and an average of 1.5 days for tasks costing below $50,000. Additional costs are represented by QAG overhead in management, moderating and logistics. These costs will total $600,000 over two fiscal years. Under the Bank guidelines for cross-support, QAG would reimburse staff for time billed when they work as panelists or on special tasks. The time spent by Regional staff in preparing for and interacting with assessment panels is estimated at 2 days per task and will total $160,000 over two fiscal years. This cost will be borne by the Regions. The estimated direct costs for the assessment of the OESW program as outlined in this document represents about 1.8 percent of the total estimated cost of the FY03 and FY04 OESW program.

Schedule of Key Activities Approach Paper Discussion with Friends of QAG Jan 28 Virtual Discussion with Chief Economists and Quality Directors by Feb 18 Final Feb 27 Notify task managers early March

(First Batch) Panel Meetings for >50K April-May Triage desk assessment April Detailed Assessments <50K May Notify task managers (Second Batch-FY04 tasks) July Triage desk assessment September-November Detailed Assessments <50K October Panel meetings for >50K September-November Synthesis report, completed by end-December

17

_____________________________ Attachment 1. Quality of Other ESW Assessment FY03-FY04 (Guidance Questionnaire for tasks costing more than $50,000) Attachment 2. Quality of Other ESW Assessment FY03-FY04 (Guidance Questionnaire for tasks costing less than $50,000)

Annex 4

Assessment Rating1= Highly satisfactory2= Satisfactory3= Moderately Satisfactory4= Moderately Unsatisfactory5= Unsatisfactory6= UnsatisfactoryNA= Not applicable

ESW FY02 OESW

A. OVERALL QUALITY 92 96

1. Strategic Relevance & Timeliness 96 100

2. Internal Quality 88 93

3. Dialogue and Dissemination 88 91

4. Likely Impact 86 95

B. BANK PROCESSES 83 81

`

Note 1 All data in this annex are number-weighted.2 The ratings in this annex represent the percentage of tasks rated "Satisfactory or Better" in ESW2 and "Moderately Satisfactory or

Better" in OESW.

SUMMARY ASSESSMENT RESULTSQUALITY OF ECONOMIC AND SECTOR WORK

The overall assessment of the quality of ESW is not meant to be a simple average of the assessment for the four aspects of quality listed (which, in turn, are not simple averages of respective underlying factors). The panel should use its judgment in weighing the relative importance of each component and factor in any particular case.

19


ESW FY02 OESW

A. OVERALL QUALITY

1. STRATEGIC RELEVANCE AND TIMELINESS 92 96

1.1 Task objectives 92 87

1.2 Relevance of the task objectives to the Bank's assistance strategy for the country 96 100

(a) towards initiating/moving the policy dialogue forward 97 100

(b) as an underpinning to strategy development 94 100 (next CAS/PRSP)

(c) as an underpinning to lending operations 96 100

(d) as an instrument for capacity-building, knowledge-sharing 92 100

1.3 Consistency of the task with the Bank's sector/thematic strategy 99 100and corporate objectives linked to the MDGs

1.4 Relevance of the task objectives to the client’s programs, and NR 100strategic objectives and priorities ?

1.5 Timeliness of the task in terms of client capacity and Bank’s 98 100 programming agenda ?

General Guidance: In assessing this section, the panelists should look at the Concept Paper or its equivalent, as well as the larger context as reviewed in the CAS, etc. In some cases objectives are consciously changed midway to reflect new realities. The task should then be assessed with reference to the objectives as they were formally redefined.

Guidance: The idea is not to penalize innovation or exploration of new directions. But does the case for undertaking this task also square with the broader corporate vision and poverty alleviation agenda of the Bank?

Guidance: Not all listed elements may be relevant to each task; score NA where appropriate. The most important reference point is, of course, the CAS under implementation at the time the task was initiated. The panelists should look for a specific justification under CAS objectives - which themselves may be defined very broadly. Apart from CAS, panelists should also be sensitive to explicit guidance of the Country Director..

QAG ASSESSMENT GUIDANCE QUESTIONNAIRE

20


ESW FY02 OESW

2. INTERNAL QUALITY 88 93

2.1 Appropriateness of coverage/scope of the task, in view 87 96of the objectives (focus, range of issues addressed,areas covered)

2.2 Clear identification of critical issues 85 96

2.3 Quality of the analysis 94 99

2.4 Quality of the empirical evidence 97 98

2.5 Appropriate consideration of/sensitivity to the poverty 77 91dimension. For example, discussion of (I) how current policies/program affect the poor? (ii) how the proposals made by thetask will affect the poor?

2.6 Appropriate consideration of/sensitivity to the gender dimension 68 93

2.7 Quality of conclusions and recommendations 74 84

Guidance: Look for analytical/methodological rigor expected of similar Bank products. (If specific “product” guidelines exist, were they respected?). Check if the weight given to analysis of individual topics is consistent with their relative importance to the objectives of the task. Give extra credit for state of the art, innovative work. Keep the objectives and size (cost) of the task, specific country context and previously available knowledge in mind.

General Guidance: While this section flows most “naturally” towards assessment of written products, the spirit behind the questions should also be applied to the internal quality and analytical content within dialogue/processes.

Guidance: While each task is expected to reach carefully analyzed findings or conclusions, the nature and extent of recommendations will vary with the task, its objectives and the state of the dialogue with the country. For example, fiduciary products are expected to have clear conclusions as to risk assessment. Good recommendations should be realistic/appropriately specific as against bland and general; they should be operational/actionable, with clear follow-up actions for Bank/Clients. In the event that the Bank chooses not to include recommendations in the body of the report, as a didactic style or for reasons of country sensitivities, did the Bank consider alternatives such as posing and weighing options and alternative? Likewise, when recommendations are not included in the report, on the basis of other documents or panel discussions with the team or managers, can the panel discern a Bank view on the issues being addressed?

Guidance: The extent to which the team sought and assimilated information from various sources. Give due allowance for availability of information in the country, and check if the team made sufficient efforts to get to the right information. Keep the size of the task in mind.

Guidance: Were the different needs of males and females considered? [e.g. Was the data used disaggregated by sex? Were the barriers to inclusion/access discussed?) Did the report attempt to identify gender issues that might affect implementation of the policy advice or recommendation? Did the report consider the different roles played by men and women in making recommendations? Did the report consider the different impact of the recommendations on men and women

21


ESW FY02 OESW3 DIALOGUE AND DISSEMINATION 88 91

3.1 Attention devoted to engaging the client and key stakeholders 85 91

3.2 Effectiveness of Client participation in: 78 91

3.3 Quality of Bank contribution/support for client-led tasks: 100 100

3.4 Appropriate consultation with other key stakeholders in the 83 93country (e.g. legislators, local government officials, civil society groups, local academics, private sector, etc. groups, local academics, private sector, etc.)?

3.5 Quality of partnership arrangements with other donors 90 95

3.6 Appropriateness of the actual/planned arrangements 85 82for dissemination

Guidance: This question seeks to assess the quality of Bank support to the Client in preparing those components of the task for which the Client has had the lead or a major, substantive role. Score NA if the task is essentially Bank-led.


General Guidance: This set of questions seeks to inquire how and to what degree the country clients, and other stakeholders such as donors, were active participants in the performance of the task under review. Effective participation and the opportunities for dialogue associated with it are recognized as hallmarks of ownership, and an important indicator of the prospects for the achievement of task objectives. Not all questions in this section may be equally relevant to any single task. Accordingly, in cases where, because of the purpose and nature of the task, selected questions are not applicable, score NA.


ESW FY02 OESW4 LIKELY IMPACT 86 95

4.1 Efforts made towards maximizing impact 83 83

4.2 Likely Impact on the Client 83 85

4.3 Likely impact on the Bank 89 91

4.4 Likely impact on the wider development community: 97 97

General Guidance: Based on the information available, including interviews with country counterparts, as appropriate, panels are asked to make judgments about expected impact of the task in relation to the development objectives of the task. The panel is not assessing actual future impact, but rather the probability of positive outcomes. Likely Impact is expected to depend on how relevant/timely the task is, how well it has been done and how receptive is the client. Panelists should seek a sense of the value-added of the task (e.g. What if the task had not been done?). The ratings should factor out substantial and specific changes in the country situation beyond the team’s control. The panelists should score N/A against those issues that are not related to explicit objectives of the task. They should also be sensitive to the size and scope of the task as planned in the Concept Paper.

Guidance: Does the concept paper identify clearly what are the measures of impact or results for the intended audiences?

Guidance: This is not a ‘necessary’ requirement of high impact. Score NA where objectives of the task did not anticipate spin-offs to the wider development community.


23


ESW FY02 OESW

5 BANK PROCESSES 83 81

INPUTS

5.1 Preparation/Processing Cost ($000)Regular Budget (BB)............................................................. 205 115

Trust Funds........................................................................... 30 31

Total Funding 235 149

5.2 Elapsed time (months)Activity initiation – Discussion Draft .................................... 13 8

Discussion Draft - Client Delivery......................................... 3 4

MANAGERIAL ATTENTION

5.3 a Adequacy of

a) resources NR 91

b) time to complete the task? 73 84

5.4 How effectively were the resources used? 91 93

5.5 Skill-mix of the task team? How comprehensive? Cross-sectoral skills? 88 96

5.6 a) Attention to Quality at Entryi) by Sector Management 83 73

ii) by Country Management 88 75

b) Attention to quality during implementation

i) by Sector Management 82 88

ii) by Country Management 83 84

Guidance: Did the managers select the right leadership? Did the manager or TTL assemble the right team in terms of country knowledge, experience, skills?


Guidance: Extent of shared understanding between managers and the team on the objectives, methodology, key milestones for progress and indicators of success, etc; an appropriate understanding on roles of core participants at the time of launching the task - through a Concept Paper or otherwise. Sector management here means the individual with a direct supervisory/advisory relationship with TTL in the performance of the task.

Guidance: Monitoring of quality/milestones, promotion of cross-fertilization, best practices, etc. Did the management provide appropriate guidance and mentoring? Were adjustments made and mid-course corrections taken in a timely manner? Were comments on drafts appropriately considered? Did management support and/or participate in the dissemination and dialogue? Typically such attention is at key points rather than continuous and is provided in relation to the size, complexity of the task and the experience, skill mix of the task team.

24

ESW FY02 OESW

OTHER

5.7 Peer Review Process 77 79

5.8 For conferences, quality of arrangements for client/participant feedback ? NR 97

5.9 Appropriateness of this task’s classification as OESW NR 85

25

Annex 5 Methodology for Sampling, Stratification and Weighting

The universe for the other ESW exercise consisted of all tasks classified as Policy Notes, Consultations/Country Dialogue, and Conferences/Workshops which were recorded in SAP as delivered to client in either FY03 or FY04. This universe includes tasks which were recorded as delivered to client at the time of sampling, but were later reclassified out of the universe (eithe r due to a change in product type, product cost, or delivery date.) This yielded a universe of 438 tasks, which were stratified by cost and product type for sampling purposes. The distribution of the universe was:

Pol Notes Conf. Consult. All Pol Notes Conf. Consult. All200K+ 37 13 9 59 200K+ 10,973 5,659 2,554 19,186100-199K 64 13 22 99 100-199K 9,139 1,793 3,086 14,01850-99K 82 31 24 137 50-99K 5,998 2,223 1,694 9,91520-49K 71 32 40 143 20-49K 2,412 1,136 1,336 4,8840-19K 64 24 36 124 0-19K 466 105 344 915Total 318 113 131 562 Total 28,988 10,916 9,014 48,918

# Delivered $ Delivered

Different sampling fractions were given to each cell in the matrix. Conferences and Consultations were sampled at a higher rate than policy notes since their universes were smaller in order to ensure robust results. Additionally, high cost tasks were sampled at a higher rate in order to ensure a representative sample by total cost. Tasks costing under $20,000 were excluded from the sampling due to the relatively low value added of assessing such tasks. The sampling fractions for each strata and resulting samples were:

Pol Notes Conf. Consult. Pol Notes Conf. Consult. All200K+ 40 60 60 200K+ 15 7 5 27100-199K 20 40 40 100-199K 13 5 9 2750-99K 15 25 25 50-99K 12 8 6 2620-49K 15 25 25 20-49K 11 9 10 300-19K 0 0 0 0-19K 0 0 0 0

Total 51 29 30 110

Sampling Percentages # Sampled

The resulting sample covers approximately 20% of the universe by number and 33% by cost. This stratification was intended to provide results which are statistically robust Bank-wide at 95% +/- 5%. Of the 110 tasks sampled from this universe, 57 tasks were not rated in the end. Based on this number, approximately 52% of the 562 tasks in the universe (291 tasks) do not belong in the universe. This yields a final sampling of approximately 53 tasks out of 271. This gives the following final confidence intervals on the final results:

26

Original Adjusted Original Final Original Adjusted# in Adjusted # in # Tasks Confidence Confidence

Universe Universe Sample Rated Inverval InvervalAFR 135 41 23 7 90% +/- 10 90% +/- 18EAP 82 39 19 9 90% +/- 9 90% +/- 13ECA 107 57 17 9 95% +/- 5 95% +/- 5LCR 46 28 10 6 90% +/- 19 90% +/- 25MNA 73 51 13 9 90% +/- 10 90% +/- 11OTH 38 16 7 3 NA NASAR 81 39 21 10 90% +/- 9 90% +/- 13Total 562 271 110 53 95% +/- 5 95% +/- 7

Distribution by Region

Original Adjusted Original Final Original Adjusted# in # in # in # Tasks Confidence Confidence

Universe Universe Sample Rated Inverval InvervalESSD 75 23 16 5 90% +/- 9 90% +/- 15FSE 57 33 7 4 95% +/- 5 95% +/- 5HDN 106 62 24 14 95% +/- 5 95% +/- 5INF 104 78 16 12 90% +/- 12 90% +/- 14PREM 168 79 32 15 90% +/- 8 90% +/- 12OTH 11 2 6 1 NA NAPSDN 41 9 9 2 90% +/- 19 90% +/- 42Total 562 271 110 53 95% +/- 5 95% +/- 7

Distribution by Network

Results for this exercise are presented on a weighted basis to adjust for over/under-representation in the sample because of stratification and sample design. The weights are computed as a ratio of the total tasks in a strata divided by the number of tasks sampled from that strata. This ensures that each task sampled is weighted as a proportion of the universe it was taken from. The weights for each cohort are:

Pol Notes Conf. Consult.200K+ 2.467 1.857 1.800100-199K 4.923 2.600 2.44450-99K 6.833 3.875 4.00020-49K 6.455 3.556 4.000

Weights

Annex 6 Statistical Tables

Number1 Of Which

2 Of Which

Region Of Tasks not Rated Rated OA R1 R2 R3 R4 R5AFR 23 16 7 7 7 6 6 7 5EAP 19 10 9 9 9 9 9 8 8ECA 17 8 9 7 8 7 6 7 6LCR 10 4 6 6 6 6 6 6 6MNA 13 4 9 9 9 8 8 9 7OTH 7 4 3 3 3 3 2 3 2SAR 21 11 10 9 10 9 10 10 8Bankwide 110

3 57 4 53 50 52 48 47 50 42

OTHER ECONOMIC AND SECTOR WORKOverall Assessment By Region

........Rating # Moderately Satisfactory or Better........

Number 1 Of Which 2 Of WhichRegion Of Tasks not Rated Rated OA R1 R2 R3 R4 R5AFR 23 16 7 14 39 25 21 0 0EAP 19 10 9 0 44 0 29 0 33ECA 17 8 9 5 13 13 13 0 7LCR 10 4 6 55 100 31 55 36 55MNA 13 4 9 34 49 16 28 18 0OTH 7 4 3 24 100 76 0 0 0SAR 21 11 10 10 47 13 24 15 10Bankwide 110

3 57 4 53 18 47 17 26 10 14


3 57 4 53 17 47 17 25 10 14

OA = OVERALL RATINGR1 = Strategic Relevance and TimelinessR2 = Internal QualityR3 = Dialogue and Dissemination R4 = Likely Impact R5 = Bank Inputs and Processes

Note 1 Tasks that have not been rated because they are not ESW (or OESW).2 Tasks appropriately coded as OESW as well as tasks that should have been coded under another ESW product code.3 of which 21 Tasks should have been coded as NLTA, 23 Tasks as internal or external relations types of products , 9 Tasks that were only created for the purpose of disseminating an earlier ESW Task and 4 Tasks that are not yet delivered to the client

Overall Assessment By Region (Weighted By $ Amount)

........Rating % Highly Satisfactory........

4 of which 10 Tasks should have been coded under another ESW product code. ( Type of Output : 8 in Policy Note, 1 in Conference/Workshop, 1 in Consultations/Country Dialog, Region : 1 in AFR, 3 in ECA, 2 in LCR, 3 in MNA, 1 in OTH, Network : 2 in ESSD, 2 in FSE, 2 in HDN, 2 in INF, 2 in PREM, Cost Category : 1 in C. 50 - 99, 4 in E. 100 - 199, 5 in F. >200k )

OTHER ECONOMIC AND SECTOR WORKOverall Assessment By Region (Weighted By Number)


OTHER ECONOMIC AND SECTOR WORK

28


3 57 4 53 55 43 57 46 57 36


3 57 4 53 56 43 57 47 58 35


Note 1 Tasks that have not been rated because they are not ESW (or OESW).2 Tasks appropriately coded as OESW as well as tasks that should have been coded under another ESW product code.3 of which 21 Tasks should have been coded as NLTA, 23 Tasks as internal or external relations types of products, 9 Tasks that were only created for the purpose of disseminating an earlier ESW Task and 4 Tasks that are not yet delivered to the client


........Rating % Satisfactory........

4 of which 10 Tasks should have been coded under another ESW product code. (Type of Output: 8 in Policy Note, 1 in Conference/Workshop, 1 in Consultations/Country Dialog, Region: 1 in AFR, 3 in ECA, 2 in LCR, 3 in MNA, 1 in OTH, Network: 2 in ESSD, 2 in FSE, 2 in HDN, 2 in INF, 2 in PREM, Cost Category: 1 in C. 50 - 99, 4 in E. 100 - 199, 5 in F. >200k )




29


3 57 4 53 23 10 19 19 27 31


3 57 4 53 22 10 19 19 27 33




........Rating % Moderately Satisfactory........

4 of which 10 Tasks should have been coded under another ESW product code. (Type of Output: 8 in Policy Note, 1 in Conference/Workshop, 1 in Consultations/Country Dialog, Region: 1 in AFR, 3 in ECA, 2 in LCR, 3 in MNA, 1 in OTH, Network: 2 in ESSD, 2 in FSE, 2 in HDN, 2 in INF, 2 in PREM, Cost Category: 1 in C. 50 - 99, 4 in E. 100 - 199, 5 in F. >200k)




30

Number1 Of Which

2 Of Which


3 57 4 53 96 100 93 91 95 81

Number1 Of Which

2 Of Which


3 57 4 53 95 100 93 91 95 82


Note 1 Tasks that have not been rated because they are not ESW (or OESW).2 Tasks appropriately coded as OESW as well as tasks that should have been coded under another ESW product code.3 of which 21 Tasks should have been coded as NLTA, 23 Tasks as internal or external relations types of products, 9 Tasks that were only created for the purpose of disseminating an earlier ESW Task and 4 Tasks that are not yet delivered to the client


........Rating % Moderately Satisfactory or Better........

4 of which 10 Tasks should have been coded under another ESW product code. (Type of Output: 8 in Policy Note, 1 in Conference/Workshop, 1 in Consultations/Country Dialog, Region: 1 in AFR, 3 in ECA, 2 in LCR, 3 in MNA, 1 in OTH, Network: 2 in ESSD, 2 in FSE, 2 in HDN, 2 in INF, 2 in PREM, Cost Category: 1 in C. 50 - 99, 4 in E. 100 - 199, 5 in F. >200k)




31

Number 1 Of Which 2 Of WhichNetwork Of Tasks not Rated Rated OA R1 R2 R3 R4 R5ESSD 16 11 5 0 78 31 0 0 0FSE 7 3 4 0 0 0 36 0 0HDN 24 10 14 30 50 26 18 5 0INF 16 4 12 10 42 19 10 10 14OPCS 3 3 0 0 0 0 0 0 0PREM 32 17 15 30 47 12 42 28 35PSDN 9 7 2 0 50 0 100 0 50Other 3 2 1 0 100 0 0 0 0Bankwide 110 3 57 4 53 18 47 17 26 10 14



Note 1 Tasks that have not been rated because they are not ESW (or OESW).2 Tasks appropriately coded as OESW as well as tasks that should have been coded under another ESW product code.3 of which 21 Tasks should have been coded as NLTA, 23 Tasks as internal or external relations types of products , 9 Tasks that were

only created for the purpose of disseminating an earlier ESW Task and 4 Tasks that are not yet delivered to the client

Overall Assessment By Network (Weighted By $ Amount)



OTHER ECONOMIC AND SECTOR WORKOverall Assessment By Network (Weighted By Number)



32












33












34

Number 1 Of Which 2 Of WhichNetwork Of Tasks not Rated Rated OA R1 R2 R3 R4 R5ESSD 16 11 5 100 100 100 100 83 100FSE 7 3 4 100 100 100 100 100 74HDN 24 10 14 84 100 80 76 87 71INF 16 4 12 100 100 100 96 100 77OPCS 3 3 0 NA NA NA 0 0 0PREM 32 17 15 100 100 92 91 100 83PSDN 9 7 2 100 100 100 100 100 100Other 3 2 1 100 100 100 100 100 100Bankwide 110 3 57 4 53 96 100 93 91 95 81

Number 1 Of Which 2 Of WhichNetwork Of Tasks not Rated Rated OA R1 R2 R3 R4 R5ESSD 16 11 5 100 100 100 100 83 100FSE 7 3 4 100 100 100 100 100 76HDN 24 10 14 82 100 79 76 87 71INF 16 4 12 100 100 100 97 100 78OPCS 3 3 0 NA 0 0 0 0 0PREM 32 17 15 100 100 93 91 100 84PSDN 9 7 2 100 100 100 100 100 100Other 3 2 1 100 100 100 100 100 100Bankwide 110 3 57 4 53 95 100 93 91 95 82










35

Number 1 Of Which 2 Of WhichOutput Type Of Tasks not Rated Rated OA R1 R2 R3 R4 R5Conference/Workshop 29 20 9 100 100 100 85 85 85Consultations/Country dialogue30 21 9 100 100 77 100 100 68Policy Note 51 16 35 95 100 95 90 96 82Bankwide 110

3 57 4 53 96 100 93 91 95 81

Number 1 Of Which 2 Of WhichOutput Type Of Tasks not Rated Rated OA R1 R2 R3 R4 R5Conference/Workshop 29 20 9 100 100 100 88 84 88Consultations/Country dialogue30 21 9 100 100 78 100 100 68Policy Note 51 16 35 94 100 94 90 96 83Bankwide 110

3 57 4 53 95 100 93 91 95 82



Overall Assessment By Output Type (Weighted By $ Amount)


4 of which 10 Tasks should have been coded under another ESW product code. (Type of Output : 8 in Policy Note, 1 in Conference/Workshop, 1 in Consultations/Country Dialog, Region: 1 in AFR, 3 in ECA, 2 in LCR, 3 in MNA, 1 in OTH, Network: 2 in ESSD, 2 in FSE, 2 in HDN, 2 in INF, 2 in PREM, Cost Category: 1 in C. 50 - 99, 4 in E. 100 - 199, 5 in F. >200k)

OTHER ECONOMIC AND SECTOR WORKOverall Assessment By Output Type (Weighted By Number)



Documents

Quality Assurance Group - documents.worldbank.orgdocuments.worldbank.org/curated/en/180811468313530171/pdf/445710WP0BOX...SAP user friendliness, insufficient guidance/support to TTLs)