Slide 1

Monitoring and Evaluation of Private Sector Development

Hiroyuki HatashimaPrivate Sector Evaluation, Independent Evaluation Group (IEG)World Bank Group

Canadian Evaluation Society ConferenceJune 11, 20131IEG is part of a larger array of evaluation and control mechanisms2World Bank Group President / ManagementManagements Internal Accountability FrameworkAnd SystemsSelf EvaluationMonitoring (QAG, DOTS, AS)Institutional IntegrityInternal AuditCompliance Advisor/ Ombudsman (IFC, MIGA)Independent EvaluationGroupInspection PanelBoard ofExecutive Directors

2IEG seeks to address both accountability and learning Accountability for results:Is the Bank Group doing what it says? Is it achieving its objectives? To what extent is it contributing to development outcomes?With what results and development impact? Learning:What projects and programs achieve results relevance, efficacy, and efficiency and why?How can M&E be strengthened in the Bank Group and in client countries? 3

IEG Private Sector Evaluation (IEGPE)*Evaluates the effectiveness of programs, projects, and activities and the strategies, policies and procedures that relate to them with particular attention to the achievement of objectives for private sector development Generates lessons for learningSupports the Board in ensuring WBGs development effectiveness and accountability*Adapted from Directors terms of reference approved by the Board (November 2005)

3Three main messages Evaluation Across BoundariesPrivate Sector operations have specific data gathering advantage/disadvantage based on their business modelM&E systems should adopt to the business practices to be effective and efficientM&E can influence learning, quality of work and outcome

Outline of PresentationContext of Private Sector Development AssistanceTypes of World Bank Group Private Sector Development interventionsTechnical Assistance projects M&EPrivate Sector Investment projects M&EUse and Influence of M&E

51.1 Context: Private Sector As Important PlayerThe private sector has taken on an essential role in contributing a wide range of development issues.Potential of private sector to contribute to development solution is still underutilized.M&E essential for track progress, assess outcome & performance and generating knowledge of what works, what does not, and why.

6Are the monitoring and evaluation systems of IFC and MIGA equipped to inform the organizations on their performance and results?1.2 Assessing M&E systems Evaluation Questions

7Outline of PresentationContext of Private Sector Development AssistanceTypes of World Bank Group Private Sector Development interventionsTechnical Assistance projects M&EPrivate sector Investment projects M&EUse and Influence of M&E

8The World Bank Group consists of five closely related institutions:IBRD: The International Bank for Reconstruction and DevelopmentIDA: The International Development AssociationIFC: International Finance CorporationMIGA: The Multilateral Investment Guarantee AgencyICSID: The International Center for the Settlement of Investment Disputes

1. Types of WBG Private Sector Development interventions1. Interventions through the Public Sector to influence the Private SectorWorld Band Financial and Private Sector Development projects, IFCs Advisory Services projects to government (Investment Climate, PPP)2. Direct interventions in partnership with the private sector and public entities operating commerciallyIFCs investments and MIGAs guarantees3. Blend interventionsIFCs Advisory Services to private companies, but with a strong public good component

Outline of PresentationContext of Private Sector Development AssistanceTypes of World Bank Group Private Sector Development interventionsTechnical Assistance projects M&EPrivate Sector Investment projects M&EBusiness Models and M&E

11123.1. Technical Assistances follow logical model results measurementImpactOutcomesOutputsActivities InputsResources used for advisory services projects

Programs associated with an advisory project

Example:Provide advisory services to local companiesImmediate deliverables of the project

Example:Diagnostic report and recommendations made10 managers were trainedLegislation drafted

Changes in knowledge, behaviors, and attitudes as a result of intervention with attribution of changes to the projectExample:Legislative or inst reformsChanges in agricultural techniques, or CG practices Consequences resulting from an intervention including improvements in clients performance and effects beyond direct beneficiaries

Example:increased investment, sales, jobs

Track indicatorsProject Supervision Report (PSR)SuperviseApproveDisburseCompletionCommitM&E systemLogframe,Identify indicators ProjectCompletion Report(PCR)3.2. M&E follows project cycle

Development Effectiveness FrameworkStrategic Relevance

Was it the right project at the right time?Output Achievement

Were key deliverables achieved? To what extent were clients satisfied with the advisory service?Outcome Achievement

Were intended short & medium-term changes in behavior achieved?Efficiency

Were costs reasonable vis--vis results? Impact Achievement

Were intended longer-term effects achieved? What would have happened without the intervention? (What is the counterfactual?)These combine into a synthesis rating In addition: Assessment of IFC Role & Contribution: Was IFC particularly catalytic or innovative in its advisory services? Was IFC able to play a unique role not easily filled by others?3.2. Advisory Services Project Completion Report as Self-Evaluation

143.4 Technical Assistance M&E ways to adopt to business practices

ProgressIntegration with project cycleM&E officer playing important and formal roles in quality assuranceQuality control cycle established M&E officers sign-off, Development Impact Dept reviewsM&E quality improvedQuality of logical framework, baseline data, quality of results discussion all improvedChallenges Standard indicators not sufficient to track resultsInadequate information in PCR to justify outcome/impact ratingsPost completion follow-up system not systematicWork quality assessment not covered in PCR

15Outline of PresentationContext of Private Sector Development AssistanceTypes of World Bank Group Private Sector Development interventionsIFC Advisory Services (Technical Assistance) projects M&EPrivate Sector investment projects M&EUse and Influence of M&E

164.1. Characteristics of Private Sector investments supported by IFC and MIGAOperations subject to market testDirect Beneficiaries pay for serviceSome degree of competitionSponsors motivated by profitabilityThird party effects (TPE) with no market feedbackEconomic distortionsPositive and negative externalitiesProvide rationale for public interventionDevelopment Agencys involvementConsistent with market principlesFocused on third party effects (TPE)

Neighbors / EnvironmentCustomersGovernment / TaxpayersEmployees4.2 Assessing Development Results - Stakeholder FrameworkCompetitorsProducers ofComplementaryProductsSuppliersNew EntrantsProjectCompany(financiers)

18How do we assess it? We use the STAKEHOLDER FRAMEWORK: there is a natural breakdown of the stakeholders those affected by private projectsA firm makes something, For sale to someone, Using labor and purchased supplies, Pays taxes to (and/or receives subsidies from) the government, In a place with neighbors And In a market with competitors.

How projects contribute to develompent varies by sector: ASK:project benefits: Employees and government (taxes), and sometimes customers through provision of better quality or more affordable goods.If we talk about utilities: customer benefit (provision of water, gas, power). If we talk about ports or roads, we have customers, the env and private sector through linkages to other industries.What about Oil and Gas? Government and neighboring communities

We will see how the framework help assess risk, concerns and mitigating measures that can be introduced into the project design to enhance its performance.

4.3. Performance dimensions of IFC investments and MIGA guaranteesObjectiveStakeholdersCriteriaStandardsAssessmentDevelopment OutcomeFinanciersBusinesssuccess(Market test)Market benchmarksfor ROICActual financial statements and objectives at approvalCustomers,suppliers, employees, governmentEconomic sustainability(Thirdparty effect)Social benchmarksfor ERRActual financial statements,adjusted by economic prices and valuesNeighbors, environmentEnvironment andsocial effects (Third party effects)WBG performancestandardsActual impact and compliance with performance standardsCompetitors,new entrants, producers of complementary productsPrivate sector development (Third party effect)Objectives on competition, ownershipstructure, demonstration effectsQualitativeassessment of achievements of objectives on competition, ownership structure, demonstration effects

Track indicators and development outcomes SuperviseDOTS**IFCs Development Outcome Tracking SystemApproveDisburseRepaid / ExitCommitIdentify indicators ExpandedSupervisionReport (XPSR)4.4. M&E to adopt investment project cycle

21Development Outcome FrameworkFinancialPerformance

FinanciersEconomic Performance

Govt, taxpayersemployees, customers, suppliersEnvi. & SocialPerformance

Neighbors & EnvironmentPrivate SectorDevelopment

Demo. effects, linkages, corp. gov, investment climate & new entrants+Standard Development Reach Indicators

IFC-wide: Taxes, employment (incl. female)Plus sector-specific: e.g. MSME loans, utility connectionsDevelopment Outcome: Overall effect on countrys development Framework: Derived from Good Practice Standards for Private Sector Evaluation (investments)Each with specific indicators, benchmarks3.2. Performance Assessment

212.2 Investment project M&E ChallengesDifficult to gather info beyond companyGaps in tracking private sector development outcomeMissing indicators, discrepancies between stated outcome and indicatorsReliance on client-submitted dataSome data based largely on assumptionsFinancial Institutions challenges in getting sub-project level information

AdvantagesFinancial data- good quality and frequently updated;Enforcement via contracts / covenantsIntegration with project cycleUse of standard indicators, with baseline and targetQuality control cycle established Dept. review, Development Impact Dept review, External Assurance for Annual Report

22Political Risk Guarantee Business model and M&E challenges (MIGA)

Business ModelCore competency as guarantee underwriting, not financingArms-length relationship to project enterprises;No statutory reporting requirement from projects;Constraints to access No membership on project company boards of directors

ChallengesIn data gatheringIn communicating with project entitiesChallenges in contextual analysisIndustry / sector intelligence

23

Function of M&E in MIGA: significant progress in intro of self-evaluation and indicators.AchievementsNew monitoring system for Environment & SocialMainstreaming self-evaluationDevelopment indicator tracking system introduced

Areas of improvementChallenges in data gatheringInconsistent E&S follow-up on missing documentsDEIS ex-ante data not covering all indicators.Pool of evaluation not adequate to report portfolio results

24

Outline of PresentationContext of Private Sector Development AssistanceTypes of WBG Private Sector Development interventionsTechnical Assistance projects M&EPrivate Sector Investment projects M&EUse and influence of M&E

253.1 Lessons can have a positive influence on results

IFC InvestmentLack of lessons / application of lessons at appraisal associated with poor outcome80% of unsuccessful projects low quality at entry was main factorLow DO with low quality at entry = 40% of them without lessons60% of them lessons were not integrated in design.Improved performance by learning from past failures (Agribusiness case study)Newer successful projects applied lessons from earlier failed projectsIFC Advisory Lessons influence design and implementationNew template asks how lessons have been applied to project designWealth of lessons from PCRSegmented categories of lessonsHigh use of past project PCR in new projectsLow use of SmartLessonsM&E officers review at entry

263.2 Issues have been detected during supervision and actions have been takenIFC InvestmentHigh quality of supervision work: 83% high (XPSR)Half of XPSR identified issues during supervision, 90% of them were detected, 63% of them were fully addressed during supervision.Credit and Environment risk systems as the most frequently used monitoring systemsIFC Advisory PSR widely used95% of staff used PSR to make adjustment. 88% of supervision issues identified in PCR captured in PSR. Poor supervision critical factor of 25% of projects rated low in DE

273.3 M&E information is increasingly used in IFC StrategiesIncreasing use of M&E dataAnnual Strategy processDevelopment impact Dept. provide data and participate discussion. Guidance: asks development issues to coverRatings and reach data used in strategy formulationHistorical DOTS rating comparisons between sectors prioritization of subsectorDOTS ratings vs. profitabilityGap analysis using reach indicators strategy to sharpen focus, areas to reach (i.e. BOP)Development return of different options using reach data (reach per $ cost of capital)

28Three main messages Private Sector operations have specific data gathering advantage/disadvantage based on their business modelM&E systems should adopt to the business practices to be effective and efficientM&E can influence learning, quality of work and outcome

Outstanding issuesDifficult to estimate benefit of M&ENeed innovative approach, some assumptions about counterfactuals;

Monitoring and Evaluation of Private Sector Development

Hiroyuki HatashimaPrivate Sector Evaluation, Independent Evaluation Group (IEG)World Bank Group

Canadian Evaluation Society ConferenceJune 11, 201331