Slide 1

Evaluation Approaches, Methods and Data in a Rapidly Changing Context: Global Crisis Response Evaluations at the World Bank Group

Independent Evaluation GroupWorld Bank / IFC / MIGA

European Evaluation SocietyHelsinki, October 2012

Anjali Kumar, Andaleeb Alam and Ali KhadrIEG World Bank Group11The Global Economic Crisis and the Challenge to WBG Rapid movement of economic events during the global crisis of 2008-2009 Some stock markets lost 50% of value in daysGlobal growth slowdown (3.9 % to -2.1% )Unprecedented scale, contagionEstimated 50-64 million more poor people Motivating strong response from WBG 117 countries received Bank loans during 2009-10; tripling of IBRD lending to $100m17 received crisis support during 19932003Call for real time evaluation of response

2

The global growth slowdown (3.9 % to -2.1% ) affected advanced and developing countries

Advanced economies: 2.6% to -3.3%Developing Bank clients: 6 % to 1 % Europe and Latin America: 7 % to 2 %2Crisis Evaluation: Nature of the ChallengeComplexity of Crisis Response EvaluationsEvents are sudden, unexpected and fast-evolving in characterEvaluative lessons are needed in a compressed time frameConventional evaluation techniques that use information based on a sequential logical framework are of limited useUncertainty prevails

3Daily data show the decline in value of the UK FTSE index from 5,600 to 3,800 in weeks

3Crisis Evaluation: Nature of the Challenge4Scope of the present paperQuestions asked in crisis evaluationData usable to answer such questionsUse of high frequency dataCrisis evaluations at other IFIs / MDBsAre such economic events becoming more frequent? Black swan events may increase because they are probabilistic outliersIncreased need for corresponding evaluative techniques

4Crisis Evaluation Nature of Evaluative Questions5

Evaluative Questions complex evaluations under uncertainty Skills demonstrated in dealing with complexity and changeSpeed of response; reflecting information on events as they occurIdentification of risks involvedResults framework, learning:Quality at entry5Crisis Evaluation: IEGs Phased ResponseIEG prepared a series of real time evaluations on WBG Crisis Response Review of WBG Response to Past Crises (2009) 17 Country Case studies no current crisis dataPhase I Evaluation of WBG Crisis Response (2010)Real time evaluation focused on volume, speed, and early results selective use of data to benchmark Bank actionsPhase II Evaluation of WBG Crisis Response (2011) Used high frequency data to analyze patterns of Bank support relative to crisis incidence6

Monthly data on emerging markets showed declines in industrial production and credit growth not revealed in annual averages61. Retrospective Evaluation - WB Response to Past Crises 17 case studies7

Evaluative questions based on preceding principles Focus on: Scale and modality of the lending response Impact on the Banks own balance sheet Partnerships with other multilateral agenciesContent of the WB crisis-response operations Macroeconomic trade and financial and fiscal Use of DataThe crisis retrospective did not look at data on the current crisis, It did trace data on previous crises- extent of World Banks response relative to its baseline lending, and relative to other international financial institutions; time taken to return to normal lending patterns. 72. WBG Response to the Global Crisis: Phase I Evaluation8

Private Capital Flows during the crisis: January 2007December 2010Evaluative questions : Design aspects readiness, relevance, poverty focus,Implementation aspects Speed, internal organization, instruments, monitoring and evaluationEarly Outcomes and Prospects- Meeting Objectives, additionality, debt sustainabilityData Use: Annual economic data used to trace effects of crisis on Bank borrower / client countries82. WBG Response to the Global CrisisPhase I Evaluation9Alternative Scenarios Poverty in Developing Countries in 2015 and 2020

Phase I Example:Data Use in Evaluation under Uncertainty: Scenario analysisTen year projections of poverty impact of the crisisBased on a series of recent data on GDP growth, trade, capital flows and demand assumptions A contribution of the analysis, from the perspective of evaluation in uncertainty, was the scenario analysis it undertook, in terms of projecting the poverty impact of the crisis, based on a series of data on GDP growth effects, trade and capital flow effects across countries, and assumptions on demand in different segments of the world economy (Figure 5).

9WBG Response to the Global Crisis -Phase II Scope of the Evaluation Utilization of high frequency data across a range of economic variablesTo track multiple dimensions of stressThese multiple measures of stress were only partially correlated. Analysed relative to other IFIs and MDBsMultidimensionality of crisisExchange rate and foreign exchange reserve stressFinancial Stress, including market, credit and banking systemSocial indicators unemployment, private consumption; and Fiscal deficit and public debt as a percentage of GDP)

10

10WB Response to the Global CrisisPhase II Evaluation MethodsUse of High Frequency data to measure multiple dimensions of stress that were not necessarily correlated

11

11Evaluating the Lending Response relative to stress Simple and Composite stress measuresSimple - changes in GDP;Composite - principal factor analysisPeak to trough, period averageIllustrative bands ranking countries according to stress in diagramAre compared with the distribution of incremental lending.Underlying these data are regressionsThat measure crisis intensity and incremental Bank response on a continuous basisCaveats and limitations of the analysisCountry demand, country performance, other IFIsControls can however be introduced for some of these factors

12Incremental Lending Relative to Levels of Crisis: World Bank 12Volume and Distribution of Support - Other IFIs / MDBsIncreased WB lending patterns were also compared with other MDBsCorrelations of each institutions incremental support to crisis intensity in client countriesBased on data obtained from relevant IFIsComparisons undertaken at multiple levels Response to changes in GDP alone, to composite overall stress measures, and compared to specific measures of stressDo not analyse other MDBs overall response Comparisons limited to countries that were common borrowers

13Incremental Lending Relative to Levels of Crisis: Comparisons with Other DonorsData for selective comparisons of the Bank relative to other IFIs

Comparisons with lending patterns of other IFIs was undertaken at multiple levels the response to changes in GDP alone, to composite overall stress measures, and compared to specific measures of stress.

Interviews with staff and evaluators at the European Bank for Reconstruction and Development (EBRD), International Monetary Fund (IMF), Inter-American Development Bank (IDB), Asian Development Bank (ADB), AfDB, European Investment Bank (EIB), and the European Union (EU)

Evaluative evidence provided in recent internal evaluations of some of these agencies was drawn upon (ADB 2011; IMF 2011; EBRD 2010). 13Crisis Evaluation and High Frequency Data - EBRDEBRDs crisis response evaluation established a three phase typologyBased on two high frequency data series on interest ratesEuropean Central Bank deposit facility rate FED funds rate14

EBRDs evaluation of its crisis response traced the character of response in three phases of time; the pre-crisis period (2006 to August 2007), the period of rising instability in the run-up to the crisis (September 2007 to the third quarter of 2008), and the period of the crisis itself (from the fourth quarter of 2008 to the end of 2009). The typology of the three phases was built up based on alternative data series that traced the course of the crisis, as illustrated in Figure 11 below, where use is made of two series of data on rising interest rates the European Central Bank deposit facility rates, and the FED funds rate, rates announced by the European and US central banks that indicate their perceptions of market trends.

14Crisis Evaluation and High Frequency Data - EBRDEBRDs evaluation then traces crisis response in each phase Eg: evolution of EBRD loan pricing in response to market signals 15

15IMF Evaluation of the Fund in the Run Up to the CrisisThe IMF evaluated the quality of its surveillance in the run up to the crisisEvolution of crisis traced also using high frequency data: EMBI Global Spread (Left)TED spread (right)16

The EMBI and the TED spread are both measures of the rise in perceptions of the risk in lending, measured against a risk free benchmark. Both are denominated in basis points. Rising indices / spreads indicates that liquidity is being withdrawn and that lenders believe the risk of default is increasing.

16Conclusions and Suggestions17Rapidly changing situations pose special challenges for evaluatorsDifferent questions That can use high speed data for their responseIn order to give real time feedbackFinancial and some macroeconomic data existIn multiple dimensions and high frequenciesHowever more systematic monthly data on GDP, fiscal deficits etc are still to be producedMore difficult are data on social dimensions Eg integrated global databases on employment or consumptionGreater global cooperation on such data would be a benefitTo policy makers and to evaluators

17