A WORLD BANK GROUP GLOBAL ENV IRONMENT FAC I L I TY PROGRAM PUBL ICAT ION

WORLD BANK GEFPost- Imp lementat ion Impact Assessment

World Bank GEF EnergyEffi ciency Projects

37468

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

WORLD BANK GEFPost-Implementation Impact Assessment

WORLD BANK GEFENERGY EFFICIENCY PROJECTS

SYNTHESIS REPORT

GLOBAL ENVIRONMENT FACILITY PROGRAM

THE WORLD BANK

© 2006

The International Bank for Reconstruction and Development/

THE WORLD BANK

1818 H Street, N.W.

Washington, D.C. 20433, U.S.A.

Manufactured in the United States of America

This report was prepared by Marbek Resource Consultants in association with Angelhoeve Associates

Incorporated, Global Change Strategies International and Lightstream Energy. It is based on the findings

of case studies in Poland (conducted by Le Groupe-conseil baastel ltée), Mexico, Thailand and Jamaica.

Special thanks to the participants at a workshop held in Washington DC, on May 25, 2005, who provided

valuable feedback on the overall findings, and Esther Monier-Illouz for editorial assistance. The study

was partly financed through the Canadian Consultant Trust Fund and the World Bank’s Task Manager

was Samuel Wedderburn.

Cover and book design: The Word Express, Inc. and Esther Monier-Illouz; book design based on work

by Jim Cantrell.

Cover image: Stuart Gregory/Getty Images

Images in text: Josh Westrich/zefa/Corbis

All rights reserved

The World Bank has used its best efforts to ensure that the information contained within this report is

accurate, however, it cannot guarantee its accuracy. The representations, interpretations, and conclusions

expressed herein do not necessarily reflect the views of the Executive Directors of the World Bank or the

government that they represent.

Rights and Permissions

The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without

permission may be a violation of applicable law. The International Bank for Reconstruction and Develop-

ment/The World Bank encourages dissemination of its work and will normally grant permission to reproduce

portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a

request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers

MA 01923, USA; telephone: 978-750-8400, fax: 978-750-4470; Internet: www.copyright.com.

iii

CONTENTS

EXECUTIVE SUMMARY 1

1. INTRODUCTION 7

2. METHODOLOGY 92.1 Impact Maps 92.2 Results Indicators 92.3 Attribution 122.4 Information Gathering 13

3. POLAND EFFICIENT LIGHTING PROJECT 153.1 Impacts 163.2 Conclusions 173.3 Lessons Learned 18

4. ILUMEX 214.1 Introduction 214.2 Framework and Approach 214.3 Project Outcomes 224.4 Intermediate Outcomes 224.5 Ultimate Outcomes 234.6 Sustainability 244.7 Impacts 254.8 Lessons Learned 25

iv

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

5. THAILAND PROMOTION OF ELECTRICITY ENERGY EFFICIENCY PROJECT 27

5.1 Introduction 275.2 Framework and Approach 275.3 Project Outcomes 285.4 Intermediate Outcomes 285.5 Ultimate Outcomes 295.6 Sustainability 305.7 Impacts 305.8 Lessons Learned 30

6. JAMAICA DEMAND-SIDE MANAGEMENT DEMONSTRATION PROJECT 33

6.1 Introduction 336.2 Framework and Approach 336.3 Project Outcomes 346.4 Intermediate Outcomes 346.5 Ultimate Outcomes 356.6 Sustainability 356.7 Impacts 366.8 Lessons Learned 37

7. REVIEW OF CROSS-CUTTING OBSERVATIONS 397.1 Overview 397.2 Design Features 407.3 Project Outcomes 417.4 Post-Project Context 427.5 Intermediate Outcomes 427.6 Ultimate Outcomes 437.7 Impacts 457.8 Conclusions 45

8. LESSONS LEARNED AND RECOMMENDATIONS FOR THE ENERGY EFFICIENCY PORTFOLIO 47

8.1 Project Identification 478.2 Project Preparation 488.3 Public Education and Outreach 508.4 Capacity Building 508.5 Project Implementation 51

Contents

v

9. LESSONS LEARNED AND RECOMMENDATIONS ON IMPACT ASSESSMENT 53

9.1 Planning for the Assessment 539.2 Data Collection 549.3 Analysis 55

TABLES

Table E1 Overview of Project Features and Impacts 2 Table 2.1 Results Framework for ILUMEX 11 Table 7.1 Overview of Project Features and Impacts 40 Table 7.2 Key Design Features 40 Table 7.3 Project Outcomes 41 Table 7.4 Post-Project Context 42 Table 7.5 Intermediate Outcomes 43 Table 7.6 Ultimate Outcomes 44 Table 7.7 Annual Impacts (average of scenarios) 44

FIGURE

Figure 2.1 Impact Map for ILUMEX 10

vii

A/C air conditioning

AGH University of Mining and Metallurgy in Krakow

CDM/JI Clean Development Mechanism/Joint Implementation

CFE Comisión Federal de Electricidad (Federal Electricity Commission, Mexico)

CFL compact fluorescent lamp

CO2 carbon dioxide

DSM demand-side management

DSMO Demand-Side Management Office of EGAT (Thailand)

EE energy efficiency

EECP Energy Efficiency and Conservation Programme (Jamaica)

EGAT Electricity Generating Authority of Thailand

ELI Efficient Lighting Initiative

FEWE Polish Foundation for Energy Efficiency

ESCo energy service company

FIDE Trust Fund for Electrical Energy (Mexico)

GEF Global Environment Facility

GHG greenhouse gas

GWh gigawatt-hour (1 million kilowatt-hours)

ILUMEX Mexico High Efficiency Lighting Project

JDSMDP Jamaica Demand-Side Management Demonstration Project

JPS Jamaica Public Service Company Limited

Kt kilotons

M&E monitoring and evaluation

MCST Ministry of Commerce, Science and Technology (Jamaica)

ACRONYMS AND ABBREVIATIONS

viii

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

MEPS Minimum Energy Performance Standards

Mt megatons

MW megawatt (1,000,000 watts)

NECEL Netherlands Energy Efficient Lighting

NGO nongovernmental organization

NOx nitrogen oxide

PCJ Petroleum Corporation of Jamaica

PELP Poland Efficient Lighting Project

PNEC Polish Network “Energy Cities”

SO2 sulfur dioxide

SOx sulfur oxide

SWH solar water heater

TPEEE Thailand Promotion of Electrical Energy Efficiency Project

TWh terawatt-hours

Note: All dollar amounts are U.S. dollars unless otherwise indicated.

1

EXECUTIVE SUMMARY

Four energy efficiency/demand-side (DSM)

management projects were assessed in

2004—5 to determine their long-term impacts

and to draw out lessons that could be applied

in the development of future World Bank/

Global Environment Facility (GEF) projects.

The projects were:

• Poland Efficient Lighting Project (PELP)

• Mexico High Efficiency Lighting Project

(ILUMEX)

• Thailand Promotion of Electrical Energy

Efficiency Project (TPEEE)

• Jamaica Demand-Side Management Dem-

onstration Project (JDSMDP)

The PELP study was conducted by Le Groupe-

conseil Baastel ltée. The three other studies

were conducted by Marbek Resource Con-

sultants, in association with various partners.

The key elements of Marbek’s methodology

were the development of impact maps, the

identification of key indicators, an approach

to attribution involving a plausible range of

counter-factual scenarios, and an approach

to information gathering based on a review

of previous reports, a field mission of 5—10

days for each project, and the use of detailed

interview protocols.

Findings

All four projects had the objectives of de-

veloping DSM capacity, reducing electricity

consumption and greenhouse gas (GHG)

emissions, and developing technical and

financial program models that could be rep-

licated. The results were:

• Major market transformation in the resi-

dential sector, primarily with respect to

lighting, but also including refrigerators

and air conditioners (in Thailand)

• No significant transformation in the

institutional, commercial, or industrial

sectors, despite having targeted those

sectors in Thailand and Jamaica Signifi-

cant and sustainable energy savings and

GHG emission reductions associated

2

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

Table E1. Overview of Project Features and Impacts

Aspect PELP ILUMEX JDSMDP TPEEE

Implementation Entity Netherlands EE CFE (main JPS (main electric EGAT (main Lighting (NECEL) electric utility) utility — now privatized) electric utility)

Timing 1995–98 1995–98 1994–99 1993–2000

World Bank/GEF $5 million $10 million $3.8 million $9.5 million Contribution

Total Program Cost $5 million $23 million $9.85 million $59.3 million

GHG Reductions 0.53 Mt 0.76 Mt 0.014 Mt 5 Mt (Direct) ($10/t) ($30/t) ($700/t) ($12/t)

GHG Reductions 3.6 Mt 5.3-9.5 Mt 0.2–0.3 Mt 27–45 Mt (Total) ($1.40/t) ($2.40/t - $4.30/t) ($33–49/t) ($1.30–2.20/t)

with the transformation of the residential

markets

• Significant program replication and exten-

sion, both in the countries themselves and

in surrounding countries

• Some development of capacity for DSM

and energy efficiency within government

institutions but with moderate to signifi-

cant gaps remaining

• Significant benefits for consumers in terms

of cost savings and improved product

quality

• Significantly enhanced opportunities for

distributors and retailers of energy-efficient

equipment

• Significantly improved competitiveness of

manufacturers (in Thailand only)

• Small contribution to the integration of

energy efficiency objectives into energy

policies

• Minimal to modest contribution to the

mainstreaming of global environmental

issues into energy policies

• Modest contribution to the development

of procedures and tools needed for global

flexibility mechanisms such as the Clean

Development Mechanism/Joint Implemen-

tation program.

Key features and impacts of the four projects

are summarized in Table E1. Three of the four

projects were implemented by the countries’

publicly owned power generating utilities,

with the exception being Poland. All four

projects took place in the mid-1990s, at a

time when the Bank/GEF favored support

for DSM projects. (Since then, DSM proj-

ects have generally been rejected in favor

of market transformation projects.) On a

cost-per-ton basis, three of the projects were

clearly successful; the exception was Jamaica.

Conclusions

Although DSM projects are seen as less attrac-

tive than market transformation projects by

some economic purists, the analysis indicates

that they can be successful in generating mar-

ket transformation over the short and medium

term. Furthermore, they can produce highly

cost-effective results. The key lesson is that

Executive Summary

3

there is a role both for direct intervention

and for longer-term information and trans-

formation approaches. And it seems clear

that drawing artificial distinctions between

DSM approaches and market transformation

approaches can be counter-productive: DSM

can help to transform markets, but that cannot

happen without the information and outreach

components of what are often labeled “mar-

ket transformation programs.”

During the analysis period, the residential

sector was the easiest one to target for energy

efficiency investments. This may be due to a

number of factors, including more homoge-

neous and simpler markets that were easier

to target, the focus on only one or a few tech-

nologies, the fact that the decision maker and

financial manager are the same person, the

longer time horizon of homeowners versus

business managers, and the fact that the up-

front investment could be more modest. The

fact that the residential opportunities could

be realized during this period reflects the

particular circumstances in those countries

(that is, there was a particular technology

opportunity – such as compact fluorescent

lamps – that could be realized).

Residential lighting, in particular, was a

good end use to target. This was because the

technical concerns had been resolved, light-

ing represented a large portion of end use in

the residential sector, and the barriers were

relatively well understood. In addition, the

technology was poised to deliver dramatic

cost reductions (although the scale of the

impending cost reductions would not have

been known when the projects were initi-

ated). In general, simple programs focused

on straightforward opportunities, and single

technologies seemed to have a greater rate

of success.

Barriers to the widespread adoption of energy

efficiency technologies in other sectors (insti-

tutional, commercial, and industrial) proved

to be insurmountable. Although programs

elsewhere may have had greater success, the

lack of financing was an important barrier in

the projects reviewed for this study.

The programs that worked best involved

tangible interventions that affected invest-

ment decisions (such as through price, avail-

ability, or quality). Education and marketing

programs worked well in support of other

programs but were not sufficient on their own

to generate significant changes in behavior.

Utilities’ interest in DSM depended on their

mandate and the policy environment. In gen-

eral, utilities are unlikely to have an interest

if they are privatized or driven by revenues.

Those with a broad mandate to promote the

public interest are more likely to promote

DSM, particularly if they wish to maintain a

good public image and to offer consumers en-

ergy-saving opportunities to mitigate the eco-

nomic impact of rate increases. Alternatively,

financial or regulatory incentives may make

it worthwhile for utilities to implement DSM.

4

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

Ultimately, a utility’s interest in DSM depends

on the vision of its senior leadership.

Lessons Learned and Recommendations

In line with the above conclusions, the report

outlines a series of lessons learned and rec-

ommendations both for the energy efficiency

portfolio of the World Bank and GEF and for

future impact assessments:

• Choosing the best projects requires good

research. Barriers to implementation

should be assessed realistically and the

goals must be both significant and achiev-

able. In most cases, projects should focus

on particular market segments rather than

attempt too many different interventions.

The most cost-effective opportunities

should be pursued first, but with the

ultimate goal of building up to the most

significant opportunities.

• Projects should be chosen to fit within

both the energy policy goals of the coun-

try and the business plan (and power

development plan) of the implementing

entity. Projects involving utilities should

be considered, but the incentives for in-

volvement need to be clear, and the risk of

potential changes in ownership and regu-

latory framework needs to be assessed and

managed. Other government agencies or

energy service companies should also

be considered as potential implementing

agencies.

• The current bias against DSM projects

implemented by utilities should be re-

considered, but programs need to be

designed with clear exit strategies. Focus

on removing barriers (including aware-

ness, capacity, availability, and price) or

on moving to regulation (energy efficiency

standards).

• Price barriers should be assessed to de-

termine if they are significant and if they

are likely to be reduced through increased

market volume. Subsidies should be con-

sidered if they are likely to be temporary or

if their benefits are greater than the cost.

• Projects should include a strong public

education and outreach component,

including detailed information on cost

savings.

• Education and awareness programs should

be linked to subsidies, labels, or other

concrete programs. Invest sufficient re-

sources or concentrate on smaller market

segments.

• Training and coaching are needed to build

local capacity to develop, design, imple-

ment, monitor, evaluate, and report on

energy efficiency programs and to design

and enforce regulations.

• Strong leadership, vision, and enthusiasm

from senior managers within the imple-

menting entity should be a prerequisite

for program selection.

• A risk management strategy and contin-

gency plans are needed for major contex-

tual changes. A staged approach can help

but, more important, the implementation

entity must have or be able to acquire the

capacity for adaptive management.

Executive Summary

5

• All projects need a monitoring and evalua-

tion component to provide the information

necessary to adjust project parameters in

response to changing circumstances and

results and to support future evaluations

and impact assessments.

7

The World Bank’s Global Environment Facil-

ity (GEF) Coordination Team has identified

the need to assess post-implementation

results of projects, particularly in the area of

longer-term impacts. The GEF attaches great

importance to a project’s sustainability and

replicability, but it has not captured lessons

learned several years after project comple-

tion. The results from this assessment will

be used to improve understanding of the

extent of impacts and to determine how the

long-term goals of GEF operational programs

are being addressed. This objective supports

the Bank’s Monitoring and Evaluation policy

(OD 10.70), which recommends that major

impact studies be conducted on a selective

basis several years after a project is completed

to measure changes brought about by the

project.

Four energy efficiency (EE)/demand-side

management (DSM) projects were assessed

in 2004—5:

• Poland Efficient Lighting Project (PELP)

• Mexico High Efficiency Lighting Project

(ILUMEX)

• Thailand Promotion of Electrical Energy

Efficiency Project (TPEEE)

• Jamaica Demand-Side Management Dem-

onstration Project (JDSMDP)

The objectives of the study were to assess the

long-term impacts of the projects identified

and to draw out lessons learned that may be

applied in the development of future GEF

projects. The key issues addressed included

the following:

• Contribution of outcomes to the achieve-

ment of expected impacts

• The project’s impacts on global environ-

mental benefits

• The project’s impacts on institutional de-

velopment

• The project’s impacts on beneficiaries (for

example, savings or knowledge)

1 INTRODUCTION

8

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

• The project’s impacts on market develop-

ment in the energy efficiency sector

• The project’s impacts on country organiza-

tions, including capacity development

• The project’s impacts on mainstreaming

global environment concerns into national

development and sector policies

• Replicability of outcomes achieved and

catalytic effect of the project

• Lessons for the sustainability of project

impacts and market transformation

• Lessons regarding achievement and mea-

surement of impacts

• Lessons for improving the design and

management of future activities

The studies also attempted to separate proj-

ect-specific impacts from those due to other

sources, including any follow-up projects.

This report provides a summary of the con-

clusions of the four studies and a synthesis

of cross-cutting observations, lessons learned,

and recommendations. In addition to this

Introduction, the report has eight chapters:

• Chapter 2 provides an overview of the

methodology

• Chapter 3 summarizes the assessment of

PELP

• Chapter 4 summarizes the assessment of

ILUMEX

• Chapter 5 summarizes the assessment of

TPEEE

• Chapter 6 summarizes the assessment of

JDSMDP

• Chapter 7 provides an review of cross-cut-

ting observations

• Chapter 8 outlines lessons learned and

recommendations for the energy efficiency

portfolio

• Chapter 9 outlines lessons learned and rec-

ommendations for the conduct of impact

evaluations

9

The PELP study was conducted by Le Groupe-

conseil Baastel ltée. The three other studies

were conducted by Marbek Resource Con-

sultants, in association with various partners.

This section describes the methodology used

by Marbek; the approach used by Baastel is

described in their report on the PELP project

(see Chapter 3).

The key elements of Marbek’s methodology

were:

• Development of impact maps

• Identification of key indicators

• The approach to attribution

• The approach to information gathering

2.1 Impact Maps

Impact maps (sometimes referred to as logi-

cal frameworks or log frames) were used to

describe the results “story” for each project.

Because all three projects reviewed by Mar-

bek predated the adoption of log frames in

project planning, an impact map had to be

retrofitted for each project to the specific

context and objectives. The development of

impact maps was an iterative process—and

would have been so even if log frames had

existed for each project—because the maps

needed to reflect not only what happened

during the project, as documented in the

Implementation Completion Reports, but also

the rest of the story from project completion

through 2004. In each case, the maps evolved

as additional information was gathered and

the storyline became clearer.

The final impact map for the ILUMEX project

in Mexico is presented for illustrative pur-

poses in Figure 2.1.

2.2 Results Indicators

To describe the outcomes and impacts, a

series of indicators was developed for each

result statement. This list was adjusted

during each study to reflect the evolving

storyline and the availability of information.

The final list of indicators for the ILUMEX

2 METHODOLOGY

10

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

ILU

ME

X S

ales

of

CF

L

Mod

el R

eplic

ated

in

Oth

er

Cou

ntrie

s

Glo

bal E

nv

Con

cern

s M

ains

trea

med

in

Nat

iona

l Pol

icy

Dev

elop

men

t of

Glo

bal F

lexi

bilit

y M

echa

nism

s

Activities

Sup

ply

CF

L

Mar

ket C

FL

Fin

ance

CF

L

Loca

l E

nviro

nmen

tal

Ben

efits

Fin

anci

al

Ben

efits

Inst

itutio

nal

Cap

acity

In

crea

sed

in

Sho

rt T

erm

Con

sum

er

Pre

fere

nce

Influ

ence

d

Tra

de A

lly

Cap

acity

and

C

onfid

ence

Im

prov

ed

Rep

licab

le M

odel

P

rovi

ded

Outputs

Project

Outcomes

Interm

ediate

Outcomes

Ultimate

Outcomes

Impacts

Tec

hnic

al

Fea

sibi

lity

Dem

onst

rate

d

Fin

anci

al

Fea

sibi

lity

Dem

onst

rate

Inst

itutio

nal

Cap

acity

In

crea

sed

in

Long

Ter

m

Pro

gram

R

eplic

ated

in

Mex

ico

Oth

er M

exic

an

Ene

rgy

Mar

kets

T

rans

form

ed

Mex

ican

R

esid

entia

l Li

ghtin

g M

arke

t T

rans

form

ed

Ene

rgy

Sav

ings

an

d G

HG

R

educ

tions

Figu

re 2

.1

Impa

ct M

ap fo

r IL

UM

EX

Methodology

11

Table 2.1 Results Framework for ILUMEX

Result Statement Indicators

Output: CFL sales • Units sold

Project Outcome: Technical feasibility • Grid compatibility demonstrated • CFL lifetime • Fixture compatibility • Lighting quality

Project Outcome: Financial feasibility • Total resource cost and benefits demonstrated • Customer participant cost and benefits • Utility cost and benefits

Project Outcome: Institutional capacity • Lessons learned by Comisión Federal de Electricidad (CFE) increased in the short term and other institutions • Experience gained by staff at CFE and in other institutions • Level of confidence of key players in ability to undertake similar programs

Project Outcome: Replicable model • Identification of key program features provided • Applicability of ILUMEX program design to rest of Mexico • Applicability of ILUMEX to other markets

Intermediate Outcome: Program replicated • Number, scope, and scale of efficient lighting programs since in Mexico ILUMEX

Intermediate Outcome: Institutional • Levels of knowledge and ability to implement programs and capacity improved in the long term policies at CFE • Levels of knowledge and ability to implement programs and policies in other government agencies • Development and testing of methods for greenhouse gas (GHG) verification • Existence of appropriate institutional structures • Existence of appropriate institutional incentives

Intermediate Outcome: Consumer • Awareness preference influenced • Confidence in quality • Attitudes toward price and energy savings

Intermediate Outcome: Trade ally capacity • Capacity of trade allies and confidence improved • Number of manufacturers and distributors • Development of supply and distribution channels • Size and distribution of retail network

Intermediate Outcome: Program replicated • Number, scope, and scale of similar programs in other Mexican energy markets

Intermediate Outcome: Program replicated • Number, scope, and scale of similar programs in other countries

Ultimate Outcome: Global environmental • Awareness and ability to calculate and certify emissions concerns mainstreamed in national policy reductions for carbon credits

Ultimate Outcome: Development of • Knowledge and experience incorporated into Clean global flexibility mechanisms Development Mechanism (CDM)/Joint Implementation (JI) mechanisms

(continued on next page)

project is presented for illustrative purposes

in Table 2.1.

12

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

Ultimate Outcome: Lighting market • Availability and marketing transformation • Price • Sales

Impact: Energy savings and GHG • Gigawatt-hours (GWh) saved reductions • Tons of GHG reducedImpact: Financial benefits • Gigawatts of capacity deferred • Total resource cost and benefits • Customer participant cost and benefits • Utility Cost and Benefits

Impact: Local environmental benefits • Reductions in NOx and SO2 emissions

2.3 Attribution

Indicators like those in Table 2.1 provided a

way to describe the changes that occurred as

a result of the projects. For each of these, we

attempted to quantify or describe the change

and the way in which the projects had a role.

This was largely a qualitative exercise that

involved rating the degree of influence of the

project on each intermediate outcome and

the degree of influence of each intermediate

outcome on the final outcomes (on the basis

of available evidence).

To calculate the impacts on energy use and

emissions, we needed to know what had

actually happened since the projects ended

(and what is projected in the near term), as

well as what would have happened in the

absence of the projects (counter-factual sce-

narios). The difference between these two

cases constitutes the incremental impact.

Unfortunately, historical and projected mar-

ket data are hard to obtain.

Nevertheless, in the interest of illustrating

the quantitative implications of our assess-

ment, we attempted to estimate impacts by

constructing three subjective scenarios for

each project:

• A With Project scenario that incorporates

all the available data on product sales

and extrapolates these data to provide a

complete time series of sales from the start

of the project until 2010 (this year was

chosen because it provides a reasonable

period of time to allow for the differences

between the scenarios to become evi-

dent)

• A No Project – High Baseline scenario

that sets a high boundary of the range of

estimates of sales that would have hap-

pened in the absence of the project and

hence represents the lower boundary of

the range of estimates of the incremental

impact of the project

• A No Project – Low Baseline scenario

that sets a low boundary of the range of

estimates of sales that would have hap-

pened in the absence of the project and

hence represents the higher boundary of

the range of estimates of the incremental

impact of the project

Table 2.1 Results Framework for ILUMEX (continued)

Result Statement Indicators

Methodology

13

Although these scenarios – particularly the

two counter-factual ones – were necessarily

based on “educated best guesses” and many

assumptions, we expected that by using a

wide range in the scenarios, as well as con-

servative assumptions, the results would be

reasonably credible.

2.4 Information Gathering

Information gathering involved a review of

all project reports and studies, along with

any post-project evaluations, assessments,

or studies. It also involved a field mission of

5—10 days to interview project participants,

beneficiaries, and informants. One critical

aspect of the field missions was to identify a

sponsor (preferably from the implementing

organization) who could facilitate access

to key informants and documents and who

could provide comments on findings and

reports. To facilitate interviews, protocols

were developed to request information on

the key indicators and to probe for informa-

tion on attribution or on counter-factual

scenarios.

15

The Poland Efficient Lighting Project was ap-

proved in 1994 and took place in 1995—8.

The $5-million project was funded by GEF

and implemented through the International

Finance Corporation. The Netherlands En-

ergy Company received the contract to be the

local executing agency and in turn subcon-

tracted the work to its subsidiary, Netherlands

Energy Efficient Lighting. The project was

designed to encourage the replacement of

conventional incandescent bulbs with ener-

gy-efficient compact fluorescent lamps (CFLs)

in households in Poland, thereby reducing

electricity consumption and the emissions of

greenhouse gases associated with electricity

generation and distribution.

This study was conducted by Le Groupe-

conseil Baastel ltee between April and July

2004. It was based on a thorough desk

review, preliminary interviews, and a two-

week mission to Poland, during which the

Study Team met with key informants and

reviewed additional documentation. The

interviewees included representatives from

the executing agency, relevant ministries,

electric utilities, municipalities, environmen-

tal nongovernmental organizations (NGOs),

and the private sector, as well as consumers,

retailers, CFL and luminaire manufacturers,

and other relevant stakeholder groups. The

Study Team also conducted a basic retail

survey in the major cities in Poland, visiting

large, medium-size, and small stores that sell

CFLs and luminaires, as well as electric and

lighting specialty shops.

The mission of the survey was to assess

availability, energy-efficient lighting promo-

tion, shelf space, prices, types, brands, and

other relevant data. To collect data on and

from some primary project beneficiaries, the

assessment team arranged a focus group of

consumers to collect detailed quantitative

and qualitative data to be measured against

the outcomes of the project and to assess

the current state of affairs. The assessment

team contracted with a nationally recognized

3 POLAND EFFICIENT LIGHTING PROJECT

16

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

social science research firm (PINTOR) to as-

sist with the organization and independent

administration of a survey to interview these

project beneficiaries. This focus group al-

lowed for a verification and triangulation of

findings from other sources.

3.1 Impacts

For consumers, there have been many long-

term direct and indirect benefits from PELP.

As the market for CFLs has been transformed,

the availability and selection of CFLs has

increased dramatically. The price of CFLs

has remained similar since the end of PELP,

which means a lower price in real terms.

Consumers are still enjoying energy savings

by using CFLs, even though it is difficult to

calculate such savings in the current Polish

context of rising electricity prices and a rap-

idly evolving electricity sector. Generally,

consumers are aware of the benefits of CFLs

and are purchasing them. However, consum-

ers need to be further informed about the

differences between high- and low-quality

CFLs. Cheaper and lower-quality CFLs im-

ported from Asia are less energy-efficient,

but many consumers are attracted to the

lower price. Because many lower-quality

CFL bulbs on the market burn out early or

do not work properly, there are indications

that consumers’ level of satisfaction has de-

creased since PELP.

Some manufacturers have benefited greatly

from this market transformation, while oth-

ers have experienced significant challenges.

Sales overall are very high, as is the penetra-

tion rate. Philips and Osram continue to

dominate the CFL market, as they did at the

time of the project. Philips has certainly ben-

efited directly from PELP (as did ES Systems in

the luminaire market), while Osram and other

manufacturers have benefited indirectly,

through a transformed market and increased

sales overall. However, local manufacturers

have not been able to compete with cheaper

Asian imports or with the big established

players. There are more “local players” in the

market now, but they are mostly distributors

of imported goods and they do not see their

businesses as sustainable. The sustainability

of the CFL market is further in question, as

consumer confidence is apparently erod-

ing due to lower-quality products and the

absence of adequate mechanisms to control

this aspect.

It seems that PELP did have a significant

impact on the development of EE-related

curricula at various levels of schooling in

the country, although this is not reflected

in education laws or policies per se. PELP

catalyzed an educational and intellectual

focus on EE-related issues in Poland. These

impacts can be observed in the wide array

of courses and curricula, especially at the

university level, dealing with this topic and

in the greatly increased number of graduates

and experts.

Some electric utilities have benefited from

PELP over time and have adopted DSM and

Poland Efficient Lighting

17

EE to avoid large investments. However, this

adoption is usually only for peak shaving

and is certainly not widespread. Through

PELP and the initiatives that grew from it,

municipalities continue to reap the benefits

of EE, as the awareness of CFLs has spread

to municipalities beyond those targeted by

PELP.

Using available information on the current

CFL market, the assessment team concluded

that the total GHG emission reduction at-

tributable to PELP approximates 3.62 million

tons of carbon dioxide (CO2). In addition,

the global environmental benefits in terms

of GHG emission reduction induced by sub-

sequent projects that were inspired by PELP

(or that replicated it to some degree) could

also be significant. Total anticipated impacts

of three key PELP follow-up or replicated

projects are in the range of a reduction of

18.4 million tons of CO2.

Another strong impact from PELP was ob-

served in the number, type, and quality of

replications that have taken place. Lessons,

approaches, and experiences from PELP have

been replicated widely, both in Poland at var-

ious levels and at the international level. PELP

lessons and approaches were replicated by

the Polish Network “Energy Cities” (PNEC),

the Polish Foundation for Energy Efficiency

(FEWE), stakeholders at the University of

Mining and Metallurgy in Krakow (AGH),

and energy consultants at municipal and

national levels in Poland, and can be seem in

initiatives such as the Polish Efficient Motor

Project. At the global level, the International

Finance Corporation’s Efficient Lighting Initia-

tive (ELI) stands as a good example of a repli-

cation activity of PELP, although its approach

is appropriately adjusted to the context of the

country in which it takes place.

3.2 Conclusions

Some stakeholders believe that PELP may

have indirectly contributed to bringing some

EE-related issues to the surface and to the at-

tention of the central government. However,

they also recall that while the government

officially supported EE and PELP, it was not in-

volved in the project, nor was there an active

relationship or mechanism in place to ensure

feedback of PELP experience to the govern-

ment at the policy level. Poland’s current ac-

cession to the EU is the main impetus for the

laws and policies in Poland, which are more

focused on renewable energy before (RE)

than on energy efficiency. EE has therefore

not been mainstreamed into Poland’s laws

and policies to the extent that could be ex-

pected. Where it has, enforcement of legisla-

tion has been lacking and requires attention.

In addition, PELP did not have an observable

impact on legislation development.

There may be a general capacity for policy

making, planning, and decision making for

EE and DSM because there are national laws

and policies that address efficiency issues.

However, due to various contextual factors,

laws are not well enforced or implemented.

18

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

At the municipal level, the capacity for EE

planning and decision making is almost

nonexistent. At both national and municipal

levels, the government’s capacity to imple-

ment decisions on EE and DSM does not

seem to be present. However, the capacity

does exist within other nongovernmental

institutions or organizations. Utilities also

have this capacity but generally are not

as interested in pursuing these initiatives,

given the current market realities in Poland.

PELP had limited success in bringing change

to that overall situation. Nevertheless, it did

support pilot initiatives in certain munici-

palities that, in partnership, have furthered

the implementation capacity of some of the

groups mentioned. Given the EE situation

in the country, the need to broaden that

experience and the reach of that capac-

ity remains significant. The incomplete

enabling environment for pursuing EE in

Poland is not conducive to enforcement of

legislation.

The capacity for information management

and awareness raising exists at certain levels

within the country, and some of it was fur-

thered through PELP. For example, institu-

tions such as AGH, PNEC, and FEWE all have

the capacity to undertake awareness-raising

initiatives throughout the country and are

doing so. However, groups such as FEWE

and PNEC rely on project-related funding and

have little core funding available to ensure

continuity and a strong logical framework

for their actions on EE. The national govern-

ment does not appear to be committed to

EE and DSM in any forceful manner, and

municipalities are constrained in both hu-

man and financial capacity for awareness

rising. As many stakeholders made clear,

PELP focused primarily on shifting the market

for CFLs and not necessarily on raising EE

awareness or developing the capacity to do

so for the long term. It should be noted that

this was a contextually appropriate choice

for the project’s emphasis. Nevertheless, it

impeded to some degree the contributions

made to long-term raised awareness and

related capacities for EE.

Energy services companies (ESCos), which

normally would be a key player in pooling

resources by marketing, implementing, and

structuring financing for EE measures at vari-

ous levels (and were envisioned as such by

PELP), are not an important player in Poland.

PELP’s efforts to build a legacy of ESCos as

a key institutional driver of broader and sus-

tainable market transformation for efficient

lighting in Poland in the longer run have not

been fruitful. This failure appears to be due,

in particular, to the absence of an adequate

market-enabling environment to allow them

to flourish.

3.3 Lessons Learned

An overarching lesson from this impact

assessment of PELP is that certain key chal-

lenges must be considered and addressed

in market transformation toward energy ef-

ficiency in developing economies:

Poland Efficient Lighting

19

• The development of an appropriate en-

abling environment for energy efficiency

actions is very important.

• Building the capacity of local governance

institutions to plan, implement, and enforce

legislation and to pool adequate human

and financial resources is paramount.

• Special attention must be paid to build-

ing and sustaining the capacity to raise

awareness on EE/DSM issues and benefits

at various levels.

• The promotion of global environmental

benefits can contribute to a more sus-

tainable, longer-term attitudinal change

toward energy-efficient practices.

• Sustaining raised awareness that will have

a long-term impact on changing attitudes

and behaviors requires that adequate

emphasis be placed on ongoing informa-

tion dissemination and education, at all

levels.

• To achieve total and sustainable market

transformation, capacities must also be

built within central government, munici-

palities, and nongovernmental actors ac-

tive in that sector.

• Where the macroeconomic and social

contexts are not conducive to EE, it is dif-

ficult to get government commitment to

steer the process and ensure an adequate

enabling environment.

• In a rapidly changing and opened (or

“opening”) economy, adequate atten-

tion must also be paid to the capacity to

compete with imported products and the

effects this might have.

The impact assessment of PELP generated

other lessons for EE projects:

• Direct incentives to manufacturers can

be an efficient way to bring about mar-

ket transformation in favor of a given EE

product.

• For sustained impact in EE or DSM, ap-

propriate champions must be identified

and integrated in the initiatives.

• In the context of EE initiatives, it can be

beneficial to work more with end users

and final consumers, who have the most

to gain from electricity savings.

• The argument about health (namely,

children’s eyesight and its relationship

to proper lighting) can be a convincing

factor in achieving public support for EE

lighting initiatives.

Assessing PELP over the long term helped

identify several lessons for improving on the

experiences of the project:

• Having an “objective” promoter of the

product to the public, someone who is seen

as independent, is an effective promotional

tool.

• Contextually appropriate project execu-

tors and partners are key factors in the

short-term and long-term success of EE

initiatives.

• The involvement of appropriate local re-

sources in the design of the project helps

ensure that the intricacies of the local

context are taken into account.

20

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

• Project implementation approaches must

be contextually appropriate.

• To address emerging constraints or to build

on successes, attention must be given to

follow-up activities.

21

This Chapter deals with the second case

study: the Mexico High Efficiency Lighting

Project, known worldwide as ILUMEX. Mar-

bek Resource Consultants conducted the

study in association with Lightstream Energy.

A field mission to Mexico City was conducted

November 29—December 7, 2004.

4.1 Introduction

ILUMEX was approved by the World Bank

in 1994 and took place in 1995—8. The

project was cofinanced by a GEF grant of $10

million, about $10 million by the Mexican

government, and a grant of approximately $3

million from Norway. It mainly involved the

promotional sale of high-efficiency compact

fluorescent lamps; 2.6 million had been

sold by the end of the project. The project

was implemented by the Comisión Federal

de Electricidad, Mexico’s main public elec-

tricity utility. It was initially implemented

in the cities of Guadalajara and Monterrey

(Mexico’s second and third largest cities) and

eventually was expanded to cover all of the

Mexican states of Jalisco and Nuevo León and

parts of the adjoining states Colima, Nayarit,

Coahuila, and Tamaulipas.

4.2 Framework and Approach

The study assessed the success, sustainability,

and attribution of the following outcomes

and impacts:

• Project Outcomes: demonstration of tech-

nical and financial feasibility, increase in

institutional capacity, development of a

replicable model

• Intermediate Outcomes: replication of

the model; additional increase in insti-

tutional capacity; consumer preference

influenced; improvement in capacity and

confidence of manufacturers, distributors,

and retailers

• Ultimate Outcomes: transformation of

the Mexican residential lighting market

• Impacts: energy savings, GHG reduction,

reductions in air pollutants of local con-

cern, capacity savings, financial benefits

4 ILUMEX

22

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

To identify and attribute impacts, we de-

veloped three scenarios: one representing

actual and forecast CFL sales (with ILUMEX)

and two counter-factual scenarios repre-

senting a low and a high range of how CFL

sales would have evolved in the absence

of ILUMEX.

4.3 Project Outcomes

In addition to the direct impacts of the CFL

sales during the program, ILUMEX demon-

strated the technical and financial feasibility

of the technology and the approach and

established a model for future programs. The

key features of the ILUMEX model were:

• Bulk purchases of high-quality CFLs

• CFL sales in local utility service centers

• Availability of low-interest financing

• Payment in installments on the electric

bill

• Subsidized prices

4.4 Intermediate Outcomes

Our assessment concluded that ILUMEX has

proved that its features have great replica-

tion value. The model was followed in a

nationwide program implemented by the

Trust Fund for Electrical Energy (FIDE) and

in a program conducted by Luz y Fuerza

del Centro, Mexico City’s utility. The utility

service center sales model was adapted and

used in ELI programs in Argentina, Peru, and

Costa Rica. Aspects of the program have in-

fluenced a broad range of programs – from

other demand-side management programs

in Mexico to a broad range of international

initiatives.

ILUMEX helped develop a better under-

standing of the role and functioning of DSM

programs within the Federal Electricity

Commission. The program confirmed the

technical viability of using CFLs, including

product quality, durability, and low impacts

on the electricity grid. It also clarified the role

of subsidies in promoting energy-efficient

technologies newly on the market. ILUMEX

contributed significantly to building a culture

of energy efficiency programs within CFE

and other government ministries and agen-

cies. It stimulated activity within a variety of

government organizations and agencies and

helped to refine visions, renew mandates,

clarify roles, and establish links among the

various institutions. The result is that Mexico

has one of the most well developed institu-

tional structures for the promotion of energy

efficiency.

The economic analysis of ILUMEX clarified

the financial implications for CFE of DSM

programs. Unfortunately, the institutional

incentives have not been realigned (that is,

lower electricity sales due to DSM cause

a financial loss for the utility), and so the

involvement of CFE as a promoter of energy

efficiency continues to rely mainly on the

utility’s broad interpretation of its public

service mandate. Given that CFE is a public

entity, making a profit is not as important as

providing electric service.

ILUMEX

23

Consumer awareness of CFLs and their pref-

erence for them has grown substantially in

the period since ILUMEX. Programs such as

ILUMEX and subsequent programs, such as

FIDE’s CFL sales program, played an impor-

tant role in achieving this outcome. ILUMEX

succeeded by exposing the technology to

consumers who otherwise would not have

considered it, by putting a strong emphasis

on product quality, and by making the case

for the financial and energy benefits. Product

quality has effectively been removed as a

consumer concern, although there is a current

trend toward lower-quality products on the

market with low price appeal. Most consum-

ers have a good appreciation of the potential

financial savings that can accrue from the

use of CFLs and, as a result, roughly one

third indicate that they intend to buy CFLs.

The incentive programs have reached mainly

middle- and higher-income customers.

Coinciding with ILUMEX, the worldwide

market for CFLs – including the Mexican

market – has grown substantially. ILUMEX

and subsequent programs have played a

significant role in expanding the Mexican

market severalfold and attracting manufac-

turers and distributors to it. Because of the

bulk sales generated by ILUMEX and FIDE,

manufacturers and distributors felt confident

enough of the sustainability of the market to

invest in distribution channels and market-

ing. There is now healthy competition from a

range of manufacturers, providing a variety of

CFLs of differing quality and price. Together

with retailers, manufacturers are experiment-

ing with a variety of marketing approaches,

including various methods of display and

packaging in a broad array of stores.

4.5 Ultimate Outcomes

The key outcome has been transformation

of the Mexican residential lighting market.

When ILUMEX began, CFLs were expensive

and hard to find, and they had very low sales

volumes. Today they are affordable, are vis-

ible in most retail outlets, and have a signifi-

cant and growing share of the market.

• Price: Prices dropped from approximately

$15CFL prior to ILUMEX to less than $3/

CFL in 2004. Although international mar-

ket developments played the largest role

in this reduction, ILUMEX was important

in establishing Mexico’s contribution to

that worldwide market and it played a sig-

nificant role in influencing consumer price

expectations in the Mexican market.

• Availability: Before ILUMEX, CFLs were

very hard to find in retail markets. Today

they take up most of the lighting shelf

space. The Mexican consumer association

PROFECO’s 2003 survey found about 28

brands and over 250 different models in the

country. Although many factors influenced

the increase in availability, the role played

by ILUMEX and its successor programs

in nurturing the Mexican market and in

building capacity and confidence among

trade allies suggests a relatively high at-

tribution.

24

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

• Sales: CFL sales in Mexico are esti-

mated to have risen from approximately

500,000 in 1995 to over 7 million in

2004. Sales are expected to reach 13

million a year by 2010. The increase has

been driven by higher electricity tariffs,

reductions in the price of CFLs, greater

consumer awareness and interest, and

better availability and marketing. ILU-

MEX had no influence on the increase in

electricity tariffs, but it had a moderate

influence on CFL prices and a significant

influence on consumer awareness and

product availability.

ILUMEX has contributed modestly to main-

streaming certain aspects of climate change

into national policy. However, such con-

cerns remain secondary to energy security

and to economic and local environmental

concerns.

4.6 Sustainability

Following the completion of the project, con-

cern was expressed by World Bank staff about

the lack of sustainability of a program that

relied on subsidies to drive sales. Although

the concern is understandable, the evidence

indicates that the program outcomes were

sustained and continue to be sustained on

several levels.

• Subsidies and Replicability: The multiple

replication of the ILUMEX model (includ-

ing its subsidy components) has shown

that under certain conditions (that is, in

a public utility with a broad mandate),

subsidies can be justified and maintained

despite the financial cost. This is because

CFE and FIDE recognized that, on a total

resource basis and for their customers, the

benefits outweighed the costs, and they ac-

cepted their public responsibility to invest

in this outcome. Fortunately, over time,

the need for the subsidy gradually disap-

peared. Successor programs were then

able to succeed solely through the other

aspects of the model. Ultimately, the need

for these programs is also disappearing,

and CFL sales are becoming self-sustaining

and market-driven.

• Capacity Development: The evidence

suggests that institutions, knowledge, and

experience—both within the government

and in the utility and the broader commu-

nity of energy professionals—have reached

a critical mass. This means that even if

particular individuals or institutions cease

to be involved, there is sufficient dissemi-

nation of expertise and best practices to

sustain the capacity in Mexico.

• Market Transformation: Provided that the

broad features of Mexico’s energy market

remain supportive (that is, that electricity

tariffs remain high in relative terms), the

changes that have taken place in terms

of CFL availability, prices, manufacturer

and retailer confidence, and consumer

awareness and preferences are likely to

be irreversible. This suggests that prices

are likely to continue falling and sales are

likely to continue rising.

ILUMEX

25

4.7 Impacts

The total CFL sales induced by ILUMEX and

the associated increase in stock are estimated

to have produced energy savings of approxi-

mately 1.4 terawatt-hours (TWh) in 2004 and

associated GHG emission reductions of 850

kilotons (Kt) of CO2-equivalent. Over the pe-

riod 1995—2004, GHG emission reductions

of 3.4 megatons (Mt) were achieved, and a

further reduction of 9 Mt is expected to 2010.

The following range of impacts is attributed

specifically to the ILUMEX program:

• Sales of between 3.7 to 5.7 million CFLs

in 2004

• Energy savings of between 740 to 1,225

GWh in 2004

• GHG emission reductions of between 430

to 715 Kt in 2004

• GHG emission reductions of between 5.3

to 9.5 Mt between 1995-2010

• NOx emission reductions of between 1.3

to 2.2 Kt in 2004

• SO2 emission reductions of between 8.4

to 13.8 Kt in 2004

• Capacity savings of between 190 to 300

MW in 2001 (the last year of capacity

constraint)

• Total resource net benefits of between $90—

155 million over the period 1995—2010

(3.5 to 6.5 times the original investment)

• Total net benefits to consumer participants

of between $140—245 million over the

period 1995—2010.

• Total net costs to the utility of between $45

to $85 million over the period 1995—2010

4.8 Lessons Learned

Our assessment suggests a number of lessons

to be applied both in future programs and,

more broadly, in developing approaches to

energy efficiency.

• Pilot programs like ILUMEX in selected

markets are a good way to achieve sig-

nificant market presence of a new en-

ergy-efficient technology at a reasonable

cost. The success can then be replicated

and exported to other markets. As energy

efficiency markets evolve, the amount of

public support for new technologies will

diminish, making further new entrants

cheaper to bring to market.

• Each of the features of ILUMEX mattered in

determining the outcomes. These features

were extraordinarily well suited to the

circumstances and may be appropriate for

a variety of future applications. But any

replication of the program needs to take

into account local conditions, including

electricity tariffs, deregulation, market

conditions, institutional capabilities, and

so on. Indeed, early pilot programs help

mature an institutional capability to carry

out further programs in the future at lower

cost and higher effectiveness.

• To be effective and sustainable, programs

that rely on capacity building need a broad

reach that includes a variety of govern-

ment and private parties that have a role

in the future evolution of the market. It is

important to share program results widely

among these entities in order to provide

26

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

the maximum opportunity for replication

and lessons learned. The ILUMEX experi-

ence was well communicated among both

energy and other government spheres—

making the programs that have followed

easier to implement and overcoming more

easily the reticence of other government

and private sector actors.

• It is important to align a program’s im-

mediate economic objectives with the

ultimate objectives of both the funding

agencies and the implementing agencies.

In this case, the World Bank and the CFE

had slightly different objectives (global

environmental objectives versus national

socioeconomic ones). However, the ob-

jectives were mutually compatible, and

this reinforcement added to the broad

long-term success of the program.

• Although in hindsight some of the techni-

cal specifications may have been exces-

sively stringent, the investment in high-

quality CFLs was crucial to overcoming

consumer concerns about CFL quality

and ensuring that quality never became

an issue once ILUMEX began.

• It is necessary to establish a significant

market presence in order to get critical

mass for a technology such as CFLs. This

justifies the focus on specific markets,

such as residential medium- and high-in-

come consumers, as well as the focus on

selected cities.

• It is important to disseminate financial

information along with technical infor-

mation so that consumers can appreciate

the savings offered and make their own

judgments about reasonable paybacks.

• Marketing strategies need to be appropri-

ate to the target customers’ needs. Thus,

strategies may vary between customer

groups in differing income strata.

• Bulk purchases proved an effective feature

of ILUMEX to stimulate the market and

to provide lower unit CFL prices for the

program.

27

This Chapter deals with the third case study:

the Thailand Promotion of Electrical Energy

Efficiency Project. Marbek Resource Con-

sultants conducted the study in association

with Global Change Strategies International.

A field mission to Bangkok was conducted

January 17—28, 2005.

5.1 Introduction

The Thailand Promotion of Electricity Energy

Efficiency project was approved in 1993 and

took place from 1993 to 2000. The project

was cofinanced by a GEF grant of $9.5

million, $5.4 million from the government

of Australia, a loan of up to $25 million

from the Overseas Economic Cooperation

Fund of Japan/Japan Bank for International

Cooperation, and funds from the Electricity

Generating Authority of Thailand (EGAT).

The national project consisted of a five-

year DSM Plan. It was implemented by the

Demand-Side Management Office (DMSO)

of EGAT.

5.2 Framework and Approach

The study assessed the success, sustainability,

and attribution of the following outcomes

and impacts:

• Project Outcomes: demonstration of tech-

nical and financial feasibility of a variety of

programs in the residential, commercial,

and industrial sectors

• Intermediate Outcomes: continuation of

the DSMO and its programs, changes in

consumer preferences, improvement in ca-

pacity and confidence of energy efficiency

allies (distributors and retailers), increase in

government institutional capacity, improve-

ment in the strength of the energy services

sector (ESCos and financial organizations),

a strengthened manufacturing sector

• Ultimate Outcomes: transformation of

the residential, commercial, and industrial

electricity appliance markets; program

replication and extension; improved load

management

5 THAILAND PROMOTION OF ELECTRICITY ENERGY EFFICIENCY PROJECT

28

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

• Impact: energy savings, GHG reductions,

reductions in air pollutants of local con-

cern, capacity savings, financial benefits

To identify and attribute impacts, studies con-

ducted by the DSMO were used. DSMO data

were extrapolated to 2010 and two counter-

factual scenarios were designed to represent

a lower and upper bound of impacts in the

absence of the project. These counter-factual

scenarios were used to calculate the incre-

mental impact of the project. Only impacts

due to T8 fluorescent and CFL lighting and

to energy-efficient air conditioners (A/C) and

refrigerators were assessed.1

5.3 Project Outcomes

In addition to the direct impacts of energy-

efficient appliance sales during the program,

TPEEE demonstrated the technical and fi-

nancial feasibility of the technology and the

approach and established a model for future

programs.

5.4 Intermediate Outcomes

The DSMO has remained strong. In par-

ticular, the EGAT Label #5 remains highly

successful, and the DSMO’s budget and

staffing remain healthy. The DSMO has

helped EGAT’s corporate image and has

developed and maintained a strong internal

evaluation and testing capacity. At the same

time, however, the rate of increase in DSMO

activities has been reduced since 2000,

there is a sense of reduced leadership at the

DSMO, and there is a poor understanding

of the DSMO mandate at EGAT. In addition,

EGAT’s privatization impasse has affected

DSMO initiatives.

DSMO activities have increased public

awareness that energy is a measurable re-

source. In particular, the energy efficiency

labeling campaigns have achieved high

recognition and have been highly effective

in changing residential appliance purchasing

behavior. Behavior change in the commercial

and industrial sectors is much less evident.

The DSMO has retained connections with a

number of allies in the private and public sec-

tors. In the public sector, DSMO has a mixed

record of maintaining links. On the positive

side, the DSMO’s links to the Ministry of

Energy’s departments such as the Department

of Energy Development and Promotion and

the Energy Policy and Planning Office are

relatively strong. But there has been a decline

in interdepartmental collaboration, and the

DSMO’s links to other government agencies

have not been strong.

Although the DSMO’s program achieve-

ments have not become a significant factor in

EGAT’s power planning process, the DSMO

and the TPEEE project appear to have had a

modest impact on certain areas of govern-

ment energy policy:

1 T8 is a widely used term for high efficiency fluo-rescent lamps, the other commonly used type being T12.

Thailand Promotion of Electricity Energy Efficiency Project

29

• The Ministry of Energy has realized the

importance of energy efficiency and has

brought all the important national insti-

tutions, including EGAT, into a newly

reorganized Ministry structure that is still

evolving.

• The programs and the policy instruments

of the Ministry of Energy are also evolving

quite quickly, based on the lessons learned

and new insights into the role of energy

efficiency in economic development.

• The Minister of Energy has recognized

and emphasized that EGAT must seek to

reduce energy demand.

The DSMO’s earlier ESCo initiatives have

served as the spark for many of the current

ESCo initiatives in the public and private

sector. The DSMO can also be given some

credit for encouraging some specific private

sector ESCos.

The DSMO retains strong links to the manu-

facturers with which it has worked closely,

and DSMO activities, particularly related

to EE Label #5, have helped some Thailand

manufacturers gain a stronger position in

international markets.

5.5 Ultimate Outcomes

DSMO programs have had a substantial im-

pact on the residential equipment appliance

market, and it appears that this impact has

only strengthened since the project’s close.

The most significant impact of DSM activities

concerns the labeling program. In the long

run, the government’s recent move to Mini-

mum Energy Performance Standards (MEPS)

may be even more significant.

With respect to residential lighting, market

transformation is under way, though it is

not yet complete for CFLs, and there is

slower progress for low-loss ballasts. For

refrigerators, significant market transforma-

tion is occurring, but it is not yet complete.

For air conditioners, market transformation

is also under way but not yet complete. A

significant challenge in that market lies in the

number of manufacturers and the difficul-

ties this poses in terms of negotiation and

product testing/verification. It is expected

that the MEPS for A/C in March 2005 will

greatly increase the number of EE units being

sold. Sales of Label #5 lights, refrigerators,

and A/C units are estimated to have risen

substantially in the period to 2004, including

85 million T8 fluorescent tubes, 5 million

CFLs, 12.4 million refrigerators, and 2.8

million A/C units.

The TPEEE project’s medium-term impact on

commercial energy efficiency has not been

nearly as significant as in the residential sec-

tor. DSMO and government procurement

policies have had some success in commer-

cial and institutional energy-efficient lighting

and A/C equipment sales, but a significant

potential remains.

Although there is currently some activity un-

der way in industrial energy efficiency, very

30

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

little of this can be traced back to DSMO or

the TPEEE project.

5.6 Sustainability

The TPEEE Program has transformed the mar-

ket for residential appliances such as air con-

ditioners, fridges, and lighting. The advent

of Minimum Energy Performance Standards

for these and other equipment will ensure

sustainability of this transformation.

The successes of the project have been sus-

tained by EGAT’s continued funding of the

DSMO. At the end of the TPEEE program,

DSMO funding from the dedicated tariff

was removed, and EGAT elected to con-

tinue funding at equivalent (or higher) levels

through its own revenue base. However,

the potential for privatization represented a

significant threat to the DSMO. Fortunately,

the office has been located in a unit of

EGAT that will not be privatized (the trans-

mission division). The operation of a DSM

program by a transmission company will be

somewhat unique and may require a clear

government policy commitment to keep the

DSMO running.

DSMO’s sustainability has also been strength-

ened by high consumer demand for its servic-

es. With the program’s high visibility and con-

sumer confidence in DSMO activities (namely,

Label #5), it is in the government’s and EGAT’s

interest to maintain the program.

5.7 Impacts

Over the period, sales of EE products have

resulted in cumulative energy savings of ap-

proximately 28 TWh. Of this total, 17.0—23.5

TWh is attributable to TPEEE. This translates

to GHG emission reductions of approximate-

ly 21 Mt CO2 equivalent from 1993 to 2004.

Of this, it is estimated that 12.6—17.4 Mt can

be attributed to the TPEEE project.

In addition to the global benefits of reduced

GHGs, the cumulative electricity savings

have also reduced emissions of air pollut-

ants that contribute to smog and acidifica-

tion, including SOx emission reductions of

approximately 9.4 Kt and NOx emission

reductions of approximately 51 Kt.

The sales of energy-efficient products have

also reduced peak demand by over 1,000

MW and produced a range of financial

benefits.

5.8 Lessons Learned

• Consumer and investor knowledge and

confidence in energy efficiency products

are key objectives for an effective DSM

program. Consumers and investors are

motivated by cost savings. This motivation

requires knowledge that is strong enough

to allow long-term energy cost savings

calculations to play an important factor

in investment decisions.

Thailand Promotion of Electricity Energy Efficiency Project

31

• DSM programs must have a strong public

awareness campaign, in part to build pub-

lic support for the policies and investments

needed to support the programs.

• A program should be designed such that

it can get “early wins” that then lead to

expanded programs. This may mean tar-

geting residential markets (they are often

easier to deal with than commercial or in-

dustrial ones), technologies with relatively

low incremental costs (such as fridges, air

conditioners, tube lighting), and programs

that do not require collaboration with oth-

er departments or agencies (the assistance

of distribution utilities, for example).

• It is important to align programs with the

interests of the key players:

• From electrical generation/utility’s

perspective: DSM can delay costly

construction of added baseline capac-

ity and peaking plants. DSM can also

bring substantial benefits for “corporate

image” if and when the programs have

the necessary outreach, educational, and

campaign components.

• From distribution utility perspective:

DSM can provide a business opportunity

when ESCos are set up. DSM also can

assist in planning and cost avoidance

of system expansion and can improve

consumer relations by providing added-

value programs.

• From consumer’s perspective: DSM

can provide regular and long-term cost-

saving opportunities through reduced

power use and lower bills.

• From manufacturer’s perspective: DSM

provides an opportunity for increased

competitiveness in domestic and inter-

national markets by creating a critical

mass of demand.

• For government: DSM allows environ-

mental improvements, fulfillment of

international environmental obligations,

and a more competitive economy. DSM

can also lead to foreign investment op-

portunities (for instance, through the

Clean Development Mechanism).

• It is important to design a robust and

flexible program that can be adapted to

changes such as economic shocks, deregu-

lation, and so on. An adaptive manage-

ment approach is essential.

• Voluntary programs are useful as a neces-

sary step toward the development of mini-

mum standards. This is relevant to both

low and high incremental cost products.

To sustain energy efficiency achievements,

however, mandatory standards may be

needed.

• DSM programs need very strong leader-

ship to get into the mainstream. EGAT’s

experience suggests that this leadership

is required at the most senior levels.

33

This Chapter deals with the fourth case study:

the Jamaica Demand Side Management Dem-

onstration Project. Marbek Resource Consul-

tants conducted the study in association with

Angelhoeve Associates Incorporated. A field

mission to Kingston was conducted February

13—18, 2005.

6.1 Introduction

The Jamaica Demand Side Management

Demonstration Project was approved for

implementation by the World Bank with

cofinancing from the Global Environment

Facility in June 1994 and was completed in

December 1999. The total cost was $9.85

million, with the Bank providing $3.8 mil-

lion. The project involved the promotion of

compact fluorescent lamps and solar water

heaters (SWHs) in the residential sector, as

well as a number of other technologies in the

commercial and industrial sectors of Jamaica.

The project was implemented by the De-

mand-Side Management Unit of the Jamaica

Public Service Company Limited (JPS).

6.2 Framework and Approach

The study assessed the success, sustainability,

and attribution of the following outcomes

and impacts:

• Project Outcomes: demonstration of

technical and financial feasibility, increase

in institutional capacity, development of

replicable model

• Intermediate Outcomes: replication of the

model, additional increase in institutional

capacity, consumer preference influenced,

improvement in capacity and confidence

of distributors and retailers

• Ultimate Outcomes: transformation of

Jamaican energy efficiency markets

• Impact: energy savings, GHG reduc-

tions, reductions in air pollutants of lo-

cal concern, capacity savings, financial

benefits

To identify and attribute impacts, three sce-

narios were developed: one representing

actual and forecast sales of energy-efficient

6 JAMAICA DEMAND-SIDE MANAGEMENT DEMONSTRATION PROJECT

34

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

equipment with JDSMDP and the other two

representing low and high ranges of how

sales would have evolved in the absence of

JDSMDP.

6.3 Project Outcomes

In addition to the direct impacts of sales

during the program, JDSMDP attempted

to demonstrate the technical and financial

feasibility of the technologies and to es-

tablish an approach and model for future

programs.

The residential program was relatively suc-

cessful and led to substantial sales of CFLs

and SWHs. The key features of this model

were bulk purchases, sales in local JPS of-

fices, installment payments on the electric

bill, and subsidized prices. The commercial

and industrial programs were not successful,

mainly due to lack of financing.

Institutional capacity for energy efficiency

was established in the form of trained experts

and the establishment of a revolving fund

within JPS.

6.4 Intermediate Outcomes

A key factor in the sustainability of the

project outcomes and the achievement of

intermediate and ultimate outcomes was

the 2001 privatization of JPS. Following

this, the energy efficiency mandate was

transferred to the Petroleum Corporation

of Jamaica (PCJ), a state-owned enterprise

that implements programs on behalf of the

Ministry of Commerce, Science and Tech-

nology (MCST).

Our assessment concluded that the most

successful feature of JDSMDP was the sales

of CFLs. Replication of this sales program

has resulted in large-scale acceptance of

the product and its benefits by the general

public. The program begun by the project

was replicated first by PCJ and then grew to

the point where CFLs are available today to

consumers in most retail outlets throughout

the island. Sales of SWHs were also relatively

successful and replicated by PCJ. None of

the other JDSMDP programs have been

replicated. This is partly due to the lessons

learned during JDSMDP and partly due to

the lack of incentives, which results from the

policy vacuum created by the privatization of

JPS. There has been no known replication of

JDSMDP programs outside Jamaica.

Public awareness of energy efficiency grew

greatly as a result of the project. Thanks to the

JDSMDP and the subsequent PCJ/MCST En-

ergy Efficiency and Conservation Programme

(EECP) in 2001, consumers are generally

aware of the energy savings potential of CFLs

and SWHs and have confidence in the technol-

ogy. Even with the energy savings potential,

however, initial cost is a significant barrier for

most investments other than CFLs. While there

is a high level of awareness within industry of

the medium- and longer-term benefits of EE

technologies, short-term financial consider-

ations are preventing their adoption.

Jamaica Demand-side Management Demonstration Project

35

The project resulted in the development of

significant institutional capacity for energy

efficiency within JPS. Some of this capacity

is still found there, but it is underused due to

the utility’s focus on energy sales, not energy

conservation. Little capacity was transferred

from JPS to the government at the close of the

project; as a result, most of it was lost. Capac-

ity in some specialized areas was developed

in agencies such as the Bureau of Standards

Jamaica and the National Environmental

Protection Agency, and this capacity has

survived.

With respect to the industry, there are now

many CFL distributors and retailers and

several SWH distributors, most of whom got

their start during the time of the JDSMDP.

But limited capacity was developed within

the energy service industry.

6.5 Ultimate Outcomes

The key outcome has been transformation

of the Jamaican residential lighting market

(and, to a lesser extent, the domestic water

heater market). When JDSMDP began,

CFLs were expensive and hard to find; sales

volumes were very low. Today they are

affordable, they are visible in most retail

outlets, and they have a growing share of

the market.

• Prices for CFLs have dropped from over

$14 to less than $4. JDSMDP played a

moderate role in this change. Prices for

SWHs have stayed relatively constant at

approximately $1,200 for a typical 50-

gallon residential unit and have not been

significantly influenced by the project.

• Both CFLs and SWHs are readily available,

largely due to the project.

• Annual CFL sales in Jamaica have risen

from approximately 5,000 prior to 1995

to 85,000 in 2004 and are expected to

reach some 130,000 by 2010. Annual

sales of SWHs have doubled from approxi-

mately 400 in 1995 to 800 in 2004 and

are expected to surpass 1,200 by 2010.

The increase in sales was driven primarily

by higher electricity costs but also by the

greater availability, consumer awareness

and confidence, and lower relative prices

for the technology. JDSMDP had a sig-

nificant impact on most of these factors,

except for electricity prices.

JDSMDP has not had a significant impact on

the mainstreaming of energy efficiency or

global environmental issues in the policies

of the government of Jamaica.

6.6 Sustainability

The greatest level of sustainability can be

seen with the CFL component of the proj-

ect. Sales and use of CFLs have become

common in Jamaica residential and com-

mercial environments. To a lesser extent,

SWH sales seem sustainable, although at

much lower levels than CFLs. Initiatives to

promote sales of photovoltaic cells were

not sustained.

36

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

There is little evidence of sustainability with

regard to institutional capacity, particularly

with respect to commercial and industrial

energy audits. This can be traced directly to

the effects of the privatization of the Jamaica

Public Service Company. The project had

been focused on the utility, and it was as-

sumed that the company would carry on with

the project activities at its completion. This

did not happen, as the newly privatized orga-

nization reengineered to concentrate on mar-

keting electricity. Some limited institutional

sustainability can be seen in the work of the

Energy Efficiency Unit at PCJ. Unfortunately,

the unit did not benefit from the JDSMDP

infrastructure and corporate memory that was

left at project completion.

Since the end of the project, the policy envi-

ronment has not facilitated market transfor-

mation. With the privatization of JPS, there

was no entity that could provide incentives

(such as lower tariffs or low-cost financing) to

purchase EE technologies or undertake retrofit

projects. The market transformation that did

take place in the CFL market was mainly

due to the relative low cost measures and

the public awareness generated by JDSMDP.

Although PCJ followed up with the EECP,

energy efficiency has not been a high priority

for the government of Jamaica.

A supply channel for distribution of EE tech-

nologies was established during JDSMDP

and continues today (particularly for CFLs

and SWHs).

6.7 Impacts

The sales of all CFLS and SWHs (including non-

JDSMDP sources) and the associated increase

in stock are estimated to have produced energy

savings of approximately 28 GWh in 2004 and

associated GHG emission reductions of 34 Kt.

Over the period 1995—2004, GHG emission re-

ductions of 176 Kt were achieved and a further

reduction of 325 Kt is expected to 2010.

The following range of impacts is attributed

specifically to the JDSMDP:

• Energy savings of 12.5—18.4 GWh in

2004

• GHG emission reductions of 14.8—22.1

Kt in 2004

• GHG emission reductions of 100—120

Kt between 1995 and 2004 and a further

125—175 Kt by 2010

• NOx emission reductions of 625—750

tons between 1995 and 2004 and a fur-

ther 790—1,100 tons by 2010

• SO2 emission reductions of 375—450 tons

between 1995 and 2004 and a further

475—655 tons by 2010

• Capacity savings of 4—8 MW in 2003 (the

last year of capacity constraint)

• Total resource net benefits of $4.—4.6

million in the period 1995—2010 (approxi-

mately half the original investment)

• Total net benefits to consumer participants

of $6—7.6 million in the period 1995—

2010

• Total net costs to the utility of $2.1—3.4

million in the period 1995—2010

Jamaica Demand-side Management Demonstration Project

37

6.8 Lessons Learned

Our assessment suggests a number of lessons

to be applied both in future programs and,

more broadly, in developing approaches to

energy efficiency:

• Privatized utilities cannot be expected to

implement DSM programs unless there are

regulatory incentives. Other government

agencies can implement DSM programs,

but they need appropriate resources and

policy support. Privatizations should not

go ahead without addressing these issues.

• Subsidies can be effective in jump-start-

ing markets for EE technologies. The sale

of EE technologies through established

service outlets also provides an important

incentive.

• Not all EE opportunities will be viable.

By pursuing modest pilot programs, it is

possible to identify which are most likely

to be successful in wider applications.

• The availability of financing is a major

barrier in countries like Jamaica. Programs

that depend on significant capital invest-

ments need to be designed to overcome

this barrier.

• Public education is an important aspect of

the success of DSM programs.

• Although most consumers are able to in-

vest in low-cost, short payback measures

(such as CFLs), most of them (whether

residential, commercial, or industrial)

need additional assistance (such as low-

cost financing) for larger investments.

• Capacity development efforts need to

consider the possibility of institutional

changes, such as privatization. This

requires a broad reach and a risk man-

agement strategy to ensure capacity is

preserved under all plausible scenarios.

• It is important to align program objec-

tives with organizational objectives. In

this case, the objectives of JPS ultimately

diverged from those of the program, and

the capacity was lost.

• Ultimately, the government has the

greatest influence on the development of

energy efficiency policy. Consequently,

government entities (ministries and agen-

cies) should be primary targets for capacity

building.

This section presents cross-cutting observa-

tions based on the results of the four studies.

39

7.1 Overview

All four projects had objectives of developing

DSM capacity, reducing electricity consump-

tion and GHG emissions, and developing

technical and financial program models that

could be replicated.

The results were as follows:

• Major market transformation in the resi-

dential sector, primarily with respect to

lighting, but also including refrigerators

and air conditioners (in Thailand)

• No significant transformation in the insti-

tutional, commercial, or industrial sectors,

despite having targeted those sectors in

Thailand and Jamaica

• Significant and sustainable energy savings

and GHG emission reductions associated

with the transformation of the residential

markets

• Significant program replication and exten-

sion, both in the countries themselves and

in neighboring countries

• Some development of capacity for DSM

and energy efficiency within government

institutions but with moderate to signifi-

cant gaps remaining

• Significant benefits for consumers in terms of

cost savings and improved product quality

• Significantly enhanced opportunities for

distributors and retailers of energy-efficient

equipment

• Significantly improved competitiveness of

manufacturers (in Thailand only)

• Small contribution to the integration of

energy efficiency objectives into energy

policies

• Minimal to modest contribution to the

mainstreaming of global environmental

issues into energy policies

• Modest contribution to the develop-

ment of procedures and tools needed

for global flexibility mechanisms such

as CDM/JI

Key features and impacts of the four projects

are summarized in Table 7.1. As shown, three

7 REVIEW OF CROSS-CUTTING OBSERVATIONS

40

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

of the four projects were implemented by the

countries’ publicly owned power generating

utilities, with the exception being Poland. All

four projects took place in the mid-1990s, at

a time when the Bank/GEF favored support

for DSM projects. (Since then, DSM projects

have generally been rejected in favor of mar-

ket transformation projects.) On a cost-per-

ton basis, three of the projects were clearly

successful; the exception was Jamaica.

7.2 Design Features

The key design features of each of the projects

are summarized in Table 7.2.

Key observations are as follows:

• Two projects had a narrow focus on resi-

dential lighting, and the other two were

broadly aimed at energy efficiency in all

sectors. All four ended up focusing on the

residential sector.

Table 7.1 Overview of Project Features and Impacts

Aspect PELP ILUMEX JDSMDP TPEEE

Implementation Netherlands EE CFE (main electric JPS (main electric EGAT (main Entity Lighting utility) utility – now privatized) electric utility)

Timing 1995–98 1995–98 1994–99 1993–2000

World Bank/GEF $5 million $10 million $3.8 million $9.5 million Contribution

Total Program Cost $5 million $23 million $9.85 million $59.3 million

GHG Reductions 0.53 Mt 0.76 Mt 0.014 Mt 5 Mt (Direct) ($10/t) ($30/t) ($700/t) ($12/t)

GHG Reductions 3.6 Mt 5.3-9.5 Mt 0.2–0.3 Mt 27–45 Mt (Total) ($1.40/t) ($2.40/t - $4.30/t) ($33–49/t) ($1.30–2.20/t)

Table 7.2 Key Design Features

Aspect PELP ILUMEX JDSMDP TPEEE

Initial Scope Residential Residential All All lighting lighting

Key Market Residential Residential Residential Residential T8s, CFLs, /Technology CFLs CFLs in 2 cities CFLs, SWHs refrigerators, A/C

Price Difference $6–1415 versus 20¢ Minimal Substantial (at Start) (T8s & Fridges) (A/C & CFLs)

Incentives MFC Subsidy 50% subsidy, 50% subsidy No Loans and (avg $2) plus financing small subsidies

Labeling No No No Fridges -yes Major aspect

Marketing Yes Some Yes, by NGOs Major Major aspect

Point of Sale Various retail Utility offices Utility offices Various retail Various retail

Other Competitive Bulk purchase with Bulk purchase Negotiations with Green learning process quality standards manufacturers rooms

Review of Cross-cutting Observations

41

• All four projects dealt with CFLs, where

the price differential was substantial. All

adopted an approach using subsidies,

though Thailand focused mainly on stan-

dards and labeling.

• Poland’s approach differed in that the

subsidy was directed to manufacturers and

was based on competitive proposals.

• Mexico and Jamaica’s approach involved

bulk purchases and sales through the util-

ity, whereas Poland and Thailand relied

on negotiations with manufacturers and

sales through normal retail networks.

• All four projects involved some level of

marketing, education, and outreach, but

Thailand focused on this more than the

others.

• Poland and Mexico focused all resources

on the residential lighting sector. Jamaica

and Thailand spread resources across

commercial and industrial sectors, with

relatively minimal results.

7.3 Project Outcomes

The main project outcomes are summarized

in Table 7.3.

Key observations are as follows:

• All four projects achieved substantial sales

and market penetration (Poland, Mexico,

Jamaica in CFLs, Thailand in T8s and

fridges).

• All four projects successfully demonstrated

the technical viability of the applicable

technologies and the financial viability for

consumers.

• The projects also highlighted the challenge

of making EE/DSM work for utilities:

• For state-owned utilities with a broad

mandate (Mexico and Thailand, at the

Table 7.3 Project Outcomes

Aspect PELP ILUMEX JDSMDP TPEEE

Sales / Market 1.2 million 2.6 million 0.1 million T8s: increased from Penetration CFLs CFLs CFLs 40 to 100% market penetration Fridges: increased from to 2 to 84% A/C: increased from 0 to 38%

Demonstration Technical Yes Yes Yes for CFLs and Yes Value Feasibility SWHs; no for others

Program Yes Yes Yes for CFLs and Residential and some Feasibility SWHs; no for commercial others

Financial Partly Partly No (utility Residential and some Feasibility (depended (depended not willing to commercial initially on initially on sustain–financing subsidy) subsidy) not available)

Capacity Building Utilities National utility, Utility, NGOs Utility some other government

42

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

time), there was acceptance of a negative

return.

• For privatized utilities (Jamaica, Thailand

shortly, and Poland), it became clear that

there is no incentive for EE/DSM.

• The need for a subsidy reduced financial

attractiveness for utilities, but projects

demonstrated that on a total resource

cost (TRC) basis, even these could be

justified.

• All four projects provided program models

that could be replicated or extended.

7.4 Post-Project Context

The key features of each country’s post-proj-

ect context are summarized in Table 7.4.

Key observations are as follows:

• Mexico and Thailand suffered major

economic shocks while Jamaica suffered

steady decline. Poland had significant

growth during the period.

• Poland and Jamaica privatized substantial

parts of the electricity system; Thailand is

currently planning significant privatiza-

tion; Mexico is not currently planning to

privatize.

• Poland had excess electricity generation

capacity throughout the period; Mexico

and Jamaica were constrained until re-

cently; Thailand remains constrained.

• Jamaica is the only one among the four

that is a “developing country.” Poland

and Mexico are members of the Or-

ganisation for Economic Co-operation

and Development, and Thailand is an

emerging economy. As a result, Jamaica

faced significant limitations in terms of

absorptive capacity and institutional

capacity, which probably affected the

project results significantly.

7.5 Intermediate Outcomes

The main intermediate outcomes are sum-

marized in Table 7.5.

Key observations are as follows:

• Substantial replication has taken place

both domestically and internationally in

three cases (but not in Jamaica’s case).

• Substantial capacity building has taken

place in utilities (Thailand and Mexico);

however, Jamaican capacity was lost due

to privatization. Some capacity has been

Table 7.4 Post-Project Context

Aspect PELP ILUMEX JDSMDP TPEEE

Economic Steady growth; 1995 devaluation Recession until 1997 devaluation Conditions focus on EU and recession 2000 and recession accession

Energy Partly privatized State-owned Privatized utility; Privatization is Context system; excess utility; excess excess capacity planned; capacity capacity capacity from 2002 from 2003 constraints

Review of Cross-cutting Observations

43

built and sustained within government

agencies, NGOs, and universities and

among other professionals.

• There has been a substantial increase in

consumer awareness, confidence, and

knowledge of financial advantages in

relation to residential lighting for all four

projects. In Thailand, this awareness is

broader and extends to refrigerators and

air conditioners.

• Distribution and retail capacity were greatly

increased in all four projects; however, only

Thailand succeeded in improving the com-

petitiveness of domestic manufacturing.

7.6 Ultimate Outcomes

The main ultimate outcomes are summarized

in Table 7.6.

Key observations are as follows:

• Energy-efficient lighting is now widely

available in all four countries (this outcome

is strongly attributed to the projects).

Table 7.5 Intermediate Outcomes

Aspect PELP ILUMEX JDSMDP TPEEE

Replication Domestic Municipal projects; Extension to No Extension of and minor commercial national program program, and Extension and institutional (FIDE) and expansion of programs; Mexico City labeling; Efficient Motors (LyFC) adoption of Program (2004) Minimum Energy Performance Standards

Foreign Efficient Lighting Efficient Lighting Model adopted Initiative Initiative by several other countries in the region

Institutional Capacity- Distribution utilities; National utility Utility privatized Utility Building universities but and other — capacity lost not government government

Consumers Greater awareness, Greater Greater Greater confidence, and awareness, awareness, awareness, knowledge of confidence confidence, and acceptance financial advantage and knowledge and knowledge of labeling of financial of financial for residential advantage advantage products, meaning financial advantage and product quality

Trade Allies Did not develop Developed strong Developed strong Developed local manufacturing; distribution and distribution and significant did develop retail capacity retail capacity manufacturing distribution and capacity plus retail capacity strong distribution and retail capacity

44

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

• Energy-efficient lighting is much cheaper

in all four countries; however, this is

mainly due to global supply and demand

factors as opposed to the projects.

• Sales of energy-efficient lighting in Poland

and Thailand are reaching market satura-

tion, whereas in Mexico and Jamaica CFLs

are at about 50 percent market saturation.

The projects are estimated to have acceler-

ated CFL sales by four to seven years.

• The effect of the projects on the main-

streaming of global environmental con-

siderations into domestic policy has

been minimal; however, ILUMEX made

Table 7.6 Ultimate Outcomes

Aspect PELP ILUMEX JDSMDP TPEEE

Availability CFLs now CFLs now CFLs now T8s, CFLs, label #5 widespread widespread widespread fridges, and A/C (strong attribution (strong attribution are widespread to project) to project) (strong attribution to project)

Price Order of Order of Order of Some effect magnitude lower magnitude lower magnitude lower (not documented) (minimal attribution) (minimal attribution)

2004 2.8 CFLs/ 1.1 CFLs/ 0.8 CFLs/ T8s: 100% of market Market household household household CFLs: 500% increase Penetration (1.1 due to project) (0.5–0.9 due to (0.4–0.6 due to in sales project) project) Level #5 Fridges: 100% increase in sales Level #5 A/C: 50% increase in sales

Influence on Minimal Minimal inside; Contribution to Move to Minimum Government Mexico’s potential EE Energy Performance Policy Contribution to Program currently Standards CDM/JI proposed

Table 7.7 Annual Impacts (average of scenarios)

Aspect PELP ILUMEX JDSMDP TPEEE

Energy Savings (GWh) 580 980 9.2 3,500

Capacity Savings (MW) 480 10 600

GHG Emission 0.68 0.58 0.01 2.5 Reductions (Mt)

GHG Reductions 0.6% 0.7% 0.5% 3.5% (share of electricity sector)

SO2 Emission

Reductions (Kt) 11.20 0.04 1.2

NOx Emission Reductions (Kt) 1.80 0.07 6.3

TRC Benefit-Cost Ratio 5.1 (2010) 0.4 (2010) 1.9 (2000)

Review of Cross-cutting Observations

45

a significant contribution to the evolution

of the CDM/JI concept.

7.7 Impacts

The main impacts are summarized in Table

7.7.

Key observations are as follows:

• Energy savings and corresponding GHG re-

ductions have been sustained and expanded

upon since the end of the projects.

• Annual reductions throughout the analysis

period were in the range of 0.5 percent

of electricity sector emissions (except in

Thailand, where they were 3.5 percent).

• There are corresponding reductions in

SO2 and NOx emissions, which is of local

environmental interest.

• Program costs are in the range of $1—5

per ton of ultimate reduction (except in

the case of Jamaica, at $40 per ton).

• The benefit-cost ratio was in the range of

2 to 5, except for Jamaica at 0.4.

7.8 Conclusions

Although DSM projects are seen as less attrac-

tive than market transformation projects by

some economic purists, the analysis indicates

that they can be successful in generating mar-

ket transformation over the short and medium

term. Furthermore, they can produce highly

cost-effective results. The key lesson is that

there is a role both for direct intervention

and for longer-term information and trans-

formation approaches. And it seems clear

that drawing artificial distinctions between

DSM approaches and market transformation

approaches can be counterproductive: DSM

can help to transform markets, but that cannot

happen without the information and outreach

components of what are often labeled “mar-

ket transformation programs.”

During the analysis period, the residential

sector was the easiest sector to target for

energy efficiency investments. This may

be due to a number of factors, including

more homogeneous and simpler markets

that were easier to target, the focus on only

one or a few technologies, the fact that the

decision maker and financial manager are

the same person, the longer time horizon of

homeowners versus business managers, and

the fact that the upfront investment could be

more modest. The fact that the residential

opportunities could be realized during this

period reflects the particular circumstances in

those countries (that is, there was a particular

technological opportunity – such as CFLs

– that could be realized).

Residential lighting, in particular, was a

good end use to target. This was because the

technical concerns had been resolved, light-

ing represented a large portion of end use in

the residential sector, and the barriers were

relatively well understood. In addition, the

technology was poised to deliver dramatic

cost reductions (although the scale of the im-

pending cost reductions would not have been

known when the projects were initiated). In

46

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

general, simple programs focused on straight-

forward opportunities, and single technologies

seem to have a greater rate of success.

Barriers to the widespread adoption of energy

efficiency technologies in other sectors (insti-

tutional, commercial, and industrial) proved

to be insurmountable. Although programs

elsewhere may have had greater success, the

lack of financing was an important barrier in

the projects reviewed for this study.

The programs that worked best involved

tangible interventions that affected invest-

ment decisions (such as through price,

availability, or quality). Education and mar-

keting programs worked well in support of

other programs but were not sufficient on

their own to generate significant changes in

behavior.

Utilities’ interest in DSM depended on their

mandate and the policy environment. In gen-

eral, utilities are unlikely to have an interest

if they are privatized or driven by revenues.

Those with a broad mandate to promote the

public interest are more likely to promote

DSM, particularly if they wish to maintain a

good public image and to offer consumers

energy-saving opportunities to mitigate the

economic impact of rate increases. Alterna-

tively, financial and/or regulatory incentives

may make it worthwhile for utilities to imple-

ment DSM. Ultimately, a utility’s interest

in DSM depends on the vision of its senior

leadership.

47

This Chapter outlines a number of lessons

and recommendations that emerge from

the experience of the four projects. These

should be viewed in perspective, however,

as the experiences in question took place in

the 1990s, and several factors – including

technologies, prices, and policy environ-

ments – have changed. Nevertheless, most

of these lessons are considered still broadly

applicable, provided that sufficient attention

is paid to local circumstances.

8.1 Project Identification

Lessons Learned:

The residential sector and the lighting seg-

ment in particular provided the best opportu-

nities for investment. For the most part, how-

ever, the projects only affected a very small

proportion of the electricity demand in each

country (let alone broader energy use). This

suggests two possible paths to take in identi-

fying future energy efficiency projects: target

market segments that are the most promising

or target the largest electricity users.

Residential programs also have the advan-

tages of targeting decision makers more di-

rectly and, at the same time, helping to build

the awareness and understanding needed to

support broader energy efficiency policies

and measures.

Recommendation: Choosing the best

projects requires good research. Barriers

to implementation should be assessed

realistically and the goals must be both

significant and achievable. In most cases,

projects should focus on particular market

segments rather than attempt too many dif-

ferent interventions. The most cost-effec-

tive opportunities should be pursued first,

but with the ultimate goal of building up to

the most significant opportunities.

Lessons Learned:

Projects work only when the incentives for

the implementing entity and the funding

agencies are aligned. This means taking into

account the energy context – in particular,

8 LESSONS LEARNED AND RECOMMENDATIONS FOR THE ENERGY EFFICIENCY PORTFOLIO

48

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

the status (privatized or not), mandate, and

incentive structure for the electricity utili-

ties – as well as the balance of supply and

demand, including the potential for load

management.

For utilities, DSM benefits may include the

opportunity to delay costly construction of

added baseline capacity and peaking plants,

as well as benefits for “corporate image.”

However, incentives for privatized utilities

are unlikely to be sufficient in the absence

of regulatory or fiscal incentives.

DSM can provide a business opportunity for

organizations other than the utility, such as

energy service companies.

Promoting the development and use of

ESCos is a particularly challenging exer-

cise that requires significant investment,

analysis of a range of considerations (such as

technical, financial, marketing, and policy

issues), and a long-term planning horizon

or outlook.

Recommendation: Projects should be

chosen to fit within both the energy policy

goals of the country and the business plan

(and power development plan) of the

implementing entity. Projects involving

utilities should be considered, but the

incentives need to be clear, and the risk

of potential changes in ownership and

regulatory framework need to be assessed

and managed. Other government agencies

or ESCos should also be considered as

potential implementing agencies.

Recommendation: ESCo programs should

not be attempted unless significant analyti-

cal and financial resources are available.

Consider these as standalone projects in

their own right. Expectations should be

long-term.

Lesson Learned:

In many countries, it may not be realistic to

expect energy efficiency projects to result in

significant mainstreaming of global environ-

mental issues into national policies.

Recommendation: Avoid overload-

ing projects with too many objectives.

Global environmental objectives can be

advanced without having them become

central to energy policies.

8.2 Project Preparation

DSM Programs versus Market

Transformation

Lesson Learned:

The experience of the four projects indicates

that traditional DSM project designs can be

effective in promoting market transformation

provided price differentials for energy-effi-

cient equipment are reasonably small or the

technology is maturing and there is a reason-

able expectation that price differentials will

drop. Under these circumstances, programs

do not need to be maintained indefinitely.

Lessons Learned And Recommendations For The Energy Efficiency Portfolio

49

Recommendation: Reconsider the current

bias against DSM projects implemented

by utilities, but design programs to have

clear exit strategies. Focus on remov-

ing barriers (including poor awareness,

capacity, availability, and price) or on

moving toward regulation (energy ef-

ficiency standards).

Recommendation: The price situation and

outlook should be assessed in detail prior

to designing market interventions. The

World Bank should realistically assess the

project’s market power.

Pilot Programs

Lesson Learned:

Projects succeeded when they were able to

achieve successes quickly and when the im-

plementing agency built on them to achieve

a critical mass of market change.

Recommendation: Ensure resources are

sufficient to achieve a large presence in

the target market. If resources are not

sufficient, then consider a smaller market

(a particular segment or a region). Pilot

programs in selected markets are a good

way to achieve significant market pres-

ence at a reasonable cost. The success

can then be replicated and exported to

other markets.

Lesson Learned:

Not all EE opportunities will be viable.

Recommendation: By pursuing modest

pilot programs, it is possible to identify

which are most likely to be successful in

wider application.

Subsidies versus Other Mechanisms

Lesson Learned:

Subsidies may be necessary when an efficient

technology has a significant price premium,

and they can even be sustained provided

mechanisms are in place to raise the funds from

an appropriate source (such as a carbon tax or

a surcharge on less-efficient energy uses). If the

market is truly transformed, however, it should

be possible to phase out subsidies over time.

Recommendation: Price barriers should be

assessed to determine if they are significant

and if they are likely to be reduced through

increased market volume. Subsidies

should be considered if they are likely to

be temporary or if their benefits are greater

than the cost.

Recommendation: If subsidies are to be

used, all options should be explored to

find the most cost-effective approach that

causes the least distortions to the market.

(The PELP approach, for example, was to

seek competitive proposals from manu-

facturers and distributors and to select the

proposal that required the least subsidy.)

Bulk purchases can help create a criti-

cal mass of demand to reduce costs and

stimulate the market.

50

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

Recommendation: If market transforma-

tion is the goal, design the program to

phase subsidies out gradually over time.

Lesson Learned:

Under the right conditions (such as a small

price difference), other direct approaches

(labeling, for instance) can work.

Recommendation: Conduct market

studies to reveal the key barriers and op-

portunities. Possibilities include labeling,

convenient sales points or payment meth-

ods (through utility bills, for instance),

availability of financing, and so on.

Lesson Learned:

With a small number of manufacturers, many

program options are possible. (TPEEE, for

example, undertook negotiated agreements

with manufacturers to eliminate non-energy-

efficient lighting from the marketplace and

substitute efficient units; this was feasible

because the price differential was low and

the number of manufacturers was small.)

Recommendation: Involve trade allies in

program design and implementation.

8.3 Public Education and Outreach

Lesson Learned:

The increase in consumer awareness and ap-

preciation of the benefits of energy-efficient

technologies was one of the main intermediate

outcomes of the projects. The key messages

were that the technology was available, it was

reliable and of good quality, and, most impor-

tant, it provided significant overall financial

benefits within a short time frame.

Recommendation: Projects should in-

clude a strong public education and

outreach component, including detailed

information on cost savings.

Lesson Learned:

Public education, outreach, and marketing

are important to support other programs but

are not likely to be successful on their own.

They must achieve a significant profile and

be sustained to be effective.

Recommendation: Link education and

awareness programs to subsidies, labels,

or other concrete programs. Invest suffi-

cient resources or concentrate on smaller

market segments.

Lesson Learned:

To a large extent, energy efficiency programs

rely on public support to sustain the investment

or the policies needed to make them work

Recommendation: Strategies should also

include a strong campaign to increase

public awareness and support for energy

efficiency.

8.4 Capacity Building

Lesson Learned:

Ultimately, government has the greatest influ-

ence on the development of energy efficiency

Lessons Learned And Recommendations For The Energy Efficiency Portfolio

51

policy. To be effective and sustainable,

however, programs need a broad reach that

includes all key stakeholders.

Recommendation: Government entities

(ministries and agencies) should be pri-

mary targets for capacity building. Capacity

building should be an integral part of all

project components.

Lesson Learned:

Local staff need to develop the capacity to

plan, implement, monitor, evaluate, and

report on programs and to pool adequate

human and financial resources. They need to

develop the capacity to design and enforce

regulations.

Recommendation: Provide training, ex-

amples, and coaching to help develop

needed skills.

Lesson Learned:

Capacity development efforts need to con-

sider the possibility of institutional changes

such as privatization.

Recommendation: Build capacity broadly

within related organizations and develop

a risk management strategy to ensure

capacity is preserved under all plausible

scenarios.

Lesson Learned:

NGOs have an important role to play

in raising awareness, advocating for the

public good, and providing overall over-

sight of government and utility plans and

policies.

Recommendation: Special attention

should be paid to building and sustaining

the capacity of NGOs and other organiza-

tions to raise awareness on EE/DSM issues

and benefits at various levels.

Lesson Learned:

Ultimately, it is the capacity and confidence

of trade allies, including manufacturers,

distributors, retailers, and professionals, that

will determine the extent of market transfor-

mation. Their capacity and confidence will in

turn depend on the ability of the projects to

generate enough momentum to justify their

investment of resources.

Recommendation: Design projects to gen-

erate quick successes and a critical mass

of demand sufficient to change attitudes

fundamentally.

8.5 Project Implementation

Lesson Learned:

Projects have a greater chance of success

if they are supported by strong champions

who have a long-term view of market trans-

formation.

Recommendation: Assess leadership

capacity within the host organization (in-

cluding potential succession) and ensure

that key senior managers and decision

52

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

makers are aware of and enthusiastically

support the main strategies.

Lesson Learned:

Many events can affect implementation as well

as project outcomes (such as economic shocks,

utility deregulation, and privatization).

Recommendation: Projects require a risk

management strategy and contingency

plans for major contextual changes (for

instance, avoid focusing all capacity

building on one entity). A staged approach

can help but, most important, the imple-

mentation entity must have (or be able

to acquire) the capacity for adaptive

management.

Recommendation: Incorporate a moni-

toring and evaluation (M&E) component

to provide the information necessary to

adjust project parameters in response to

changing circumstances and results. Note:

the monitoring and evaluation informa-

tion will also assist the conduct of future

impact studies (see Chapter 9). Indepen-

dent evaluation should be commissioned

to support claims of success and requests

for program extensions.

Lesson Learned:

The development of strong distribution net-

works and sales outlets is important. Both

public and private models have the potential

to succeed.

Recommendation: Consider factors such

as capacity, convenience, trust, visibility,

and overhead before selecting an ap-

proach. The use of additional incentives

such as financing and integrated billing

may also determine the choice.

Lesson Learned:

Consumer and trade ally confidence is crucial

and is very difficult to regain if lost.

Recommendation: In the early stages,

quality standards should be set high

enough to avoid any chance of customer

dissatisfaction. This may require investing

more than the technical optimum level

of resources. Bulk purchases provide a

means of imposing higher standards at

lowest cost. Resources should be devoted

to quality assurance.

Recommendation: To ensure accuracy

and reliability, labeling programs require

a dependable quality assurance compo-

nent, with random testing and complaint

investigation.

53

This Chapter presents the methodology

lessons learned from undertaking the four

impact assessments.

9.1 Planning for the Assessment

Lesson Learned:

Impact assessments are greatly facilitated

when information is collected throughout

the project life cycle and an appropriate

evaluation framework is established from

the beginning.

Recommendation: All projects should

include a monitoring and evaluation

framework that includes a log frame/

impact map, as well as indicators for

measurement of progress on outcomes.

The framework should extend beyond

the project completion and establish a

projected baseline of information, as well

as mechanisms and templates to collect

post-implementation data. Note: this infor-

mation is also essential to implement an

adaptive management approach.

Lesson Learned:

There is substantial confusion (particularly in

recipient countries) about the various forms of

audits, completion reports, assessments, and

evaluations. There is often a sense that projects

are overanalyzed, and there is a sensitivity to

potential criticism concerning government

policies and management following projects.

Recommendation: Ensure the distinction

between different assessments is clearly

stated in the M&E framework. Use the

term “Post-Implementation Impact As-

sessment” to differentiate these from

evaluations.

Lesson Learned:

Project information and ongoing monitoring

information can be easily lost during staff

changes at the World Bank or in the imple-

menting entity.

Recommendation: Develop a robust in-

formation management plan to safeguard

9 LESSONS LEARNED AND RECOMMENDATIONS ON IMPACT ASSESSMENT

54

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

information and ensure it is available for

subsequent studies. This should be “insti-

tutionalized.”

Lesson Learned:

Log frames/impact maps are at the core of

an impact study and are a key tool to estab-

lish the impact storyline. Even if the results

framework was relatively clear at the time of

project implementation, the story is likely to

have changed during the intervening years.

Recommendation: Invest resources to

develop a detailed map (updating the

initial project log frame as appropriate).

Be prepared to produce several iterations

as information is collected and the “story”

emerges.

Lesson Learned:

Attribution of impacts is an extremely difficult

and uncertain exercise. There are conceptual

challenges (that is, defining the baseline) as

well as practical challenges in obtaining in-

formation and credible views on the relative

importance of the various factors that may

have influenced the outcomes. In particular,

it may be almost impossible to get a consen-

sus on the degree of influence and timing of

various factors.

Recommendation: Use multiple ap-

proaches to attribution, including charac-

terizing the importance of the various fac-

tors (high, medium, or low, for example)

and the development of counter-factual

scenarios. In cases where information is

very speculative, obtain a variety of views

and use them to develop scenarios.

9.2 Data Collection

Lesson Learned:

It is very difficult to obtain information on

actual events (such as time-series data on

sales) in some countries, let alone data for

counter-factual scenarios.

Recommendation: Obtain multiple opin-

ions and proxy data to attempt to construct

realistic scenarios through extrapolation.

Recommendation: It is useful, where

possible, to reconvene members of the

original project steering committee as a

focus group.

Recommendation: If possible, commis-

sion specialized market studies to mea-

sure key indicators through surveys and

focus groups.

Lesson Learned:

Notwithstanding the general lack of time-

series information, much more information

is now available on the Internet, even in

developing countries.

Recommendation: Conduct a thorough

search of available information prior to

the field mission and adjust the interview

protocols accordingly. In particular, exam-

ine information available through Country

Lessons Learned and Recommendations on Impact Assessment

55

Communications in accordance with the

United Nations Framework Convention

on Climate Change, information held by

the International Energy Agency, and host-

country government Web sites.

Lesson Learned:

Identifying and gaining access to key infor-

mants is essential. This can be difficult, how-

ever, particularly without a local contact.

Recommendation: Ensure someone (prefer-

ably from the implementing organization)

is available and willing to act as sponsor.

(This will require a significant contribution

of time and effort on their part.) Arrange

interviews and meetings ahead of time with

their help and that of local Bank staff. A lo-

cal sectoral expert may also be useful. Plan

field missions to take into consideration

public holidays and the availability of key

stakeholders. Formal letters from the Bank

to key people in government departments

and utilities (at both the senior and the

middle management level) are particularly

important to secure cooperation.

Lesson Learned:

Field missions are essential opportunities to

collect opinions as well as documentation

that are hard to obtain otherwise. However,

it is easy for missions to be diverted from

core objectives.

Recommendation: Plan and arrange

meetings ahead of time; however,

keep schedules and planning flexible

to accommodate new information and

changes in client, stakeholder, and as-

sessment team agendas. Ensure that

as many required data as possible are

gathered during the field work, as it is

often difficult to obtain information after

leaving the field.

Recommendation: Develop detailed inter-

view protocols based on data gaps defined

by the indicators. Provide a summary of

the intended interview questions ahead of

time. (In most cases, a summary may be

preferable to a full version because many

questions will not apply to individual

informants and because a longer version

may be intimidating.)

9.3 Analysis

Lesson Learned:

The lack of certain data makes it difficult to

develop credible estimates of impact.

Recommendation: Develop scenarios that

reflect the upper and lower boundaries of

attributed impact.

Lesson Learned:

Understanding the context of the changes is

very important to drawing conclusions on

attribution.

Recommendation: Invest resources to

understand the macroeconomic and

policy contexts and how they changed

56

Post-implementation Impact Studies of World Bank — Gef Energy Efficiency Projects

in the period following completion of

the project.

Lesson Learned:

In addition to understanding the project and

country context, it is important that the analy-

sis be grounded in a thorough understanding

of the evolution of the World Bank’s own

approaches.

Recommendation: Ensure the study team

is thoroughly briefed on the current portfo-

lio of similar projects, the current thinking,

and the main development questions of

the day in the World Bank.

Lesson Learned:

The significance and value of the analysis is

greatly enhanced by a comparative study of

several projects.

Recommendation: Continue to under-

take evaluations of clusters of related

projects.

World Bank Global Environment Facility

Coordination Team Environment Department

THE WORLD BANK

1818 H Street, NW

Washington, D.C. 20433, USA

Telephone: 202.473.1816

Fax: 202.522.3256

Email: GEOnline@worldbank.org

Web: www.worldbank.org/gef