Crossing Methodological Borders to Develop and Implement an

Approach for Determining the Value of Energy Efficiency R&D

Programs

Crossing Methodological Borders to Develop and Implement an

Approach for Determining the Value of Energy Efficiency R&D

ProgramsPresented at the

American Evaluation Association/Canadian Evaluation Society Joint Conference

Toronto, Canada October 28, 2005

Scott Albert, GDS AssociatesHelen Kim, NYSERDA

Rick Ridge, Ridge & AssociatesGretchen B. Jordan, Sandia National Laboratory

2

The NYSERDA PortfolioThe NYSERDA PortfolioTotal New York Energy $mart Budget, By Program Area ($ millions)

Total 8-Year Budget = $961.8 million*

NYS Cost Recovery Fee

$9.0 0.9%

Research and Development

$210.8 21.9%

Business and Institutional

$359.1 37.3%

Residential (non Low-Income)

$170.717.7%

Low-Income $128.4 13.4%

Program Administration

$64.6 6.7%

Environmental Disclosure

$2.9 0.3%

Metrics and Evaluation $16.2 1.7%

Source: New York Energy $martSM P rogram - Financial Status Report. As of December 31, 2004.*Including Estimated Interest Earnings

3

R&D Budget Through 12/31/04R&D Budget Through 12/31/04

$13$11

$18$13 $13

$2

$32

$21

$60

$20 $21

$4

$52

$28

$67

$22

$30

$7

0

10

20

30

40

50

60

70

80

WholesaleRenewables

End-UseRenewables

DG-CHP EMEP NextGeneration

Power &Storage

Invoiced Committed Budgeted

4

ObjectiveObjective• Develop and pilot-test an evaluation model for NYSERDA’s

R&D program area covering 1998 through 2004 that recognizes:– R&D programs and their societal impacts are difficult to

evaluate by their nature.– The outcomes are subject to multiple and uncontrollable

influences that are difficult to foresee. – The cycle for product development is 5 to 15 or 20 years and

many of the energy and economic impacts of R&D projects may not be fully realized and measured for many years.

– Given the multiple and compounding effects that happen along the way, it is also very difficult to be exact about the attribution of impacts to any one particular effort.

– When evaluating an entire portfolio of R&D projects, objectives and outcomes vary by project.

5

R&D Portfolio Logic ModelR&D Portfolio Logic ModelNYSERDA Select & Manage R&D Projects to:-drive portfolio changes over time to respond to current needs, and-Provide public benefits

Develop new or improved product

Study, Prove Concepts

Demonstrate products, inform markets

Study to inform policy & R&D community

Test & improve products

Dissemination builds common knowledge base-Lab prototypes-Future R&D & product options

-Investment/interest growing-Commercial scale product developed -Potential demonstrated

Product proven/ introduced in market

Informed policies & programs;R&D opportunities & standards identified, publicized

- Data from tests- Establish standards- Hands on experience (industry)-Feedback to R&D

White papers, workshops;Policy-relevant research

-Intermediate scale prototypes- Performance/cost specifications improving

Producers, consumers,

policy makers see value

New knowledge:-papers, articles -data

- Data from tests in different context- Feedback to R&D & policy makers- Visibility & data from showcases

Inputs:Funds, staff, NYSERDA competencies, partnerships

Activities

Outcomes

Outputs

Policy and Product development and pre-deployment process (5-10 years)

Knowledge for future R&D and productsFirms have credibility & market infrastructure is supportive

Products manufactured as replacement, stand alone, or part of system and purchased by early adopters

External Influences:Cost, Performance of existing technologies; Industry willingness to take risks; Uncertainty of R&D; Energy prices; Government policies

Research for Policy DemonstrationProduct Development Pre-deployment

Educate, provide incentives to supply & delivery

-Training, certification-Production incentives-Innovative designs-Other barriers lowered

Business infrastructure supports

the product

Environmental benefits Energy benefits- generation, energy/load management, efficient use

Economic benefits -cost of compliance, NY jobs

- New- Accelerated- Expanded

6

Six Stages of the R&D ModelSix Stages of the R&D Model

• Information for policy makers and R&D community• Product development stage 1 – study and prove

concepts • Product development stage 2 – develop new or

improved products • Product development stage 3 – product testing • Demonstration• Pre-deployment

7

The Value/Cost MethodCombines Two Approaches

The Value/Cost MethodCombines Two Approaches

• Aggregate approach– Analyzed data collected for each of NYSERDA’s 638 R&D

projects (since 1998) in the portfolio.– Basic statistics, such as the number of projects, expenditures

by technology type, leveraged funds, and the stage of development were calculated to describe the entire R&D portfolio.

• Peer Review– Analyzed using an adaptation of the Composite Performance

Rating System (CPRS) used to evaluate the U.S. Department of Commerce’s Advanced Technologies Program (ATP).

– Peer review approach was applied to a small sample of successful R&D projects, covering each of the six R&D stages (project types).

8

ATP: Composite Performance Rating System Constructed Bottom-up; Used

Top-down

ATP: Composite Performance Rating System Constructed Bottom-up; Used

Top-down Performance Distribution for the Portfolio

Distributionby

Tech Area

Distributionby

Firm Size

Distributionby

Locationetc.

Project 1Case Study

Project 2Case Study

Project 3Case Study

Project 4Case Study

Project nCase Study

...

• Unique cases• Aggregate statistics• Composite scores• Performance distributions• Minimum net portfolio benefits

CPRS 1 CPRS 2 CPRS 3 CPRS 4 CPRS n

ATP MethodR.Ruegg, Nov. 2002

9

AGGREGATE ANALYSISAGGREGATE ANALYSIS

• Expanded and updated R&D database in order to carry out a comprehensive descriptive analysis of the entire R&D portfolio.

• Variables Considered– Funding– Technology Area– Co-Funding Entity– Project Status– Expected Benefits from R&D Projects

10

Questions Addressed by Aggregate Analysis

Questions Addressed by Aggregate Analysis

• How does NYSERDA funding per project vary by project type?

• How does NYSERDA funding per project vary by program?

• What is the frequency of the various project types?• What goals are being served by the various project

types?• What are the primary goals served by the portfolio?• What are the sources of funding, by project type?• What is the funding share contributed by partners?• How does NYSERDA funding and co-funding vary by

project type over time?• How does the mix of technologies and issues examined

change over time?

11

Results:Aggregate Analysis

Results:Aggregate Analysis

12

NYSERDA Funding, by Project Type

NYSERDA Funding, by Project Type

Test/Improve Product16%

Demonstration42%

Research for Policy15%

Pre-deployment22%

Develop/Improve New Product

64%

Proof of Concept20%

Product Development

21%

13

Funding by GoalsFunding by Goals

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Energy

Environment

Economic

Percentage of Projects with Expected Benefit

14

Co-Funding Sources Co-Funding Sources

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Research forpolicy

Study, proveconcept

Develop newor improved

product

Test andimproveproduct

Demonstration Pre-deployment

Project Type

Allo

cate

d F

undi

ng

NYSERDA GOVERNMENT ACADEMIC COMMERCIAL OTHER

15

Percent of Projects by Technology and Year Percent of Projects by Technology and Year

0%

20%

40%

60%

80%

100%

1999 2000 2001 2002 2003 2004

Energy Using Technology Environment Materials

Other Generic Power Systems Alternative Power

Cogeneration Energy Storage Transportation

16

Peer Review Focused on Six Success Stories as a Pilot TestPeer Review Focused on Six

Success Stories as a Pilot TestR&D Stage Technology Area Project Name

Information for Policy and R&D Community

HVAC 21st Century HVAC Research Consortium

Product Development Stage 1 – study and prove concepts

Demand Resposne Aggregating Distributed Generators (also in R&D Demonstration stage)

Product Development Stage 2 – develop new or improved products

Environmental EMEP – Development of Continuous Ambient Paticulate Monitor

Product Development Stage 3 – product testing

Transportation Evaluation of Truck Stop Electrification for NYS (also in Product Development Stages 1 and 2, Demonstration, and Pre-Deployment)

Demonstration Industrial Turnkey Pump and Compressed Air Program

Pre-deployment Power generation Green Power Marketing Program

17

Indicator VariablesIndicator Variables

• Choice of indicator variables for the R&D portfolio guided by the R&D portfolio logic model.

• Six categories of outcomes, identified in the logic model were selected: – Knowledge creation,– Knowledge dissemination,– Commercialization progress,– Energy benefits,– Economic benefits, and – Environmental benefits.

18

Accomplishment PacketsAccomplishment Packets• Project-specific accomplishment packets were then

developed to document objective evidence regarding the six outcomes:– Knowledge creation– Knowledge dissemination– Commercialization progress– Realized and potential energy benefits – Realized and potential economic benefits– Realized and potential environmental benefits – Value versus cost (not a specific outcome, but this

item was also included in the peer-reviewer response packet for 0 to 4 rating)

19

Review ProcessReview Process

• Reviewers willing to participate were sent:– Peer Review Instructions,– Conflict of Interest Form– Peer Review Assessment Form, and– the Peer Review Information Packet for their

specific project.• Over a period of five weeks, the reviewers

completed their assessment and returned them for data entry.

20

Results:Peer Review

Results:Peer Review

21

Weighted Rating By ProjectWeighted Rating By Project

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

TruckstopElectrification

Compressed AirProgram

21st CenturyHVAC

EMEP AirParticulateMonitoring

Aggregating DG

Me

an

Ra

tin

g (

We

igh

ted

)

22

Overall Ratings by OutcomeOverall Ratings by Outcome

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Knowledg

e Cre

ation

Knowledg

e Diss

eminati

on

Commerci

aliza

tion

Progr

ess

Energ

y Ben

efits

Econom

ic Bene

fits

Enviro

nmen

tal B

enef

its

Value

vers

us C

ost

Mea

n R

atin

g (

Un

wei

gh

ted

)

23

Overall Ratings, by Project, by Outcomes

Overall Ratings, by Project, by Outcomes

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

KnowledgeCreation

KnowledgeDissemination

CommercializationProgress

Energy Benefits Economic Benefits EnvironmentalBenefits

Overall Value

Un

wei

gh

ted

Rat

ing

Truckstop Electrification Compressed Air Program 21st Century HVAC EMEP Air Particulate Monitoring Aggregating DG

24

Conclusions: Aggregate AnalysisConclusions: Aggregate Analysis

• Assumes more risk than the commercial sector in the earlier stages of technology development, while in the latter stages, the reverse is true.

• Covers a wide range of technologies that are aimed at achieving potentially significant energy, economic and environmental benefits.

• Leverages funds on a 4.3 to 1 ratio.• Partners with a wide range of public and private

organizations and institutions.• Evolves over time in response to the societal

needs and opportunities to address them (i.e., the technologies and issues addressed in the R&D portfolio are not static).

25

Conclusions: Peer ReviewConclusions: Peer Review

• Peer review scores from the pilot test averaged 3.34 (on a 0-to-4 scale) across all assessment categories.

• There are substantial benefits across all documented accomplishment areas for the five projects assessed.

• Significant progress is being made toward the eventual achievement of measurable 3-E benefits.

26

Conclusions: Peer Review ProcessConclusions: Peer Review Process

• The information provided in the review packets for the five selected projects was adequate

• The instructions provided were clear• The criteria used in the assessments were clearly

defined• The criteria used in the assessments were the right

ones• It is very important for NYSERDA to assess the value

of its R&D programs• The results of the peer review process should be

useful for NYSERDA decision-makers• Reviewers can assess a fair amount of information if

the information is presented in a clear and organized format.

• Statistical analyses revealed that the ratings provided by the peer reviewers were reliable.

27

Next StepsNext Steps

• Routinize the collection of key indicator data for all R&D projects.

• Perform aggregate analysis on all projects • Focus significant effort on a more

representative sample of projects