Main Headquarters: 120 Water Street, Suite 350, North Andover, MA 01845 With offices in: NY, ME, TX, CA, OR

www.ers-inc.com

HOW TO DESIGN A GAS PROGRAM IMPACT EVALUATIONJonathan B. Maxwell, Energy & Resource Solutions (ERS)Kathryn Parlin, West Hill Energy & Computing

04/10/2023

Examine results for 13 gas evaluations Realization rates Variation of realization rates Net-to-gross Non-gas impact

Ramifications on evaluation designs

AGENDA

2

04/10/2023

Gas evaluation less mature than electric

Gas realization rates somewhat lower than electric

Evaluated project savings varies widely from reported—even in large custom programs

Account for interaction with other fuels

Account for non-energy benefits & costs

KEY POINTS

3

04/10/2023

GAS PROGRAMS EXAMINED

4

Program or Portfolio or Targeted Measure Type

No. Progra

msSample

Size

Residential single family new construction

1 25

Multifamily retrofit 1 6

C/I new construction & retrofit 2 48

C/I performance contracting 1 6

Commercial retrocommissioning 4 34

Industrial 1 29

Agriculture 1 30

Specialized (pipe insulation, bid program)

2 73

TOTAL 13 251

04/10/2023

Applicants typically estimated site-specific savings Expectation of good estimates

Evaluators estimated site-specific savings for all Majority “enhanced” level of

evaluation engineering rigor Many program types & administrators;

many evaluation engineering firms Actor bias unlikely

PROGRAM & EVALUATION TYPES

5

04/10/2023

0.68 median portfolio realization rate

Lower RR than for similar electric portfolios 5 of 13 portfolios had both electric & gas

RRs 0.70 median elec RR for the 5 0.53 median gas RR for the 5

SAVINGS REALIZATION RATES

6

.08 .08 .21 .33 .53 .64 .68

.72 .91 .92 .93 .98 1.07

04/10/2023

Measures variation of realization rates (Stratified ratio estimation) Lower is better Higher requires larger sample to get high

precision Unrelated to magnitude of realization rate

0.4 to 1.0 typical in EE evaluation 0.4 to 0.6 typical for electric EE

programs with site-specific analysis

ERROR RATIO - DEFINITION

7

04/10/2023

ERROR RATIO - ILLUSTRATION

8

04/10/2023

ERROR RATIO - RESULTS

9

Median 1.04 Shown with 6 outlier projects removed

04/10/2023

Why are error ratios so poor? Less mature programs Difficult for applicants (& evaluators) to

measure Baseline less clear (11% projects with 0

RR) Fuel switching Inherently difficult-to-predict measures

(RCx)

What to do? Intensively study gas measures Do not increase sample sizes at expense of

rigor per site

ERROR RATIO - OBSERVATIONS

10

04/10/2023

Similar methods used as with electric Mostly enhanced for these portfolios

0.85 median NTG factor 0.31 to 1.09 range for NTG factor More consistent than realization

rate

NET-TO-GROSS

11

04/10/2023

Results from 3 portfolio evaluations

Infrequent, but significant when it occurs

Customers add ~22% in utility bill savings

Should include in benefit-cost calculations

GAS MEASURE IMPACT ON ELECTRICITY, STEAM, OIL

12

Portfolio TypeElectric Impacts (kWh/ MMBtugas)

Other Energy Impacts (MMbtu/ MMBtugas)

Commercial New Construction 1.8 0.00

C/I Retrofit 15.1 0.02

Manufacturing--Ag-Food 18.5 na

04/10/2023

Results from 4 program evaluations

Infrequent but significant when it occurs Labor & water most common savings Customer save up to $0.75 for every $1 gas

saved Should include in benefit-cost calculations

IMPACT ON NON-ENERGY COSTS

13

Group Delivery ProgramNon-Energy Impact

(/ MMBtugas)

C/INew Construction $0.00

Existing Facilities $9.46

Loan Fund $1.22

1-4 Res. ENERGY STAR Homes $0.91

Multifamily Multifamily Building (Existing) $0.06

Summary - Evaluation Planning

Gas programs are less mature than electric

Expect large error ratios--evaluated savings varies widely from reported

Don’t sacrifice digging deeper for more sites Invest in enhanced M&V until programs mature

Allow time for vigorous feedback with program staff

Account for interaction with other fuels

Account for non-energy benefits & costs

aesp.org

04/10/2023

Gas programs are less mature than electric

Expect large error ratios--evaluated savings varies widely from reported

Don’t sacrifice digging deeper for more sites Invest in enhanced M&V until programs

mature

Allow time for vigorous feedback with program staff

Account for interaction with other fuels

Account for non-energy benefits & costs

SUMMARY - EVALUATION PLANNING

15