Upload
zondits
View
60
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Zondits
Citation preview
Main Headquarters: 120 Water Street, Suite 350, North Andover, MA 01845 With offices in: NY, ME, TX, CA, OR www.ers-inc.com
MERGE AHEAD: Integrating Process and Impact Evaluations
2014 ACEEE Summer Study Patrick Hewlett, P.E. ERS August 21, 2014
Example: Consider a small-business direct install program that underwent concurrent but separate process and impact evaluations. Process finding: “The total potential energy savings associated with franchises is significantly greater than with individually owned businesses. To better achieve program savings goals, the utilities should engage franchise accounts with multiple eligible facility locations.”
Impact finding: “Franchises were found to have higher free-ridership than non-franchises and should therefore be carefully screened during the incentive application process.”
WHY INTEGRATE?
8/26/2014 2
Status quo and barriers to integration An integrated approach for three phases:
Evaluation planning Data collection Reporting
Joint recommendations: Quantified Harmonized
A blueprint for PAs
OVERVIEW
8/26/2014 3
STATUS QUO
8/26/2014 4
Traditional Process approach: • Big-picture assessment of program effectiveness • Delivery, marketing, outreach, incentive structures,
customer satisfaction
Traditional impact approach:
• Focused on producing one number: net savings • Detailed file review, site visits with M&V,
engineering analysis
Efficiency program improvement through recommendations
Timing Process evaluations typically occur before impact Actionable feedback for PAs, can be implemented more quickly
Budget Two evaluations might not be funded by same source Levels of funding are often different Additional budget required to allow for strategic planning and joint
recommendation development
Objective Traditional process and impact evaluations not thought to intersect However, recommendations benefit from an intersection
BARRIERS TO INTEGRATION
8/26/2014 5
Process recommendations not informed by comprehensive file review and field findings
Impact recommendations often not grounded in
program costs, logistics
CONSEQUENCES
8/26/2014 6
Result: unrelated, isolated recommendations Program hesitant to act on recommendations that
might be out-of-touch or idealistic.
AN INTEGRATED APPROACH
8/26/2014 7
Traditional Process
Approach
Program benchmarking
Interviews with program staff
Define research objectives
Application review
Interviews with customers, contractors
Compliance: age, EUL, cost, payback
Tracking system review
Process findings
Estimation best practices
Implementation
Customer/contractor experience
Process recommendations
Integrated Approach
Benchmarking to define program needs
Set joint research objectives
Plan for joint data collection
Collaborative project file review
Initial application/ algorithm findings
Joint survey questions
Field data collection on compliance factors
Discrepancy analysis driving prioritized, joint recommendations based on savings, cost
Traditional Impact
Approach
Areas of uncertainty
Define research objectives
Review program technologies
Project file review
Algorithm review
On-site data collection and M&V
Interviews with customers, contractors, nonparticipants
Savings results
Realization rates
Net-to-gross
Impact discrepancies
Impact recommendations
• Program benchmarking • Interviews with program staff • Define research objectives
• Benchmarking to define program needs • Set joint research objectives • Plan for joint data collection
EVALUATION PLANNING
8/26/2014 8
Traditional Process
Approach Integrated Approach
Traditional Impact
Approach
• Areas of uncertainty • Define research objectives • Review program technologies
• Application review • Interviews with customers, contractors • Compliance: age, EUL, cost, payback • Tracking system review
• Joint survey questions • Collaborative project file review • Initial application/ algorithm findings • Field data collection on compliance factors
DATA COLLECTION
8/26/2014 9
Traditional Process
Approach Integrated Approach
Traditional Impact
Approach
• Project file review • Algorithm review • On-site data collection and M&V • Interviews with customers, contractors, nonparticipants
• Estimation best practices • Path to implementation • Customer, contractor experience • Process recs
• Discrepancy analysis driving prioritized, joint, actionable recs based on savings, cost
REPORTING
8/26/2014 10
Traditional Process
Approach Integrated Approach
Traditional Impact
Approach
• Savings results • Realization rates • Net-to-gross ratios • Impact discrepancies • Impact recs
A key benefit of an integrated approach is quantified and harmonized recommendations.
Quantified recommendations Allows program to prioritize recommendations by savings
and/or costs Impact team can quantify the program’s shortcomings by
savings magnitude, highlighting areas of greatest need Process team research informs the costs and barriers of
implementation and proposes solutions Result: program administrators able to weigh
recommendations, prioritize by need, and more immediately act.
AN INTEGRATED APPROACH
8/26/2014 11
QUANTIFIED RECOMMENDATIONS
# ProjectsImpact on kWh RR
Impact on kWh RR
# Projects
-8%2%
-8%0%
-6%1%
0%0%
0%0%
-1%0%
-1%0%
-2%1%
-6%8%
-2%3%
-6%0%
-2%0%
102 -41% 15% 41
Unsubstantiated savings 5 0
Totals
4 2
Difference in controls 9 0
Estimating Performance
Incorrect load profile 6 2
Difference in operating hours 17 21
Deemed savings assumed
7 2
Incorrect technology 3 0
2 1
Inspection
Inoperable equipment 1 0
Quantity or size error
Measure Screening
Ineligible measure 13 0
Incorrect baseline 8 2
Interactivity
Category
Negative Positive
Administrative Administrative error 27 11
8/26/2014 12
QUANTIFIED RECOMMENDATIONS
# ProjectsImpact on kWh RR
Impact on kWh RR
# Projects
-8%2%
-8%0%
-6%1%
0%0%
0%0%
-1%0%
-1%0%
-2%1%
-6%8%
-2%3%
-6%0%
-2%0%
102 -41% 15% 41
Unsubstantiated savings 5 0
Totals
4 2
Difference in controls 9 0
Estimating Performance
Incorrect load profile 6 2
Difference in operating hours 17 21
Deemed savings assumed
7 2
Incorrect technology 3 0
2 1
Inspection
Inoperable equipment 1 0
Quantity or size error
Measure Screening
Ineligible measure 13 0
Incorrect baseline 8 2
Interactivity
Category
Negative Positive
Administrative Administrative error 27 11
8/26/2014 13
Savings tracking and measure screening issues led to a 19% drop in kWh RR
QUANTIFIED RECOMMENDATIONS
# ProjectsImpact on kWh RR
Impact on kWh RR
# Projects
-8%2%
-8%0%
-6%1%
0%0%
0%0%
-1%0%
-1%0%
-2%1%
-6%8%
-2%3%
-6%0%
-2%0%
102 -41% 15% 41
Unsubstantiated savings 5 0
Totals
4 2
Difference in controls 9 0
Estimating Performance
Incorrect load profile 6 2
Difference in operating hours 17 21
Deemed savings assumed
7 2
Incorrect technology 3 0
2 1
Inspection
Inoperable equipment 1 0
Quantity or size error
Measure Screening
Ineligible measure 13 0
Incorrect baseline 8 2
Interactivity
Category
Negative Positive
Administrative Administrative error 27 11
8/26/2014 14
Operating hours assumptions and deemed savings had relatively high swings, but is action needed?
Process finding: “Database is fully populated and includes appropriate fields for tracking applications.” Impact recommendation: “Eliminate all duplicate records in the tracking dataset and always update tracking system with the most current estimates of savings.” Integrated recommendation: “Database design conforms to best practices. Apparently, QC processes are inadequate because tracking errors degraded the realization rate by 6%. Convene a team of database administrators, program implementers, and the contractor to map out the tracking database and identify specific checkpoints in the system for reducing these errors.”
HARMONIZED RECOMMENDATIONS
8/26/2014 15
HARMONIZED RECOMMENDATIONS
8/26/2014 16
Process finding: “Customers prove eligibility through extensive screening of project applications, which drives down participation.” Impact recommendation: “Ineligible customers have been approved due to inaccurate account information as determined through field data collection. Each project should require an on-site pre-inspection to ensure eligibility.” Integrated recommendation: “Develop a benchmarking system to screen customer-claimed usage by square footage and facility type. Require field inspection for those customers that do not meet the benchmark, which is expected to occur about 15% of the time.”
An integrated evaluation approach requires strategic planning for joint objectives. Assignment of roles, responsibilities Agreement on method of valuing recommendations Joint interviews with program staff Coordination of delivery schedules and check-ins
Regular communication among teams Leading to joint recommendations
Process research informs feasibility of implementation Impact research highlights areas of greatest need
Result: actionable, grounded recommendations
prioritized by need.
A BLUEPRINT FOR PAS
8/26/2014 17
CONTACTS Patrick Hewlett, P.E.
ERS
(978) 478-5305
Sue Haselhorst
ERS
(978) 478-5325
8/26/2014 18