PY2012 AUTO-DR PROGRAM Process Evaluation Findings November 12,
2014
Slide 2
Presentation Agenda PY2012 Auto-DR Program Process Evaluation
Findings 2 Overview of Evaluation Effort Summary of Methods Key
Findings & Recommendations
Slide 3
Overview of PY2012 Auto-DR Process Evaluation Effort PY2012
Auto-DR Program Process Evaluation Findings 3 Opinion Dynamics
conducted a process evaluation to explore a variety of research
areas. This presentation focuses on findings from the following
areas: Participant Snapshot Motivations and Benefits Barriers and
Challenges to Participation Program Processes Communication
Slide 4
Summary of Evaluation Efforts PY2012 Auto-DR Program Process
Evaluation Findings 4 The evaluation team conducted the following
tasks: Evaluation TaskDescription Program Materials and Database
Review Analyzed the Auto-DR and other DR program databases to
characterize and understand the participant population Program
Staff InterviewsConducted telephone or in-person interviews with
program staff from each of the three IOUs Program Stakeholder
Interviews Conducted telephone interviews with 25 program
stakeholders across the three IOUs. These stakeholders include key
account representatives, vendors, program verification engineers
and DR program staff 2012 Participant SurveyFielded telephone
survey to 179 program participants with contact information,
resulting in 49 completes 2013 Participant Interviews Conducted
telephone interviews with six 2013 participants across the three
IOUs
Slide 5
2012 Participant Survey Response Rates PY2012 Auto-DR Program
Process Evaluation Findings 5 We fielded a telephone survey to a
census of 153 customers with valid contact information out of a
total of 224 enrolled participants 49 participants completed the
survey, for a statewide response rate of 32% IOUNumber of Survey
Respondents Number of Participants with Valid Contact Number of
Participants Enrolled PG&E104593 SCE3497113 SDG&E51118
Total49153224 Participant Population and Respondents, by IOU Over
32% of enrolled participants had missing or invalid contact
information Response rate: American Association for Public Opinion
Research (AAPOR) Response Rate 4.
Slide 6
Survey respondents were generally representative Statewide
PY2012 Auto-DR Program Process Evaluation Findings 6 Respondents
are generally representative, however, given the limited number of
completes by IOU (49), we present findings at a Statewide level In
general, survey respondents represent: 22% of total statewide
enrolled participants, and by IOU, represent 11% of PG&E
participants, 30% of SCE participants, and 28% of SDG&E
participants [1] [1] A mix of active [2] (82%), partial [3] (6%),
and inactive [4] (12%) participants [2] [3] [4] A mix of eight DR
programs, including both capacity and pricing programs, as well as
aggregator-managed programs A range of sectors, including
manufacturing, agriculture, building operators, and hospitality 65%
of participants with a single site and 35% of participants with
multiple sites Represent 23% of realized load-shed for PG&E
customers and 20% of realized load- shed for SCE customers
Slide 7
2012 Program Participation Snapshot PY2012 Auto-DR Program
Process Evaluation Findings 7 In 2012, there were: 224 unique
enrolled customers, totaling 1,119 sites* The largest share of
participants comes from SCE (113 customers, 634 sites), with the
smallest from SDG&E (18 customers, 115 sites) The program paid
out incentives of ~$40 million to the 224 participants enrolled
from November 2008 December 2012** Three quarters of participants
are active (i.e., participated in at least one DR event in 2012)
Over one third participate in multiple DR programs Over one half
install technology the control multiple equipment types (52%),
primarily HVAC *A site represents technology installed at one
customer site at one point in time defined by their service account
identifier. **Covers participants with incentive data
Slide 8
Customers perceived a variety of benefits to participating in
the program PY2012 Auto-DR Program Process Evaluation Findings 8
The majority of respondents first became aware of the program
incentives through their utility account rep (55%)
FindingIllustrative Quote The majority of respondents (61%)
reported that the technology lowered their operational costs [The
technology gave us]the ability to have greater control over the
facility, control energy costs, and receivecredits for
participating in [demand response] events. About half of
respondents (49%) reported that the technology reduced their energy
use on non-event days [The technology helped]reduce energy usage
and save money in the process. Nearly half of respondents (43%)
reported that the technology increased their operational efficiency
The same technology controls the entire refrigeration system with
advanced capability, graphic trends, alarms, better interface.
Responding Participant Benefits of Participating in Auto-DR Program
(n=49) About 1/3 of respondents reported reducing operational costs
was primary selling point of program
Slide 9
Customers report that technologies make participating in events
easier PY2012 Auto-DR Program Process Evaluation Findings 9
Respondents Indicating Whether Installation of the Technology Made
Event Participation Easier, More Difficult, or the Same (n=19)
Note: We asked this question to those respondents who were already
enrolled in a DR program before they installed the Auto DR
technology. 68% said that technology made event participation
easier a reduction in costs, helping meet savings goals
inconveniences in time of day their employees had to work, when the
lights shut off, and events increased production time of
respondents who participated in an event in 2013 said it had no
impact on their business
Slide 10
The Auto-DR Program channels customers into DR programs PY2012
Auto-DR Program Process Evaluation Findings 10 Auto-DR incentives
motivated respondents to enroll in DR programs (Program requires
enrollment in DR program prior to receiving incentives) A little
over half of respondents enrolled in DR programs after enrolling in
Auto-DR Of these, most indicated they would not have enrolled in
the DR program if the Auto-DR incentives were not available
Slide 11
Challenges revolve around technology and installation, event
participation PY2012 Auto-DR Program Process Evaluation Findings 11
CategoryDid your company encounter any challenges when
participating in the program? FrequencyPercent No challengeNo
challenges2242% Technology and Installation Challenges Technology
didnt work/work correctly (4) Some areas/equipment cannot be
altered (2) Set-up/programming challenges (2) Staff learning to use
technology (2) Technology did not meet load reduction expectations
(1) Communication issues (1) 1224% DR Event Participation
Challenges Customer/client satisfaction (3) Load reduction required
(1) Timing of event days (1) Demand response program challenges (1)
Too time-intensive (1) Changing production schedule (1) 816%
Program Processes Challenges Application process (2) Challenges
with utility staff (1) Problems with vendor (1) Lack of
coordination (1) 510% Decision-Making Challenges Too good to be
true/ no money given (2) Tough to convince management that systems
payback (1) Disagreement within company (1) 48% Other 24%
Challenges Encountered while Participating in the Program (Multiple
Response: n=49) Its still not operational, it didnt work, and after
installation, [the] services were poor.
Slide 12
Reactions to the 60/40 incentive structure Auto-DR Evaluation
Interim Stakeholder Findings 12 Customer implications None of the
2013 participants interviewed were affected by the 60-40% incentive
structure. Some were unaware of it, while others said that the
vendor received the incentive and assumed all risk Vendor
implications: 6 of 7 vendors say they have lost customers because
of the changed incentive structure Vendors report that lending
institutions unlikely to finance projects Small vendors most upset,
and may make some small vendors exit the program Engineer
implications: May improve realization rates between load shed
estimate and event participation (increases customer incentive to
participate and decreases vendor incentive to put in aggressive
estimates)
Slide 13
Participants are generally satisfied, however there are
opportunities to improve application processing PY2012 Auto-DR
Program Process Evaluation Findings 13 Vendor Comments: The
processing time can be extremely long; it was longer than the
expectation. It took a long time to process the application issues
with verifying the information provided in the application. Six
months later, most of the applications for enrolling other sites
have not progressed. Respondents were least satisfied with the
length of time it took to fill out the application, and the length
of time to process the application Vendors note they lose customers
because of wait time Vendors report that engineering review process
is time consuming Account reps indicate slow review process, takes
too long, complex process
Slide 14
Customers and stakeholders suggest opportunities to improve
communication PY2012 Auto-DR Program Process Evaluation Findings 14
Overwhelmingly, respondents suggest improving communication and
information provided about the DR program and program events
Customers suggest providing more information about when events were
coming, the number of events that would be called, and receiving
notification earlier, and meeting at the beginning of the event
season to remind participants of up- coming events Stakeholders
report issues regarding communication of how incentives are
calculated and communication regarding project status Vendors want
transparency in the incentive calculation to reduce the risk of
receiving lower incentive. In particular, want more information on
how baselines are chosen to estimate load reduction Engineers note
that their baselines are typically more conservative than vendor
baselines (i.e., peak load rather than average load) Vendors report
that they typically dont know project status and suggest online
project status tracking system from IOUs Account reps indicate that
they would like to know best IOU contact to refer key
questions
Slide 15
Key Opportunities & Recommendations PY2012 Auto-DR Program
Process Evaluation Findings 15 Communication Provide enhanced
training to vendors and acct reps regarding the program Conduct
project-specific meetings with key stakeholders to better explain
the program, increase transparency of baseline calculations, and
clarify concerns Provide status updates to stakeholders and
participants throughout the participation process Barriers and
Challenges to Participation Enhance vendor quality control
activities to ensure operability of technology We understand that
the program is open to any vendor with a qualifying technology and
the IOUs cannot make vendor recommendations; however we suggest
that the IOUs consider approaches that address operability of
technology to increase participation as well as enhance customer
satisfaction
Slide 16
Potential future research areas PY2012 Auto-DR Program Process
Evaluation Findings 16 Explore effects of 2012-2014 program design
changes by tracking the following metrics year-over-year to assess
their impact: Number of newly enrolled customers Number of
participating vendors Proportion of participants that participate
in events in a given year Explore data mining options to
characterize participants to: Provide insights into the type of
participants who provide consistent and reliable load-shed Identify
incremental impacts and characteristics of participants by
assessing load impact and other metrics between Auto-DR and
non-Auto-DR participants Target recruitment by exploring
differences between participants who enrolled in a DR program
before or after installing the Auto-DR technologies, as well as
profile enrollment by DR program
