Customer Expectations & WTP For Improvements in Service
from Electricity DNOs
Pilot Debrief
February 2008
Agenda
• Fieldwork Methodology
• Interviewer Feedback
• Stated Preference Methodology
• Stated Preference Findings
• Summary of Recommended Changes
Fieldwork Methodology
• Objectives of pilot were to test:– the recruitment process – the survey hit rate– the clarity and flow of the questionnaire– the appropriateness of the language used– the accuracy of all routings– ease of use of the show material– design and understanding of the stated preference exercises– the interview duration
• Approach:– Businesses: 40 CATI phone-post/email-phone (41 achieved)– Residential: 100 CAPI in-home interviews (99 achieved)
• Quotas:– Business:
• 10 EPN (11 achieved), 10 SPN (10 achieved), 20 LPN (20 achieved)• Mix of size (20 small; 15 medium; 6 large) & sector• Minimum 10% with outage experience (39% achieved)
– Residential:• 25 EPN (25 achieved), 25 SPN (25 achieved), 50 LPN (49 achieved)• Mix of age & SEG
General Feedback: Interview Duration
31.3
18.2
35.3
15.2
0
10
20
30
40
50
25 minutes orless
>25-30minutes
>30-45minutes
45 mins+
% R
esp
on
de
nts
4.9
21.9
29.3
43.9
0
10
20
30
40
50
25 minutes orless
>25-30minutes
>30-45minutes
45 mins+
% R
esp
on
de
nts
DOMESTICMean = 32 minutes
BUSINESSMean = 43 minutes
General Feedback: Comments on Domestic Questionnaire (non SP)
• Problems with clarity & flow:– Question 34: ‘did you find each of the levels of
service described easy to follow?’… if you said ‘no’, Question 35 then said ‘which levels weren’t clear to you?’; seem like separate issues
– Need showcard for income question
• Problems with routing:– Q22b; if they say don’t know should skip
subsequent question
• General comments on what would make it easier– Concern that people will think we are trying to
sell them services for an electricity company. Also, many people less aware of Ofgem than of Ofwat or the FSA, so they were very cautious. Maybe a more elaborate explanation of the survey might help.
• Interview duration:– Longer than budgeted for in most instances;
cannot be done within current fee (due both to time required for each and impact of length on hit rates) and current incentive (due to length)
• Other:– Some issues with defining rural versus urban –
we will use a postcode dbase to do so in the mainstage
– Add all screening questions (eg age, SEG) to main questionnaire
– Add question about number of children 5+
General Feedback: Comments on Business Questionnaire (non SP)
• Problems with clarity & flow:– Some people assumed it was a sales call
despite the current intro. Need to further emphasize that Ofgem are a regulatory body and that the research is looking at improvements that could be made to the services provided to their company
– RQ13 – need a “don’t know code– Q23: some said “both”; add this code?– Q26: needs a don’t know or can’t
remember– Q40: needs re-phrasing so it doesn’t
suggest anything will be sent; needs to separate leaflet and email; needs to be multicode
• Problems with routing:– none
• Interview duration:– Longer than budgeted for in most
instances; cannot be done within current fee (due both to time required for each and impact of length on hit rates)
• Other:– Some issues with defining rural versus
urban – again, we will use a postcode dbase to do so in the mainstage
Stated Preference Methodology
• Objective: to obtain estimates of customers’ willingness-to-pay for improved service levels (or willingness-to-accept for reduced service levels)
• No revealed preference evidence, so need to rely on stated preferences– Customers are not offered different service levels and prices in real world
• Two possible stated preference approaches:– Contingent Valuation questions (CV) – ask respondents directly questions
about valuations
– Discrete choice experiments (SP DCE) – elicit valuations through tradeoffs• Based on hypothetical choices• Essential that price is one of the attributes
Example Choice: Experiment 1 (Domestic)
Choice 3
Average number of power cuts longer than 3 mins in normal weather conditions
Average duration of power cut
Average number of power cuts shorter than 3 mins in normal weather conditions
Information provided during power cuts
Annual Electricity Bill
Choice (mark "X" in preferred option)
Cost£200
(no change)£198
(£2 decrease)£205
(£5 increase)
Provision of InformationTelephone information line
Telephone information lineplus call backs to provide
information updatesTelephone information line
Duration of Loss of Power100 mins on average
115 mins on average(more than now)
80 mins on average(less than now)
Which electricity distribution service would you choose?
Frequency of Power Cuts (more than 3 minutes)4 in 5 years
7 in 5 years(more than now)
5 in 5 years(more than now)
As Now Alternative 1 Alternative 2
Frequency of Short Interruption (less than 3 minutes)5 in 5 years
6 in 5 years(more than now)
5 in 5 years
CV vs SP DCE
Contingent Valuation (CV)Discrete Choice Experiments
(DCE)
May present cognitive problems
DCE does not explicitly ask about money values, so may
be easier for people to understand
Typically collect one response from each individual
More efficient - typically collect multiple responses
from individuals
Argued that CV should be chosen when WTP for the good or service in total is
required
Consumer surplus methods can be used to estimate the
value of a total good from DCE data
Attributes Tested for Domestic Customers
All Distributors except LPN LPN
Experiment 1
Frequency of power cuts over 3 mins Frequency of power cuts over 3 minsAverage duration of power cuts over 3
minsAverage duration of power cuts over 3
mins
Number of short power interruptions Number of short power interruptions
Provision of Information
Experiment 2
Restoration of supply (time) Restoration of supply (time)
Compensation for restoration of supply Compensation for restoration of supply
Compensation for multiple interruptions Compensation for multiple interruptions
Planned interruptions - notice Planned interruptions - notice
Experiment 3
Undergrounding Provision of InformationNetwork resilience to major storms Network resilience to flooding
Network resilience to flooding Reduction in carbon emissionsReduction in carbon emissions
Cost also included in all experiments
Attributes Tested for Business Customers
Cost also included in all experiments
All Distributors except LPN LPN
Experiment 1
Frequency of power cuts over 3 mins Frequency of power cuts over 3 mins
Average duration of power cuts over 3 mins
Average duration of power cuts over 3 mins
Number of short power interruptions Number of short power interruptions
Provision of Information
Experiment 2
Restoration of supply (time) Restoration of supply (time)
Compensation for restoration of supply Compensation for restoration of supply
Making and keeping appointments Making and keeping appointmentsPlanned interruptions - notice Planned interruptions - notice
Experiment 3
Network resilience to major storms Network resilience to floodingNetwork resilience to flooding Reduction in carbon emissions
Reduction in carbon emissions Energy efficiency advice
Energy efficiency advice Provision of Information
Stated Preference Methodology
• Potential problem with separation of attributes across experiments is that results may over-estimate total wtp
• Therefore we also include a higher-level experiment, including all attributes to test total wtp– We also include a series of CV questions to elicit total wtp for best service
• The higher-level experiment is simplified so that ‘group’ of attributes are varied simultaneously
• Two levels tested– Lowest service level– Highest service level
• Still complicated
Example higher-level choice: 6 choices
Choice 1
Frequency of Power Cuts over 3 mins Average number of power cuts longer than 3 mins in normal weather conditions
Duration of Loss of Power Average duration of power cut
Frequency of Power Cuts less than 3 mins Average number of power cuts shorter than 3 mins in normal weather conditions
Provision of Information Information provided during power cuts
Restoration of Supply (Normal Weather) Restoration of electricity supply as a result of problems not related to weather
Compensation for Failure to Restore Supply (Normal Weather) Compensation if your electricity is not restored in this time
Compensation for Multiple Interruptions Number of interruptions lasting 3 or more hours in a year which will entitles you to compensation payment
Planned Interruption to Electricity Supply Amount of notice that customers must be given before planned interruptions to electricity supply
Undergrounding An ongoing commitment to underground overhead lines for amenity reasons
Network resilience to major storms Number of customers affected by major storms
Network resilience to flooding Expenditure on flood defences
Reduction of carbon emissions Replacement of mobile generation equipment and vehicles with those using less polluting vehicles
Cost Annual Electricity Bill
Choice (mark "X" in preferred option)
Neither
Which electricity distribution service would you choose?
5% of overhead linesper annum
7 in 5 years(more than now)
Guaranteed within 18 hours
Alternative 1
None
Replace 10% per year with those using less polluting fuels
80 mins on average(less than now)
£50 plus £25 for every additional 12 hour period
2 days notice
7 in 5 years(more than now)
Telephone information lineTelephone information lineplus call backs to provide
information updates
Compensation after5 interruptions
Compensation after5 interruptions
120 mins on average(more than now)
£207(£7 increase)
Guaranteed within 18 hours
Increase by 15%
£196(£4 decrease)
144000 customerson average in a year(20% less than now)
216000 customerson average in a year(20% more than now)
Continue usage of current equipment and vehicles
Current expenditure
Alternative 2
1 in 5 years(less than now)
3 in 5 years(less than now)
£50 plus £25 for every additional 12 hour period
2 days notice
Stated Preference Methodology – Analysis
• Analysis based on principles of utility maximisation
• ‘Utility’ assigned to each alternative, made up of:– Attributes of the alternative– Characteristics of the respondent, e.g. income, etc.
• Assumption that error distribution is extreme value (iid) leads to logit model
• Maximum likelihood techniques used to obtain values of unknown parameters– Find the βs that optimise explanation of the model
• Model outputs– Coefficient estimates (and t-statistics)– Model fit statistics
Utility Specification
inni XXXU ...221
Random error reflecting influence
of parameters not included, differences
between individuals/responses
Parameters to be estimated
Characteristics of
alternative/individual
Logit Model Formulation
Assumption that random terms () are independent and identically distributed across alternatives (IID) and that they
vary with an Extreme-Value distribution
Jj
Uj
Ui
i e
eP
Stated Preference Methodology – Analysis
• Coefficients used to:– Determine the relative importance of attributes– Determine the monetary value for attributes– Specify utility functions for prediction models
• Monetary values for attributes identified in lower-level games
• May be adjusted by values for groups of attributes identified by higher-level game or by CV question– This is a judgement call
Stated Preference – Did it Work?
• Trading behaviour
• Background Questions on respondent understanding
• Model Results
Most domestic respondents were willing to choose new service options in at least one of the choices• Domestic – London
• Domestic – Other DNOs
Trading Exp 1 Exp 2 Exp 3 Exp 4 Always Current Option 3 2 2 n/a
Always Alternative 1 0 0 1 0
Always Alternative 2 0 0 0 0
Neither/No Choice 0 0 0 2
Trading among Alternatives 46 47 46 47
Total Observation 49 49 49 49
Trading Exp 1 Exp 2 Exp 3 Exp 4 Always Current Option 4 3 3 n/a
Always Alternative 1 1 1 0 1
Always Alternative 2 0 0 0 0
Neither/No Choice 0 0 0 1
Trading among Alternatives 45 46 47 48
Total Observation 50 50 50 50
Most business respondents were willing to choose new service options in at least one of the choices• Business – London
• Business – Other DNOs
Trading Exp 1 Exp 2 Exp 3 Exp 4
Always Current Option 4 5 4 n/a
Always Alternative 1 0 0 1 2
Always Alternative 2 0 0 0 0
Neither/No Choice 0 0 0 2
Trading among Alternatives 16 15 15 16
Total Observation 20 20 20 20
Trading Exp 1 Exp 2 Exp 3 Exp 4
Always Current Option 0 0 0 n/a Always Alternative 1 0 0 0 0 Always Alternative 2 0 0 0 0 Neither/No Choice 0 0 0 1
Trading among Alternatives 21 21 21 20 Total Observation 21 21 21 21
Examining choices at different cost levels implies that many domestic customers were willing to pay 15% increases in DUOS for service improvements
Experiment 1 Experiment 2 Experiment 3
Other DNOs
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt A(% adjustment to DUOS)
% C
ho
osi
ng
Alt
ern
ativ
e A
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt B(% adjustment to DUOS)
% C
ho
osi
ng
Alt
ern
ativ
e B
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt A(% adjustment to DUOS)
% C
ho
osi
ng
Alt
ern
ativ
e A
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt B(% adjustment to DUOS)
% C
ho
osi
ng
Alt
ern
ativ
e B
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt A(% adjustment to DUOS)
% C
ho
osi
ng
Alt
ern
ativ
e A
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt B(% adjustment to DUOS)
% C
ho
osi
ng
Alt
ern
ativ
e B
A wider range of costs was investigated in the higher-level experiment with large numbers of domestic users choosing alternatives with large cost increases
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15% 20% 25% 30%
Cost Level of Alt A(% adjustment to DUOS)
% C
ho
osi
ng
Alt
ern
ativ
e A
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15% 20% 25% 30%
Cost Level of Alt B(% adjustment to DUOS)
% C
ho
osi
ng
Alt
ern
ativ
e B
Examining choices at different cost levels implies that many business customers were willing to pay 15% increases in DUOS for service improvements
Experiment 1 Experiment 2 Experiment 3
Other DNOs
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt A(as % of Annual Bill)
% C
ho
osi
ng
Alt
ern
ativ
e A
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt B(as % of Annual Bill)
% C
ho
osi
ng
Alt
ern
ativ
e B
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt A(as % of Annual Bill)
% C
ho
osi
ng
Alt
ern
ativ
e A
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt B(as % of Annual Bill)
% C
ho
osi
ng
Alt
ern
ativ
e B
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt A(as % of Annual Bill)
% C
ho
osi
ng
Alt
ern
ativ
e A
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15%
Cost Level of Alt B(as % of Annual Bill)
% C
ho
osi
ng
Alt
ern
ativ
e B
A wider range of costs was investigated in the higher-level experiment with large numbers of business users choosing alternatives with large cost increases
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15% 20% 25% 30%
Cost Level of Alt A(as % of Annual Bill)
% C
ho
osi
ng
Alt
ern
ativ
e A
0%
20%
40%
60%
80%
100%
-15% -10% -5% 0% 5% 10% 15% 20% 25% 30%
Cost Level of Alt B(as % of Annual Bill)
% C
ho
osi
ng
Alt
ern
ativ
e B
Most respondents understood the experiments
• ‘Were you able to make the comparisons in the choices we presented to you’
– Domestic• 48 of 49 respondent in London reported they were able to make the
comparisons• 47 out of 50 respondents in other DNOs reported they were able to make the
comparisons
– Business• 19 of 20 respondents in London reported they were able to make the
comparisons• 20 of 21 respondents in other DNOs reported they were able to make the
comparisons
A sizeable proportion of domestic customers felt that the levels of service in the choices were unrealistic, and the experiments were difficult
• ‘Were the levels of service we have been asking about in the choices realistic?’
– London, 11 of 49 domestic respondents felt the service levels were not realistic or too difficult• A 1 in 10 year electricity cut is not realistic because we are having electricity cut 6 times in a
year• I don’t think all the changes will be possible within the costs stated• It is seriously insane to raise the bills again and again• No electricity company would cut emissions, as they are not efficient enough to replace the
current system• There were too many options and everything was too similar• I don’t believe the whole concept, it’s not clear enough• They are all the same• Too much complicated
– Other, 8 of 50 domestic respondents felt the service levels were not realistic or too difficult• Because they are not achievable, ie the cost of putting all these cables underground isn’t
going to be economically viable• The last thing I want is for my bill to go up• They are talking about increasing expenditure by certain percentages, but the original
expenditure is not listed… so 15% increase of what?• The comparisons are not obvious, got to really read in order to give an answer• The different choices or packages didn’t always seem logical.• Too much to absorb all at one for the average consumer who is having enough problems
paying their bills
Most business respondents felt that the levels of service in the choices were realistic
• ‘Were the levels of service we have been asking about in the choices realistic?
– London, 3 of 20 respondents felt the service levels were not realistic• 7 power cuts in 10 years is outrageous• Because we had 3 power cuts in the last 2 years of long duration• They do not employ enough people, so realistically it is not going to happen
– Other, 4 of 21 respondents felt the service levels were not realistic• Because some reductions are not able to be guaranteed. There are elements
outside your control, i.e. the power cut we experienced was due to accidental cutting through power cables.
• Some options offered more for less….doesn’t make sense.• Typical of today’s climate, people want all these things, but are not prepared
to pay for them. Realistically, these improvements would probably cost too much.
• Some of the improvements would be very difficult to achieve.
Model results are encouraging
• Domestic Customer Models– Results are very encouraging– Cost coefficient is significantly negative– Most other coefficients have intuitive signs, many are significant– Resulting wtp values look reasonable
• Business Customer Models– Many statistically insignificant terms, probably a result of small sample sizes
and heterogeneity of sample, e.g. in terms of size of business– Expect model to be improved with larger sample size in main survey– Most important variables:
• Power cuts• Restoration of supply• Reduction of carbon emissions
Interviewer Feedback on SP
• “Very complex”
• “Too much to consider”
• “Consider reducing and rotating over 2 interviews to make each survey shorter”
• “Maybe cut down on the sets of choices; if possible, 6 instead of 8. Choice sets are quite long”
• “Most respondents found the exercise difficult or complex at first but then adapted to it. It is a bit difficult to keep them interested in the survey till the end”
• “The interview should be shortened if possible”
• “It’s a lot of reading”
Conclusions
• Subject area is abstract and the choices are complex, but – despite some concerns on the part of interviewers – respondents seem to have been able to undertake the SP experiments
• Some, mostly minor, revisions required for main survey– Main area for review is cost levels
• Mainstage interviews:– 2,100 domestic face-to-face interviews (150 interviews conducted in each of the
14 DNO territories)– 1050 CATI business interviews (75 per DNO territory)
Recommended Changes
• Review cost levels– Pilot cost levels:
– Fewer reductions and more increases for Experiments 2 and 3, e.g. 2 reductions, 6 increases and base
– Recommend that we incorporate larger cost increases
• Review wording of attributes and levels to simplify as much as is possible (may not be much scope for this)– Include ‘better than now’ and ‘worse than now’ descriptors, where appropriate
• Review ‘Network resilience to flooding’ definition– Complaint that it was difficult to make comparisons when base level is not defined
• Reduce number of choices in lower-level experiments from 8 per experiment to 6 per experiment
Levels 1 2 3 4 5 6 7 8 9Exp 1 - 3 -15% -10% -5% -2% 0% 2% 5% 10% 15%
-10% -4% 0% 2% 6% 10% 16% 22% 26%-12% -6% 0% 3% 7% 12% 18% 24% 28%-15% -8% 0% 4% 8% 14% 20% 25% 30%
Higher Level
Other Recommendations
• Minor Text Changes– Compensation for Multiple Interruptions: minor error in wording ‘which will entitles you…’,
plus (£x) in text to be removed– Network Resilience to Major Storms: include a ‘,’ separator in the numbers, e.g. 180,000
rather than 180000
• Minor Changes to Questions/Routing– Slightly reword Q34 and/or Q35 (domestic) / Q46 and/or Q47 (business)– Amend routing for Q22(domestic) / Q33 (business) “don’t knows”– Use postcode dbase to define rural & urban– Add age & SEG data from screener to main questionnaire (domestic only)– Add question about number of children 5+ (domestic only)– RQ13 (business) – need a “don’t know code– Q18 (domestic)/Q23 (business): some said “both”; add this code?– Q20 (domestic)/Q26 (business): needs a don’t know or can’t remember– Q40 (business only): needs re-phrasing so it doesn’t suggest anything will be sent; needs
to separate leaflet and email; needs to be multicode
• Expand explanation of survey & of Ofgem
• Reduce the length of the questionnaire or increase fees & incentives