Upload
claus
View
24
Download
3
Embed Size (px)
DESCRIPTION
Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service Delivery. Jackie Berger APPRISE. Presentation Overview. Process Field Study design Sample of findings Audit observation findings Measure installation observation findings - PowerPoint PPT Presentation
Citation preview
1 Managed by UT-Battellefor the Department of Energy
Jackie BergerAPPRISE
Quantitative Findings from On-Site Evaluation of Energy Efficiency Program Service Delivery
2 Managed by UT-Battellefor the Department of Energy
Presentation Overview
Process Field Study design
Sample of findings– Audit observation findings
– Measure installation observation findings
– Final inspection observation findings
Program recommendations– Training
– Equipment
– Management
3 Managed by UT-Battellefor the Department of Energy
PROCESS FIELD STUDY DESIGN
4 Managed by UT-Battellefor the Department of Energy
Process Field Study Focus
Audit approach and implementation
Energy education delivery
Measure installation/operation of crews
Post-weatherization quality assurance inspection approach and implementation
Client interaction
Interactions among agency staff
Implications for management, procedures, and training
5 Managed by UT-Battellefor the Department of Energy
Process Field Study Approach
Go beyond anecdotes
Quantify findings across all visits– Develop check lists and rating scales
– Train weatherization experts to implement consistently
Enrich data with descriptive information
Make recommendations for program improvement based on significant issues
6 Managed by UT-Battellefor the Department of Energy
Check List ExampleAudit Air Leakage andInsulation Diagnostics
Applicable Observations
Action Taken
# %
Measured surfaces 100 94 94%
Inspected windows 100 96 96%
Inspected all accessible attics 78 69 88%
Measured insulation in all accessible attics 79 70 89%
Created access to inaccessible attics 33 3 10%
Measured insulation in exterior walls 95 47 49%
Measured insulation in basement/crawlspace 74 55 74%
Inspected for all typical bypasses 100 62 62%
Visual inspection for air sealing opportunities 100 83 83%
Used blower door while inspecting for leaks 96 64 67%
7 Managed by UT-Battellefor the Department of Energy
Rating DesignRating 1 2 3 4* 5*
% of Points needed 0%-49% 50%-74% 75%-89% 90%-99% 100%
Total points Number of Points needed for each rating
4 0-1 2 3 -- 4
5 0-2 3 4 -- 5
6 0-2 3-4 -- 5 6
7 0-3 4 5 6 7
8 0-3 4-5 6 7 8
9 0-4 5-6 7 8 9
10 0-4 5-7 8 9 10
11 0-5 6-7 8-9 10 11
12 0-5 6-8 9-10 11 12
13 0-6 7-9 10-11 12 13
14 0-6 7-9 10-12 13 14
15 0-7 8-10 11-13 14 15
8 Managed by UT-Battellefor the Department of Energy
Rating ExampleAttic Insulation Quality
Quality of Attic Insulation1 All air sealing work completed first
2 Exhaust fans vented to exterior as needed
3 Heat producing devices or systems protected from insulation contact
4 Attic checked for knob and tube wiring
5 Workers wore respirators, safety glasses, gloves, and hard hats while insulating attic
6 Insulation installed in sufficient quantity (bags per ft2) to meet R-value requirement
7 Proper insulation material chosen for attic conditions
8 Open blow insulation is level and of consistent depth
9 Attic ventilation maintained
10 Confined areas blown to dense pack
11 Proper containment used to protect client and belongings
Needs Improvement
Excellent
Mean Rating # Rated
Rating 1 2 3 4 5Total Points needed 0-5 6-7 8-9 10 11
Bold Points needed 0 0 0 6 6
Attic Insulation Quality 4% 18% 42% 19% 18% 3.3 57
9 Managed by UT-Battellefor the Department of Energy
Field StudyFlow Chart
10 Managed by UT-Battellefor the Department of Energy
Agency Selection WAP Agencies
– Almost 1,000 deliver services across the country
– 400 comprised agency billing data sample
– Field Process Study – 20 agencies in 20 different states
Selection based on:– Number of states per region based on WAP funding
– State selection based on part of region and program size
– Agencies selected to represent:
•Urban/Rural •Education
} Based on agency survey
•PY 2008 planned jobs •Training
•Measure selection •Quality Control
11 Managed by UT-Battellefor the Department of Energy
Observation Team Social Scientists
– One week observation at agency Weatherization manager and staff interviews Observations – client intake, audit, measure installation, final
inspection
– Written agency summary report
Weatherization Experts– 3-Week Observations at Agency
Observations – audit, measure installation, final inspection
– Post Observation Data Entry
Training Monthly update calls Final debriefing
12 Managed by UT-Battellefor the Department of Energy
Process Field StudyObservations
13 Managed by UT-Battellefor the Department of Energy
PROCESS FIELD STUDY AUDIT OBSERVATION
FINDINGS
14 Managed by UT-Battellefor the Department of Energy
Audit ObservationsCheck Lists
Audit preparation Audit introduction Audit home walk through Heating assessment Air conditioning assessment Ventilation assessment Water heater assessment Air leakage and insulation diagnostics Diagnostic testing* Combustion safety testing* Refrigerator assessment Lighting assessment Exit interview
*Should have been performed, was performed, performed correctly
**Described on write-up, recorded thoroughly and accurately
Ratings Audit introduction Auditor attempt to engage client Client engagement in audit Auditor communication skills Quantity of education Quality of education Exit interview Completeness of inspection Completeness of testing Quality of testing Auditor efficiency Auditor professionalism Auditor respect of client’s home Auditor safety practices Audit comprehensiveness* Planned work comprehensiveness* Audit write up*Excellent, good, poor
15 Managed by UT-Battellefor the Department of Energy
Audit Introduction
Intro LaterIntro or Later
# % # % # %
Explained WAP 74 71% 6 6% 80 77%
Explained auditor’s role in WAP 76 73% 4 4% 80 77%
Explained client’s role in WAP 39 38% 9 9% 48 46%
Asked what client hoped to get from WAP 27 26% 4 4% 31 30%
Reviewed energy bill(s) 12 12% 0 0% 12 12%
Discussed health and safety 63 61% 4 4% 67 64%
Discussed comfort 59 57% 2 2% 61 59%
Asked about problems with energy usage 34 33% 1 1% 35 34%
Out of 104 observations completed.
16 Managed by UT-Battellefor the Department of Energy
Audit Introduction Rating Design
Quality of Introduction1 Auditor introduced self2 Auditor explained purpose of visit3 Auditor explained WAP process4 Auditor reviewed energy bills with client5 Auditor asked about comfort issues6 Auditor asked about energy issues7 Auditor asked about H&S issues8 Auditor assessed client’s desired outcomes
Needs Improvement
Excellent
Rating 1 2 3 4 5Total Points needed 0-3 4-5 6 7 8Bold Points needed 0 0 0 3 3
17 Managed by UT-Battellefor the Department of Energy
Audit Introduction Rating
Needs Improvement
Excellent
Rating 1 2 3 4 5Number of Audits 32 33 25 11 2Percent of Audits 31% 32% 24% 11% 2%Mean Rating 2.2
18 Managed by UT-Battellefor the Department of Energy
Audit Diagnostic Tests
Tests Performed
Should Have Been
Should Have Been
& Was
% Performed
% Performed Correctly
Blower door testing 94 91 97% 84%
Zonal pressure diagnostics 57 24 42% 87%
Infrared camera 41 20 49% 70%
Steady state efficiency 56 44 79% 98%
Air flow test on forced air furnace 35 14 40% 93%
Proper coolant charge 5 0 0% N/A
19 Managed by UT-Battellefor the Department of Energy
Audit Write-UpRating Design
Quality of Audit Write-Up1 Did not fail to clearly document need for lead safe weatherization if needed2 Included estimated material quantities3 Specified any needed materials4 Clearly documented client health and safety issues5 Recommended measures prioritized by savings to investment ratio6 Set air leakage reduction targets for crews7 Directed crews to access areas inaccessible during the audit8 Contained clear directions for crews9 Documented client concerns/needs10 Documented client commitments if made11 Included special instructions if needed12 Clear and easy to understand
20 Managed by UT-Battellefor the Department of Energy
Quality of Audit Write-Up Rating
Number Rated
Needs Improvement
Excellent
Mean Rating1 2 3 4 5
Percent with Each Rating
Quality of Audit Write-Up 91 25% 23% 31% 12% 9% 2.6
21 Managed by UT-Battellefor the Department of Energy
Communication of Audit Findings
Audit write-ups do not convey educational opportunities
Work order not always clear or specific enough for installer to understand the desired outcome
Tools that improved information transfer– Pre-work walk through with installer
– Attaching photos of critical areas to the work order
– In process check-ups with the installers
Air sealing is a particular area for improved communication
Need for flexibility to change work order when problems are found during installations
22 Managed by UT-Battellefor the Department of Energy
Audit Strengths Inspection – 99% inspected every accessible room
Heating system – 94% inspected
Filters – 72% inspected
Ventilation– 72% inspected the kitchen
– 78% inspected the bathroom ventilation.
Water heater – 93% inspected
Air conditioning – 82% inspected outside unit.
Insulation – 89% measured all accessible attics.
Testing – 97% conducted blower door test (84% correctly)
23 Managed by UT-Battellefor the Department of Energy
Audit Opportunities for Improvement
Bathroom ventilation flow – 24% assessed
Hot water – 39% checked temperature at the faucet
Shower flow – none of the auditors measured flow
Insulation – 49% measured in exterior walls.
Blower door – 67% used while inspecting for leaks
Zonal pressure diagnostics – 42% done (87% correctly)
IR camera – 49% used (70% correctly)
24 Managed by UT-Battellefor the Department of Energy
Audit Findings
Opportunities
Utilize energy bills
Increase client engagement
Identify client-specific opportunities for reducing energy use
Auditor understanding of pressure boundaries
Auditor understanding of testing purpose and procedures
Conduct worst case draft testing
Strengths
Respect and concern for clients
Explanation of WAP program and process
Use of data collection forms
Auditors meet with contractors to explain audit findings and work scope
25 Managed by UT-Battellefor the Department of Energy
PROCESS FIELD STUDY MEASURE INSTALLATION OBSERVATION FINDINGS
26 Managed by UT-Battellefor the Department of Energy
Measure Installation ObservationsCheck Lists
Measure installation preparation Client interaction Introduction Air sealing procedures Attic insulation Thermostat installation CFLs
Ratings Windows – sealed, interior storm, exterior
storm, sash kit, repair Doors – weather stripping, treated,
replacement, installation Air sealing (8 different locations rated) Overall air sealing quality, cleanliness Attic insulation quality, cleanliness Attic hatch work quality Walk-up attic work quality, cleanliness Wall insulation quality, cleanliness, siding
preservation/interior wall finishing Basement insulation quality, cleanliness Crawl space - vapor barrier insulation,
perimeter wall insulation, vents sealing, ceiling insulation and air barrier installation
Ventilation – existing fans ducted to outside, whole house, attic, kitchen, bathroom
27 Managed by UT-Battellefor the Department of Energy
Measure Installation ObservationsRatings (continued)
Heating system – tune-up, repair, replacement, installation, vent correction Thermostat installation Air conditioning – sealing around, tune-up, replacement Evaporative cooler – repair, replacement Duct sealing, duct insulation, duct improvement cleanliness Hot water temperature adjustment, heater wrapped, pipes insulated, heater repaired,
replaced, heater venting work Subcontractor treatment of refrigerator and home Dryer venting Low cost – AC cooling coils cleaned, HVAC filters cleaned or replaced, faucet aerators,
showerheads, CO detector, smoke detector, fire extinguisher Crew efficiency, professionalism, cleanliness, safety practices Communication, attempt to engage client, client engagement, education quantity,
education quality
28 Managed by UT-Battellefor the Department of Energy
Crews and Contractors Observed
Number Percent
Crews 35 31%
Contractors 74 65%
Both 5 4%
TOTAL 114 100%
29 Managed by UT-Battellefor the Department of Energy
Measure Installation Preparation
Information Available
Number Percent
Household demographics 52 46%
Audit report 56 49%
Work order 111 97%
Materials list 73 64%
Out of 114 observations.
30 Managed by UT-Battellefor the Department of Energy
Air Sealing Rating DesignQuality of Air Sealing in Specific Places
1 Fire rated materials used around chimneys, flue pipes, recessed lighting, etc.
2 Materials integrated with existing finishes
3 Used supply-air respirators when using two-part foams
4 Blower door used to target air sealing
5 Blower door and smoke used to verify air sealing effectiveness
6 Caulk not used to seal gaps larger than 1/8”
7 Large holes and gaps covered with rigid materials
8 Area cleaned before caulk or foam applied
9 All material scraps and packaging removed from home
10 Materials installed consistently and are uniform in appearance
11 Gloves and eye protection used when cutting, using caulk or foam
12 Used hard hats when working in confined spaces
Needs Improvement Excellent
Rating 1 2 3 4 5
Total Points needed 0-5 6-8 9-10 11 12
Bold Points needed 0 0 0 5 5
31 Managed by UT-Battellefor the Department of Energy
Measure Installation Air Sealing Ratings
Number Rated
Needs Improvement
Excellent
Mean RatingRating – Quality of work 1 2 3 4 5
Percent with Each Rating
Attic floor penetrations 58 10% 41% 34% 9% 5% 2.6Knee walls 19 5% 58% 37% 0% 0% 2.3Second floor rim joist 3 0% 33% 67% 0% 0% 2.7Basement penetrations to outside
24 8% 46% 46% 0% 0% 2.4
Basement to conditioned space 12 17% 50% 33% 0% 0% 2.2Crawl space to conditioned space 14 0% 64% 29% 7% 0% 2.4Crawl space to outside 9 0% 56% 44% 0% 0% 2.4Garage to conditioned space 8 0% 63% 38% 0% 0% 2.4
32 Managed by UT-Battellefor the Department of Energy
Measure Installation Strengths
Air sealing prioritized the right areas – 77 percent prioritized sealing at the top and bottom of the envelope.
Attic insulation followed best practices – 77 percent completed attic floor sealing prior to insulation, 88 percent had no gaps or voids, and 95 percent used appropriate materials.
33 Managed by UT-Battellefor the Department of Energy
Measure InstallationMid-High Ratings
Area Ratings
Window work 3.5 – 4.0
Door work 2.8 – 3.5
Attic insulation 4.0
Wall insulation 2.9
Basement insulation 3.6
Crawl space work 3.1 – 4.0
Ventilation work 4.1 – 4.8
Thermostat installation 4.1
Air conditioning work 3.0 – 4.0
Water heater work 3.1 – 4.0
34 Managed by UT-Battellefor the Department of Energy
Measure Installation Opportunities for Improvement
Air sealing diagnostics – 22 percent used the blower door to guide air sealing and 11 percent used the zonal pressure test to affirm appropriate pressure boundaries.
Air sealing – 57 percent sealed all major opportunities.
Area Ratings
Air sealing 2.4
Heating system work 2.0 – 2.5
Smoke detector installation 2.6
CO detector installation 2.0
Low Ratings
35 Managed by UT-Battellefor the Department of Energy
Measure Installation Findings
Opportunities Increased use of blower door
when air sealing
Respect for clients’ homes (booties, covering furniture)
Crew member safety
Increased assessment of HVAC contractors
Explain CFLs when installing
Client education
Strengths
Crew flexibility and dedication (above and beyond work scope)
Crew problem solving
Some very high quality work
Experienced HVAC contractors
36 Managed by UT-Battellefor the Department of Energy
PROCESS FIELD STUDY FINAL INSPECTION
OBSERVATION FINDINGS
37 Managed by UT-Battellefor the Department of Energy
Final Inspection ObservationsCheck Lists
Home walkthrough Diagnostic tests* Combustion safety tests* Draft tests* Occupant interaction
*Should have been performed, was performed, performed correctly
Ratings Inspection completeness Testing completeness Testing quality Inspector attempt to engage client Client engagement Inspector communication skills Education quantity Education quality Inspector efficiency Inspector professionalism Inspector cleanliness Inspector safety practices
38 Managed by UT-Battellefor the Department of Energy
Final InspectionOccupant Interaction
Applicable Number
Work Completed
# %
Visit expected 91 88 97%
Adult occupant present 91 87 96%
Discussed work performed 89 67 75%
Discussed energy/cost savings 89 25 28%
Discussed health & safety issues 89 26 29%
Discussed job satisfaction 90 63 70%
39 Managed by UT-Battellefor the Department of Energy
Final InspectionCO Combustion Safety Tests
Tests Performed
Should Have Been
Should Have Been
& Was
% Performed
% Performed Correctly
Heating system 54 39 72% 97%
Water heater 50 38 76% 95%
Gas stove 26 8 31% 100%
Space Heater 7 0 0% N/A
Ambient 45 18 40% 100%
40 Managed by UT-Battellefor the Department of Energy
Number Rated
Needs Improvement
Excellent
Mean Rating1 2 3 4 5
Percent with Each Rating
Quality of Testing 88 16% 17% 19% 13% 35% 3.3
Final Inspection Testing Quality Rating Design
Quality of Testing
1 Inspector test equipment calibration current
2 Tests done correctly
3 Test results used to grade work quality
4 Inspectors proficient in operating test equipment
5 Combustion appliances disabled during blower door and duct testing
41 Managed by UT-Battellefor the Department of Energy
Final Inspection Strengths
The inspectors discussed the work that was performed – 75 percent discussed the work that was performed with the client.
Inspections were complete – 90 percent examined all accessible rooms.
Blower door testing was usually conducted – 85 percent conducted this test (85 percent of those performed correctly.)
42 Managed by UT-Battellefor the Department of Energy
Final Inspection Opportunities for Improvement
Zonal pressure test – 33 percent performed the test (87 percent done correctly.)
IR camera – 44 percent used the camera (95 percent done correctly.)
Combustion safety – 31 percent performed this test on the gas stove (100 percent done correctly.)
Worst case draft test – 60 percent performed this test on the heating system (92 percent correctly.)
43 Managed by UT-Battellefor the Department of Energy
Final Inspection Findings
Opportunities
Increased client education– Explain measures installed– Reinforce client action plan
Improved testing quality
Increased assessment of installation quality
Reduced use of auditor to conduct final inspection – additional perspective
Strengths
Good job when following work order closely
Time saved when contractor attends
Information to clients – who to call if they have problems
Referrals to additional assistance programs
44 Managed by UT-Battellefor the Department of Energy
SUMMARY
45 Managed by UT-Battellefor the Department of Energy
Average RatingsBy Agency and Overall
46 Managed by UT-Battellefor the Department of Energy
Training Needs
Energy bill utilization – to assess needs and educate clients Building science fundamentals Critical thinking – assessing unique situations Combustion safety testing Zonal pressure testing IR camera use Ventilation assessment Safe work practices – worker safety and lead safe work Interviewing skills – how to understand the client’s needs Client education
47 Managed by UT-Battellefor the Department of Energy
Equipment Needs
GPS
IR camera
Boroscope and fiber optic scope and video
Personal safety equipment
48 Managed by UT-Battellefor the Department of Energy
Management Opportunities
Standards and procedures
Policy manuals
Forms and checklists