National Retrospective Evaluation of the Weatherization Assistance Program (WAP)

Overview

INTRODUCTION

At long last, the retrospective evaluation of the U.S. Department of Energy’s Weatherization Assistance Program (WAP) is moving ahead. WAP is a federal program that provides grants to states, who then provide grants to local agencies to weatherize low-income homes. The previous WAP national evaluation was conducted two decades ago.

RETROSPECTIVE EVALUATION: PROGRAM YEARS 2007 & 2008

The retrospective evaluation has several components. An impact analysis will be conducted to estimate national energy savings and program cost effectiveness. Non-energy benefits will also be estimated. A comprehensive process evaluation will address program characterization, operation, training, and quality assurance. Special technical projects will focus on homes heated with bulk fuels, large multi-family buildings, and indoor air quality. Lastly, two special surveys will be conducted of occupants and weatherization staff. Each of these components is discussed below. The complete evaluation plan is publication ORNL/CON-498.

Impact Analysis

The central impact evaluation question is this: how much energy savings can be attributable to WAP in Program Years 2007 and 2008? This project is using well-known analytical approaches to answering this question. Electricity and natural gas billing histories will be collected pre- and post-weatherization for a sample of weatherized homes (the treatment group) as well as for a sample of comparable homes that were not weatherized (the comparison group). The comparison group for the PY 2007 treatment group is homes weatherized in PY 2008. Similarly, the comparison group for the PY 2008 treatment group is homes weatherized in PY 2009. A national sample of 400 local weatherization agencies will be selected to provide information on about one-third of their weatherized homes. It is expected that approximately 1000 utilities will be contacted to collect billing histories. Billing history data will be normalized using three analytic methods, including PRISM.

An important complementary evaluation question is this: how cost effective are these savings? Theoretically, all installed weatherization measures should be cost effective, since all had to meet the Savings-to-Investment-Ratio Test (SIR). Evaluations are conducted to compare theory to reality. This project will collect cost information associated with each weatherized home included in the treatment sample. Total energy savings will be divided by total costs to estimate a benefit to cost ratio for WAP.

Non-Energy Benefits

WAP is believed to deliver significant benefits beyond the reduction in energy usage, including improved health and safety and improved energy bill affordability.  The evaluation will attempt to quantify these and many other types of non-energy benefits. 

From a utility perspective, weatherization could reduce arrearages and service shut-offs. Occupants could benefit from increasing comfort, less illnesses, less homelessness, higher home values, and safer homes. Societal benefits could include reduced greenhouse gas emissions, reduced water and air pollution, water conservation, higher local employment, and increased local economic activity caused by the multiplier effect. This project will use primary data and a wide range of secondary data sources to estimate total non-energy benefits.  Additionally, the non-energy benefits will be broken into three categories – occupant benefits, utility benefits, and societal benefits.

Process Evaluation

All states and agencies will receive surveys to collect data about their operations, approaches to training and client education, and quality assurance procedures. Data gleaned from these surveys will provide a snap-shot of the pre-ARRA WAP in Program Year 2008. Respondents will also be given the opportunity to indicate strengths and weakness associated WAP during that program year.

The Process Evaluation will also include two additional studies that are designed to furnish insights regarding implementation of WAP:

  • Case Studies – The evaluation team will conduct case studies with six to ten high performing weatherization agencies to develop an understanding of how the administrative and operational procedures that are used by these agencies lead to improved weatherization outcomes; and
  • Field study – The evaluation team will conduct observations for a sample of homes treated by 25 agencies to examine how audit procedures, client education, weatherization staff training, and quality assurance affect weatherization program outcomes for sampled homes.

Special Technical Studies

It is difficult to measure the impacts of energy efficiency programs on homes that use bulk fuel data because of the many challenges associated with collecting data on bulk fuel usage.  These challenges relate to householders’ use of different fuel suppliers, incomplete tank fill-ups, and use of additional fuels. This evaluation will expend considerable resources to install sub-meters to measure energy savings in homes heated with bulk fuels. These special studies will include single-family homes heated with propane, single family homes heated with fuel oil, mobile homes heated with propane, and large multi-family buildings heated with fuel oil. Results from the bulk fuel impact analysis will be compared to results for other heating sources to determine whether similar savings are achieved.

A sample of homes will also be monitored for impacts on indoor air quality. Depending on the availability of funds, additional studies will focus on refrigerators and air conditioning. Sample sizes for treatment and control groups in these studies are in the two hundred range, with the exception of the large multi-family buildings heated with fuel oil study, which will only include 24 treatment buildings. While these studies are being conducted under the auspices of the retrospective study, they will in fact be collecting sub-metering data during WAP Program Year 2010.

Special Surveys

The original retrospective evaluation includes two additional surveys. One is the Occupant Survey. This survey has four parts: energy knowledge, non-energy benefits, occupant health, and client satisfaction.  The first three parts will be administered pre- and post-weatherization. Approximately 800 occupants will be recruited to take this survey, with a similar number of comparison homes. These data will be used for several aspects of the evaluation, including the estimation of non-energy benefits and the process evaluation.

The Weatherization Staff Survey is designed to collect data about weatherization as a career, usefulness of training, extent of training, and weatherization staff knowledge. This is a first of its kind survey. Like the special studies discussed above, these two surveys will collect data relevant to the WAP ARRA period.

Project Team

Oak Ridge National Laboratory (ORNL) is leading the evaluation and is joined by a team of independent evaluators, lead by APPRISE, the Energy Center of Wisconsin, Dalhoff & Associates, and Blasnik & Associates.

Contact

Dr. Bruce Tonn is the ORNL Principal Investigator for this project.