Develop evidence-based insights into your program’s performance through third-party process and impact evaluations. Learn how to develop effective data collection strategies and timely evaluations to identify important program achievements as well as opportunities for making program improvements.
Evaluation & Data Collection – Overview
- Description
- Stages jQuery(document).ready(function() { jQuery('#block-views-handbook-static-blocks-block-3 h2.block-title').text('Stages'); });
- Tips for Success
- Examples
- Toolbox
- Topical Resources
- Quick Links
Description
As the manager of an energy efficiency program, you need to be asking questions about the performance and outcomes of your program:
- How well is the program accomplishing its objectives?
- How effective are the marketing campaigns?
- How satisfied are participants with the available incentives, including any financing offered?
- Just how much of a difference are we making in the local home improvement market?
- Is the program cost-effective?
- How can we improve program performance?
Questions like these—about performance and outcomes—ultimately involve evaluation of the effects of a program.
There are three common types of energy efficiency program evaluations: impact, process, and market evaluations. These can include a wide range of assessment studies to determine the effects of a program.
Common Types of Energy Efficiency Program Evaluations
The three evaluation types are not mutually exclusive. While their focus may differ, they each have important interconnections.
Impact evaluation: An evaluation of the program-specific changes (e.g., changes in energy and/or demand use), directly or indirectly induced, associated with an energy efficiency program.
Impact evaluations can help you:
- Determine the extent to which your program has achieved the projected energy savings.
- Understand the non-energy benefits (or co-benefits) such as avoided emissions and job creation that directly or indirectly result from a program.
- Quantify program outcomes for use in communications and marketing and outreach.
Process evaluation: A systematic assessment of an energy efficiency program. Their purpose is to document program operations and identify and recommend improvements to increase the program’s efficiency or effectiveness for acquiring energy resources while maintaining high levels of participant satisfaction.
Process evaluations can help you:
- Identify the extent to which existing processes and practices are, or are not, contributing to your program’s performance.
- Identify ways to improve program performance.
Market evaluation: An evaluation of the change in the structure or functioning of a market, or the behavior of participants in a market, that results from one or more program efforts. Typically, the resultant market or behavior change leads to an increase in the adoption of energy efficient products, services, or practices.
Market evaluations can help you:
- Gain insights into the relevant market for your program.
- Identify how the market or market participants have changed as a result of your program.
- Understand what approaches had a positive effect on the market.
Source: Energy Efficiency Program Impact Evaluation Guide, State and Local Energy Efficiency Action Network, 2012.
Although active program management will involve data collection and analysis for continuous improvement, evaluations of impact, process, and market effects are typically conducted independently from day-to-day program management. These evaluations may be conducted by a third-party evaluator or an internal team.
Results from third-party evaluations not only provide actionable information for program improvement, they also credibly demonstrate results to funders, stakeholders, and the public. For example, a program that can show its cost-effectiveness—in terms of progress toward goals compared to program expenditures—is more likely to be viewed as successful by appropriators and stakeholders.
Internal process optimization is performed in-house and more often than formal, third-party process evaluations, so that program managers can make timely changes to improve the program. Routine internal evaluation is a critical program management tool that will help improve program performance, and thus improve the outcomes of third-party evaluations. See the Program Design & Customer Experience handbook for information on how to build internal evaluation into program management.
This handbook introduces Evaluation & Data Collection and leads to a series of handbooks exploring this area in detail. Taken together, these handbooks provide guidance for program administrators on:
- Plan evaluations, focusing on external evaluations conducted by third-party evaluators. Topics include setting the scope, schedule, and budgets; designing appropriate data collection systems and processes; developing appropriate cost-effectiveness criteria; and engaging evaluators and other partners. This handbook also discusses how to integrate evaluation with all program aspects, especially the daily activities that affect data collection quality and relevance. Internal evaluations are covered in the Program Design & Customer Experience handbooks.
- Develop the resources needed to support external evaluation activities, including implementation of tools and systems that will support data collection and quality assurance necessary for effective evaluation, and structuring of data release agreements.
- Conduct and manage evaluation activities, including working with external evaluation partners and other stakeholders, as well as with all internal organization actors with a role in affecting evaluation activities and quality assurance.
- Communicate evaluation results to contractors, funders, regulators, the public, and other audiences. This handbook discusses how to use third-party evaluations, particularly impact evaluations, to help positively influence these stakeholders and provide necessary validation of the program.
Better Buildings Neighborhood Program Evaluation
To understand how the Better Buildings Neighborhood Program affected the residential energy efficiency market and industry, the U.S. Department of Energy worked with third-parties to conduct a comprehensive impact, process, and market effects evaluation. The full suite of evaluation reports, completed in June 2015, include:
- Evaluation of the Better Buildings Neighborhood Program (Final Synthesis Report Volume 1)
- Savings and Economic Impacts of the Better Buildings Neighborhood Program (Final Evaluation Volume 2)
- Drivers of Success in the Better Buildings Neighborhood Program—Statistical Process Evaluation (Final Evaluation Volume 3)
- Process Evaluation of the Better Buildings Neighborhood Program (Final Evaluation Volume 4)
- Market Effects of the Better Buildings Neighborhood Program (Final Evaluation Volume 5)
- Spotlight on Key Program Strategies from the Better Buildings Neighborhood Program (Final Evaluation Volume 6)
For more information about the evaluation, see the Better Buildings Neighborhood Program Accomplishments webpage.
Step-by-Step
Tips for Success
Develop data collection and evaluation plans in conjunction with program design
Many Better Buildings Neighborhood Program partners found that it was important to communicate during the program design phase with organizations and individuals that will collect or supply data for the evaluation. In this way, the involved individuals and organizations understand why the data is being collected and are invested in the process. Also, involving evaluators in the design of the program ensures that the right data is collected from the start.
- The Southeast Energy Efficiency Alliance (SEEA) programs experienced challenges collecting the information needed for third-party evaluation, measurement, and verification activities. SEEA contracted a third party, Cadmus Group, to evaluate its programs’ activities, and the evaluator found that the data they used as inputs - programmatic data from the Quarterly Program reports to DOE and utility bill data to perform their analysis - did not always correlate entirely with the data that programs were collecting from homeowners and contractors. SEEA and Cadmus worked with each of the programs to fill in the gaps. In some cases the programs were able to provide this information. For others, they were unable to because it only existed in paper format or the program did not have the time or resources to provide the information. To address this issue, Cadmus developed a flexible approach to determine savings for each program, based on database reviews, billing analysis, technical desk reviews, whole-house energy modeling, and engineering analysis.
- Cadmus conducted technical desk reviews when sufficient billing data was not available, but when SEEA provided data files from simulation models or other project documentation for a sufficient sample of participant homes.
- They used whole-house energy modeling to assess the reported savings when sufficient billing or simulation data was not available, but program tracking data was complete enough to populate required model inputs.
- Engineering analysis was used to estimate savings for measures not incorporated into the models or to evaluate the applicability of data from other program locations when little or no data other than the program tracking data was available.
- When data was not available, Cadmus compared weather and participant profiles to other program locations for similarities.
- For EnergyWorks in Philadelphia, problems with data tracking became apparent during the data analysis conducted for the program evaluation. EnergyWorks determined that these problems could have been avoided by clearly articulating and communicating data collection goals to program staff and defining key program performance indicators early in the program. The program established an overarching goal of 2,000 upgrades but found that it would have also been good to have set smaller specific goals tied to performance in order to better understand trends as they were occurring over the life of the program and how these trends were impacting the overall goal of 2,000 upgrades.
Establish data sharing relationships as early as possible
Though potentially challenging, establishing relationships for sharing energy consumption data is critical for evaluating program impact on energy and cost savings. Many Better Buildings Neighborhood Program partners found success by approaching utilities during the program planning phase, or at least several months in advance of when they planned to start collecting data, to outline shared goals, assets, tools, needs and constraints. Clear and concise data requests helped speed up utilities’ response times for providing the data and alleviated utility concerns and questions regarding data needs.
- Energize Phoenix formed a partnership with the local electric utility, Arizona Public Service, while designing the program and coordinated with them throughout program development. Energize Phoenix found that understanding Arizona Public Service’s concerns and challenges related to data sharing was a key ingredient in forging a successful partnership, as was instituting a formal agreement to clarify roles and responsibilities.
- Southeast Energy Efficiency Alliance (SEEA) found that not all of their programs were successful in obtaining utility bill data. Common obstacles included that the utility did not have the technology infrastructure to easily export the information, would only release data for a fee (based on how many records were pulled), or simply did not have the time or resources to provide the information even if the program had a signed client release form from the homeowner. Among SEEA's programs, those that were most successful in obtaining utility billing information–including NOLA WISE in New Orleans, Louisiana; Local Energy Alliance Program (LEAP) in Charlottesville, Virginia; Atlanta SHINE in Atlanta, Georgia; and DecaturWISE in Decatur, Georgia - consulted with the utilities to determine what information the program needed to include in the client release form. Additionally, some programs developed a written memorandum of understanding with the utility specifying data collection and transfer roles and responsibilities. SEEA programs also found it best to make data requests to utilities on a quarterly basis to minimize the burden on the utility as many utilities do not have staff dedicated to data exporting. Some programs received data more frequently, but in these situations the utility had the means to easily pull and export data.
- When local utilities Philadelphia Gas Works (PGW) and Philadelphia Electric Company (PECO) shared customers’ energy usage data with EnergyWorks, all parties made sure that the proper data sharing requirements were observed and all parties signed the necessary forms. Philadelphia EnergyWorks built its customer data release approval language into the program’s loan application form to minimize the number of additional forms that a customer or contractor would need to handle.
- EnergySmart in Eagle County, Colorado, successfully developed partnerships with utilities during and after the Better Buildings Neighborhood Program grant period, but in hindsight found it would have been more beneficial to engage utilities prior to submitting the original DOE grant application. By not fully engaging utilities up front, EnergySmart created the following environment where the utilities are only partially included in the program and retained similar or redundant in-house services. As EnergySmart continued forward, they were able to gain the trust of the utility by offering help, data, and information. EnergySmart also shared their results with the utility’s management and board of directors. Through this gained trust, utilities were more willing to share data.
Invest in information and communications technology
Paper-based or spreadsheet-based information collection processes can be low cost to develop and easy to roll-out, but more often than not, they become cumbersome to aggregate and store the data from many sources. Many Better Buildings Neighborhood Program partners found that investing in information and communications technologies (ICT) eased program implementation and was well worth the effort because they were able to regularly monitoring progress and automate what would otherwise be time-intensive, manual processes. For more information on the future of ICT, see ACEEE’s How Information and Communications Technologies Will Change the Evaluation, Measurement, and Verification of Energy Efficiency Programs.
- Garfield Clean Energy in Garfield County, Colorado, at first used a series of Excel spreadsheets and hardcopy file folders to track participants, their energy upgrade measures, and resulting energy savings. As the number of participants reached into the hundreds, the program realized that spreadsheets did not offer the level of sophisticated searching and reporting that was needed to analyze the results of their work. They explored several online customer relationship management systems and contracted with a third-party developer to customize their selected system so that it could track building and energy data, energy upgrades, contractors, dollars spent, rebates awarded, and deemed energy savings. The customization and data entry work, which took several months to complete, enabled Garfield Clean Energy to create detailed reports based on a wide variety of reporting parameters, and to better analyze the effectiveness of program activities.
- When Enhabit, formerly Clean Energy Works Oregon, scaled up their pilot program to the entire city of Portland, it was clear to them that an IT solution was needed to meet the demands of funding agencies, media requests, and good project management from the customer perspective. Enhabit worked with software company EnergySavvy to develop a unified service delivery platform to manage the home energy upgrade process from application to completion. The software platform provides a web-based interface between homeowners, contractors, and lenders, enabling each party to document progress through the Enhabit program. The platform also streamlined data collection and analysis.
- In Boulder, Colorado, EnergySmart used spreadsheets to manage data for its predecessor energy efficiency programs. As the program expanded under the Better Buildings Neighborhood Program, it became clear that EnergySmart needed to pursue a more user-friendly, real-time, cloud-based IT system for tracking customers through the implementation process. They selected a system to allow for tracking of many metrics in a much more consistent, accurate, and organized fashion than the previously used spreadsheet. The system can be accessed in the field by EnergySmart Energy Advisors using iPads or tablets to enter basic customer information, building baseline information, assessment findings for upgrade opportunities, completed upgrades with associated energy and cost savings, rebates and financing received, and the supporting documentation. The collected data is compiled for reporting to various stakeholders, including the U.S. Department of Energy, county commissioners, and city staff and leaders. The ability of Energy Advisors to access the system in the field allows for much greater efficiency and accuracy than the static logging of data upon returning to the office.
Provide adequate time for data system development and testing
Many Better Buildings Neighborhood Program partners found that setting up their information technology (IT) systems early in the program design stage ensured that data terms and data entry procedures were consistently applied by all system users. Reaching agreement with stakeholders (e.g., contractors, lenders, marketing partners, evaluators, program staff) on what data the data system would collect, known as system requirements, and how the collected data would be used to evaluate the program helped programs ensure that the data collected was complete. Programs have also found that they receive data of the quality needed for graphs and cost-effectiveness calculations when stakeholders agree up front that the data will be used for these purposes and not just to track energy savings and expenditures.
- Be SMART Maryland found that transitioning from spreadsheet-based data collection system to a customized energy IT system was crucial to administering a multifaceted energy efficiency program with rigorous data collection requirements. Investing in their system while they were still designing their program allowed Be SMART to smoothly integrate the system into the program’s operations and to ensure quality data collection and integrity over time. Be SMART also found that while spreadsheets were useful tools in collecting data, their use in analyzing data and generating reports was limited, since the program had to go through a time-consuming consolidation process to combine data from different sources and spreadsheets.
- In Boulder County, Colorado, EnergySmart found that it took between four to six months for a database developer and coding consultant to fully develop and test the data system because of its high level of complexity and the customization required. The program also found that having actual users test the system with real inputs and real reporting requirements helped ensure better data quality and user-friendliness. In addition, EnergySmart found that before beginning database development, it was important to reach agreement among stakeholders on what reporting will be expected, and design the database to facilitate building and exporting the reports. For EnergySmart, it was important to set expectations with report recipients about the IT system’s reporting capabilities early on in the process, so recipients did not expect reports that the system was unable to produce.
Use compatible formats for data sharing and reporting, and work with partners to implement standard data exchange protocols
Many Better Buildings Neighborhood Program partners found that it was critically important to use compatible formats for data sharing and reporting with partners. Aligning data formats and collection plans with national data formats (e.g., Home Performance XML schema (HPXML), Standard Energy Efficiency Data platform (SEED), Building Energy Data Exchange (BEDES)) ensured compatibility for aggregation and reporting.
- For Arizona Public Service’s (APS) Home Performance with ENERGY STAR® Program, a lack of transparency and access to data meant it took hours each month to compile progress reports. Coordination with trade allies was difficult for similar reasons–both the utility and its contractors lacked visibility into project status and task assignment, as well as the ability to identify program bottlenecks, which impacted APS customer service. Program delivery metrics, from administrative overhead to customer and trade ally satisfaction, were lower than expected. APS then began the search for a more dynamic software platform to engage customers, track and manage projects, empower trade allies, and analyze and report results. The program needed HPXML, an open standard that enables different software tools to easily share home performance data. The new HPXML-compliant platform, EnergySavvy’s Optix Manage, resulted in higher cost effectiveness and greater satisfaction for the program, including 50% less administrative time to review and approve projects, a 66% reduction in data processing time for APS reporting, 31% less contractor administrative time to submit projects, and a three-fold increase in trade ally satisfaction. HPXML also had the added benefit that contractors can choose their own modeling software.
- The New York State Energy Research & Development Authority (NYSERDA) heard from home performance contractors and other stakeholders that a more streamlined data collection process was needed to reduce the paperwork burden and time spent on a per project basis. In response, the program launched the NY Home Performance Portal in July 2013. This web-based interface made it easier for customers to choose and apply for the home performance program and made the application process for a home energy assessment clear, fast, and simple. In 2015, NYSERDA further refined their data collection process and began processing of all projects in a web-enabled interface designed to facilitate program coordination. This new platform allowed NYSERDA to automate project approvals for 85-90% of projects. In addition, the platform supported HPXML which facilitates data sharing among multiple New York programs, thereby reducing the administrative burden for contractors participating in multiple programs. It allowed NYSERDA to automate the work scope approval process through validation of standardized data. An additional benefit of HPXML for NYSERDA was creating an open modeling software market.
- Massachusetts Department of Energy Resources (MassDOER) provides statewide oversight to energy efficient programs administered by utilities under the Mass Save brand. Originally, contractors from Conservation Services Group, Inc. and Honeywell International Inc. used audit software customized for the program in their home energy assessments. When MassSave piloted the Home MPG program, contractors were also required to generate an Energy Performance Scorecard for each home. The existing audit software, however, did not have this capability. To address this problem, software developers added the Energy Performance Scorecard capability, so the contractors could use the same software to record the audit results and generate the energy performance scorecard. Despite implementation delays, this solution allows the use of the Energy Performance Scorecards to potentially expand to statewide.
Examples
The following resources are examples from individual residential energy efficiency programs, which include case studies, program presentations and reports, and program materials. The U.S. Department of Energy does not endorse these materials.
Toolbox
The following resources are available to help design, implement, and evaluate possible activities related to this handbook. These resources include templates and forms, as well as tools and calculators. The U.S. Department of Energy does not endorse these materials.
Topical Resources
The following resources provide additional topical information related to this handbook, which include presentations, publications, and webcasts. Visit Examples for materials from and about individual programs.
Quick Links
Browse key topics that many residential energy efficiency programs need to address. Select a Quick Link below to see curated resources, including case studies, presentations, tools, calculators, templates, and more. See a full list of all Quick Links (link to Quick Links landing page). If you have suggestions for additional Quick Links, please tell us.
Community events allow for face-to-face contact with target audiences and provide a venue to deliver messages to potential customers by trusted sources.
Community-based social marketing aims to change the behavior of a target audience within a group, neighborhood, or town based on principles of behavioral psychology and community dynamics.
Cost-effectiveness tests compare the benefits of a utility or non-utility program’s investment in energy efficiency with its associated costs. The five most common tests used by public utility commissions are: the participant cost test (PCT), the utility/program administrator cost test (PACT), the ratepayer impact measure test (RIM), the total resource cost test (TRC), and the societal cost test (SCT).
Customer engagement tactics aim to increase the likelihood that a customer will participate fully in a program by forming a relationship between the customer and the program or contractor. From pledges (which provide contractor leads and engage potential customers without requiring a full commitment from them) to testimonials (which encourage homeowners to share their upgrade experience with others), engaging customers in multiple ways can help increase program successes.
Establishing written data collection agreements will clarify data ownership, privacy, permission, types of data, and procedures for transferring data a program will obtain. Data collection agreements signed by homeowners are typically required for utilities to release billing data to a program or contractor; programs might also need to establish a formal agreement with lenders for any kind of data sharing. Program administrators are responsible for protecting consumer privacy and ensuring that all relevant parties use the appropriate forms, obtain data owner permissions, and avoid the release of confidential data.
Data exchange specifications help facilitate the transfer of data between software systems used by a program and its partners or stakeholders. A standard specification for transferring data reduces the need to develop a data transfer protocol each time two systems need to transfer information. Home performance XML (HPXML) is one emerging example for transferring data collected during an in-home assessment to a program’s IT system.
Managing data quality requires establishing processes to ensure that data collected is accurate and used appropriately for evaluative purposes. Inaccurate or misleading data can invalidate evaluation results and negatively impact program performance. Strategies for data quality management include establishing data collection processes that minimize the risk of collecting bad data (e.g., pre-populated fields, testing of data collection forms/tools, clear directions and training for anyone collecting program data), desk-auditing collected data for perceived anomalies, onsite verification, and other strategies.
Energy modeling, or simulation, is the practice of using software to model the energy performance of a home or its systems. Modeling is typically used by contractors or programs to estimate the energy savings that a customer could expect from completing various energy efficiency improvements.
To predict energy savings, program administrators select among different approaches (e.g., modeled savings, deemed savings, hybrid methods) that use information about the quantity and type of building improvements and engineering estimates of expected energy savings. Energy savings estimates may differ from actual energy use because of weather, home size, number of occupants, occupant behavior, installation quality, or other factors.
Impact evaluations are the traditional approach to verifying program energy savings because they measure actual reductions in consumption by a residential energy efficiency program’s participants. Formal impact evaluations seek to isolate the attributable impacts of a program’s efforts by adjusting for weather differences and independent forces by comparison with a control group.
Incentives are tools to motivate potential or current customers or stakeholders to take a prescribed action by lowering risk and decreasing upfront costs. The incentive can be financial (e.g., rebates, limited-time offers, special interest rates) or non-financial (e.g., public recognition, prizes, awards).
Information Technology (IT) systems help program administrators efficiently and effectively track energy upgrade activities, workflow, and data involving customers, contractors, utility and financial partners, and other stakeholders. Program managers need to consider the pros and cons of buying or developing a data collection solution that will meet their specific needs for aggregating, analyzing, storing, and visualizing data for program management, as well as a source of data for evaluations.
The financial industry includes a range of lending institution types. These include community banks, credit unions, non-bank finance companies (leasing company or specialized financial institutions), national energy efficiency lenders, community development financial institutions, utility-partnering lenders, and state-chartered (state-level) bond authorities. Each lender type will have its own goals and unique perspectives with respect to financing residential energy efficiency upgrades.
Neighborhood sweeps are geographically targeted campaigns to reach a specific audience in an identified community over a defined period of time.
Energy efficiency programs provide identifiable benefits beyond energy savings, such as health and safety improvements, job creation, economic development, avoided emissions, and water savings. Quantifying these non-energy benefits may help program administrators demonstrate progress toward stated program and/or policy goals, or increase general awareness and support for program activities.
Pilot projects allow programs to gain direct experience in their markets, while testing and refining program design and processes before full-scale launch.
A process evaluation systematically assesses an energy efficiency program in order to document its operations and identify improvements that would enhance its efficiency or effectiveness, while maintaining high levels of participant satisfaction. A process evaluation may be accomplished in-house or performed by an independent third-party.
Process flow diagrams illustrate key steps, decision points, and interaction points between programs, contractors, and partners from home energy upgrade project inception to completion. They are an important tool for ensuring effective coordination at critical points in the assessment and upgrade process, and identifying opportunities to streamline program processes.
Program dashboards are tracking tools that summarize metrics for monitoring progress toward meeting program goals, objectives, and efficient program processes. For many programs, they are an important tool for assessing and improving programs over time and communicating results to partners and stakeholders.
Quality assurance ensures that upgrades meets agreed upon technical standards established by programs, contractors, and/or other partners. Strategies range from requiring certification of contractors, through processes to ensure the technical quality of installed improvements, to soliciting customer feedback once projects have been fully completed. Quality assurance protects homeowners by providing an independent review of the upgrade work performed by contractors to ensure that it meets programs' technical standards. Quality assurance also protects the reputation of a program.
A request for proposals (RFP) is often necessary to engage the services of a program implementation partner or third party evaluator. A RFP should have a well-defined scope of work and clear description of how proposals will be evaluated.
Funding for organizational and program activities can come from a variety of sources including, but not limited to, ratepayer funds collected by utilities, grants (federal, state, foundation, etc.), and income from services provided to program participants or contractors. In many cases, individual revenue streams will have specific requirements on how the funding may be spent as well as specific reporting requirements.
Public, private, and non-profit organizations often seek to work in partnership with investor-owned and municipal utilities to provide energy efficiency services. Utilities may already offer energy efficiency services that other organizations can enhance or promote, and utilities typically have access to energy consumption data that helps track program success.
Last Updated: 04/01/2016