This area of the website takes you through step-by-step processes for planning and conducting the In-progress Peer Reviews and Impact Evaluations described on this website. In general, for each type of evaluation, this "how-to" section will help you to:
- Clarify your evaluation purpose, questions, and intended use,
- Consider appropriate methods for conducting the evaluation,
- Ensure that appropriate quality assurance procedures are established,
- Provide guidance for communicating needs and expectations to an evaluation contractor, and
- Monitor the process to ensure that a defensible and useful evaluation is completed.
Remember, evaluations help managers determine if timely adjustments are needed in program design or implementation to improve the rate, or quality, of achievement relative to the committed resources. Evaluations also provide programs with the evidence they need to demonstrate impacts/ results.
Begin by selecting the type of evaluation activity in which you are interested.
- Planning and Conducting a Peer Review
- Planning and Conducting a General Program Evaluation
- Planning and Conducting a Benefit-Cost Evaluation of a R&D Program
Web pages on "how-to" guidance for Planning and Conducting a Merit Review and a Stage Gate Review will be added at a future date.
Planning and Conducting In-Progress Peer Reviews
Objective review and advice from peers—peer review—is one of the standard mechanisms for effective management of highly complex and/or technically challenging projects and programs. It is widely used in industry, government, and academia. Peer review is a powerful and effective tool for enhancing the relevance, effectiveness, and productivity of the Office of Energy Efficiency and Renewable Energy's (EERE) research, development, and demonstration (RD2) activities because it taps the experiences and insights of experts in the field.
The EERE Peer Review Guide defines in-progress peer review as:
A rigorous, formal, and documented evaluation process using objective criteria and qualified and independent reviewers to make a judgment of the technical/ scientific/business merit, the actual or anticipated results, and the productivity and management effectiveness of programs and/or projects.
The primary purpose of the EERE Peer Reviewer Guide (Guide) is to provide managers and staff with guidance in establishing formal in-progress peer review that provides intellectually fair expert evaluation of EERE Research, Development, demonstration and Deployment (RD3) and supporting business administration programs, both retrospective and prospective.
The Guide describes in detail a process for planning and conducting a peer review for an individual project, a project portfolio, subprogram, or entire program.
Below are links to key information from the Peer Review Guide, for the following areas:
- Purpose and principles
- Minimum EERE requirements
- Preparation activities
- Conducting the review
- Post-review activities
Planning and Conducting a General Program Evaluation
No matter what type of impact evaluation you are going to undertake, there are certain general steps that can be followed to ensure your evaluation produces valuable and useful information. If you devote sufficient foresight and planning to the evaluation process you will enhance the value of information garnered from an evaluation. This section of the website gives you key points and covers the 14 steps within the process outlined below:
Define Purpose
Step 1: Decide the Evaluation Objectives
Step 2: Determine the Resources Available
Step 3: Determine the Timeline for Completing the Evaluation
Define Program Logic
Step 4: Construct or Confirm a Program Logic Model
Determine Questions and Methods
Step 5: Specify the Questions the Evaluation Must Answer
Step 6: Develop a Research Design
6a: Determine the Type(s) of Research Design Needed to Answer the Evaluation Questions
6b: Select Data to Be Collected and Develop a Data Collection Plan
6c: Select the Analytical Methods for Answering the Evaluation Questions
Step 7: Identify the Information that Will Go into the Evaluation Report
Step 8: Establish the Quality-Assurance Review Process
Statement of Work
Step 9: Develop the Statement of Work and Select an Evaluation Contractor
Monitoring Implementation
Step 10: Develop an Evaluation Plan and Conduct the Evaluation Study
Step 11: Monitor the Evaluation Contractor During the Study
Use the Results
Step 12: Determine Distribution of Final Report and Results and Distribute
Step 13: Make, or Monitor the Making of, Decisions about the Program Based on the Evaluation Results
Step 14: Establish/Update Program Records for Use in Future Evaluations
Planning and Conducting Benefit-Cost Evaluations of Public R&D Programs
Below are the steps in an EERE approach to retrospective benefit-cost evaluation that includes assessing energy, economic, environmental, energy security, and knowledge diffusion impacts—all as part of the benefit-cost assessment.
It should be emphasized that depicting the evaluation approach as a series of sequential steps does not mean that the process is either linear or formulaic. Conducting a successful study requires an evaluator experienced in both the art and science of benefit-cost analysis, able to capture complex and multiple direct and indirect effects of public investment using creativity and a variety of techniques within the organization of a step-wise approach, and aware of the dynamic nature of the process.
The EERE Guide for Conducting Benefit-Cost Evaluation of Realized Impacts of Public R&D Programs covers the eight steps within the process outlined below.
Step 1: Begin an evaluation
- Identify EERE programs for study
- Define a "program cluster" for benefit-cost evaluation
- Select specific technologies within the program cluster for detailed analysis
- Characterize the program cluster, the selected technologies for detailed analysis, and other technologies in the program cluster
- Identify EERE costs of the program cluster and, if possible, of the specific technologies selected for detailed analysis
Step 2: Estimate energy and economic benefits
- Define the "next-best alternative" (or "defender technology") that would have been used in lieu of each selected technology (or group of selected technologies), and estimate the change in resource use in comparison with the alternative
- Estimate the changes in resource use in comparison with the next-best alternative, expressing the results in physical units
- Determine "additionality"—benefits attributable to EERE's investment
- Provide a separate treatment of energy effects, including fuel types and quantities
- Estimate resulting year-by-year dollars of energy, labor, and other resource savings for each selected technology or group of technologies
- Treat qualitatively any economic effects of the selected technologies not captured by the quantitative economic assessment, as well as any economic effects realized from the remaining part of the cluster
Step 3: Estimate environmental benefits
- Bring forward from Section II-2 estimated year-by-year changes in fossil energy expressed in physical units, after the next-best alternative and attribution analyses; and estimate corresponding amounts of air emissions
- Apply EPA's Co-Benefits Risk Assessment (COBRA) Model to estimate public health benefits from reduced air emissions in terms of mortality, morbidity, and year-by-year dollars of health care avoided
- Determine GHG reduction impacts
- Describe other notable environmental effects
Step 4: Estimate security benefits
- Express the estimated net reduction in fossil energy in terms of imported barrels of oil equivalent (BOE)
- Describe notable effects of the cluster technologies on the nation's energy infrastructure
Step 5: Estimate knowledge benefits
- Identify modes of knowledge benefits from the cluster investment
- Construct databases needed for analyzing knowledge creation and dissemination
- Trace pathways of knowledge flows and compare knowledge effects attributed to EERE with those of other organizations
- Identify notable knowledge outputs and innovations attributed to or linked to the program cluster
- Analyze knowledge dissemination through collaborative research
Step 6: Calculate economic performance metrics
- Combine year-by-year dollar benefits from economic resources and health cost effects and compute economic performance measures, following conventions stated in the Guide
- Characterize the extent to which the EERE cluster investment has been worthwhile from the standpoint of monetized benefits and costs and in terms of the other effects considered
Step 7: Perform sensitivity analysis
- Identify and discuss areas of major uncertainty in the analyses
- Select areas for which sensitivity analysis will be performed, and, for each, substitute alternative values for uncertain input variable and calculate results
Step 8: Report results
- Ensure the study report provides essential content characteristics by following the "Contents Checklist"