Detailed Information on the
Oil Technology Assessment

Refocus the program on longer-term, high-risk research that will provide public benefits that otherwise would not have occurred without the program's intervention.

Develop guidance that specifies a consistent framework for analyzing the costs and benefits of research and development investments, and use this information to guide budget decisions.

Work with the Congress to terminate the program this year.

Conduct periodic, independent evaluations of all program components in order to focus program activities and target resources toward achieving relevant and beneficial outcomes that otherwise would not have occurred without the program intervention.

Identify quantitative test criteria that will be used to validate all Current Year and Budget Year performance goals when test results become available.

Measure: Additional economically recoverable domestic oil (annual incremental additional billion barrels (Bbbl) of oil).

Explanation:This measure is the cumulative total economically recoverable oil resource added from existing and expected projects. Estimates assume level funding at the FY04 President's request ($15 million/year) through 2025. The baseline production is the AEO 2003 Reference case production forecast and price assumptions. Program benefit estimate is based upon the National Energy Modeling System (NEMS). The targets for this measure reflect the cumulative total output through 2025 from the NEMS model analysis. This analysis is to be repeated each year to obtain a comparative actual value reflective of the R&D success.

Measure: Additional economically recoverable domestic oil (annual incremental additional million barrels (MMbbl) of oil).

Explanation:This measure is the annual economically recoverable oil resource added by the program. Since project results are not known in advance, the annual addition will be calculated based upon the technology project impacts completed during the previous year. Computer models will calculate these additions and, where possible, supplement them with actual data. The target numbers are based upon level funding starting with the President's FY04 budget ($15 million/year) and the AEO 2003 reference price track.

Measure: Administrative costs as a percent of total program costs.

Explanation:This "overhead rate" measure is not a true efficiency measure but is a meaningful surrogate used for all DOE applied R&D and related programs. The objective is to maintain a reasonable overhead rate for effective operation while ensuring that the vast majority of funds address the program purpose. Administrative costs include all Program Direction costs plus costs for supporting activities and analysis funded through programmatic appropriations, including program support. The targets and actuals represent corporate figures (i.e., for all Fossil Energy Research and Development (R&D)) because some Fossil Energy R&D Program Direction costs are difficult to parse at the program level in a meaningful way. Appropriation levels for Fossil Energy R&D programs and for Fossil Energy R&D Program Direction directly affect whether the target is achieved.

Is the program purpose clear?

Explanation: The program's purpose is to enhance U.S. energy security by managing and funding oil exploration and production (E&P) research; ensuring that oil technology that produces public benefits is utilitzed to the advantage of US producers in the market; and supporting the development of information and policy options that benefit the American public. Program areas include Enhanced Oil Recovery/CO2 Injection, Domestic Resource Conservation, and Environmental Science.

Evidence: Exploration and Production and Environmental Product Plans October 2002; Oil and Gas Product Plan October 2002; Microhole Road Map Workshop Summary at www.npto.doe.gov/news/microholetech.html; budget documentation (e.g., FY 2004 Congressional Justification), National Energy Policy.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program promotes national energy security through enhanced oil recovery and increases the supply of energy by increasing domestic production.

Evidence: Budget documentation and program plans.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: The program funds projects comparable to those funded by private industry, and generally for the direct benefit of private industry.

Evidence: DOE's FY 2004 Research and Development (R&D) Investment Criteria submission for Oil E&P states "The independent operator's business model approach (including the largest independents) is to "buy technology from the service companies' as needed." This illustrates that a maket for these technologies exists, and that DOE research is often duplicative of, or competes with and potentially crowds out, private investment.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There is no evidence that an alternative model would be more efficient or effective. The program uses a combination of technology development, risk assessment, regulatory streamlining tools, and regulatory impact analysis to address all aspects of high priority environmental issues. The program is currently investigating methods of repayment of R&D support funding as part of its refocusing effort. Additionally, the program has refocused towards longer-term efforts and away from downstream projects, especially in the Effective Environmental Protection program.

Evidence: Budget documents.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: DOE has not presented information regardijng its R&D investments at a detailed level discussing variables such as years to commercialization, public benefits, technological risk, cost share or plotting economic, environmental and/or security benefits.

Evidence:  

Does the program effectively articulate potential public benefits?

Explanation:  

Evidence:  

If an industry-related problem, can the program explain how the market fails to motivate private investment?

Explanation:  

Evidence:  

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The long-term goal is to increase economically recoverable oil resource base by 2.3 billion barrels by FY 2025. (The baseline is the AEO 2003 Reference case.)

Evidence: See the "Measures" section of this PART.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has established ambitious targets and timeframes. Projects last three to five years, with new competition required if a decision is made to move to the next phase. New proposals are judged against other new proposals.

Evidence: See the "Measures" section of this PART.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: Achievement of annual measures (as well as long-term goals) is based on measurement using the TORIS/NEMS models. Detailed further analysis is necessary to ensure that the critical assumptions in these models are valid and transparent, and to document any resource base expansion attributable to DOE.

Evidence: See the "Measures" section of this PART.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: Targets for annual measures are ambitious as required to achieve the long-term measures.

Evidence: See the "Measures" section of this PART.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Projects within the Oil Technologies portfolio are designed with annual and project life-cycle goals and milestones that are geared to meeting the annual and long-term program goals. Each cooperative agreement between DOE and participating partner has detailed milestones and key decision points. These milestones are reviewed annually between DOE management and the performing organization, as well as at all project funding decision points. If a partner is not meeting stated requirements, then the decision is made not to go into the next budget period of that cooperative agreement. A strong measure of a partner commitment to the program goals is their cost share that is 50 percent or greater for demonstration projects (i.e. Independent Program Field Demos) and 20 percent for Research and Development projects.

Evidence: Program solicitations; Joule Quarterly Milestones; PRoMIS database; PART measures.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The Oil E&P Program received external/independent review as part of the "Energy Research at DOE: Was it Worth It?" National Research Council (NRC), July 2001. The NRC is beginning a review of Fossil Energy program benefits (including the Oil Program). However, on an ongoing basis, the program relies on industry review to evaluate effectiveness and performance. These efforts do not meet the requirements for independent review, and the program will explore including third-party review.

Evidence: National Academy of Sciences/Nation Research Council report: "Energy Research at DOE: Was it Worth It?" (2001)

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Department has not submitted budget documents linking performance goals to resource levels in a complete of transparent manner.

Evidence: Budget documents.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Improvements in benefits modeling and efforts to connect long- and short-term goals through the JOULE performance tracking system are concrete steps that could help planning efforts. In FY 2003, the Oil Upstream Program was refocused to two technology areas (Exploration & Production and Domestic Resource Conservation), and the Gas and Oil Environmental Program eliminated its downstream activities and focused on two major research areas: water management and access to petroleum resources on Federal lands.

Evidence: Fossil Energy Top-to-Bottom review; Budget documentation; Integrated performance measures; Public Workshop Proceedings ; JOULE quarterly reports.

If applicable, does the program assess and compare the potential benefits of efforts within the program to other efforts that have similar goals?

Explanation: The program did not submit information regarding its R&D investments that demonstrates how DOE prioritizes programs based on potential benefits.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: Program did not submit R&D Investment Criteria information demonstrating how risk, years to commercialization, etc. are used in prioritizing. However, National Energy Modeling System (NEMS) is used to determine which projects should get priority based on R&D success and supply impact.

Evidence: Product plans; NEMS models.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Program contracts and cooperative agreements clearly outline the major milestones and performance requirements that the participating partner must meet. These milestones are tracked and performance measured through quarterly and other technical reporting requirements. This information is available through the Project Management Information System (ProMIS) and the publicly available extraction from PROMIS, the Fossil Energy Research Database (FRED). Accomplishments and key milestones are reported weekly to the Laboratory Director. Significant accomplishments are transmitted to the Assistant Secretary of Fossil Energy (FE) and are reported in technical fact sheets (TechLines) available to the public on the DOE/FE web site. Joule is used as a performance management tool to track results on a quarterly basis.

Evidence: The JOULE submittal with results collected and posted by the project managers includes reports, contract activity, and technological developments; retroactive metrics analyses; and periodic peer reviews. (For example, in FY 2003 this included the new oil and gas modeling system, the microhole work, and interagency work in synthetic based muds, produced water, and Federal lands.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: The National Energy Technology Laboratory (NETL) has identified a schedule of incentives holding key product personnel responsible for results under their control. It was one of only two organizations in the entire Federal governement to win the Office of Personnel Management's (OPM) Pillar Award for outstanding efforts in linking performance to accountability.

Evidence: OPM's Pillar Award for linking performance to accountability.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Funds are obligated based on an annual Program Implementation Plan approved by DOE management and consistent with appropriations. Essentially all funds are obligated in the budget year. Cooperative agreement spending is audited as required by procurement rules.

Evidence: Financial reports, Budget documents.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Fossil Energy Top-to-Bottom Review identified organizational changes to reduce layers of management, reduce the manager-to-employee ratio and to more closely tie management to program goals. However the program has not demonstrated that dollar savings have accrued from this reorganization. The program provided no other evidence of administrative/program delivery effciencies or that cost effectiveness measures are in place.

Evidence: Top to Bottom Review completed in the year 2002.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program is fairly well coordinated with other programs with similar goals. For example, the program has entered into a Memorandum of Understanding with related programs at the Department of the Interior to address technical concerns related to oil and gas drilling on federal lands that have resulted in access limitations and/or delays.

Evidence: Product plans; Federal Leadership Forum; memoranda of understanding with the Department of the Interior's Bureau of Land Managmeent and the Mining and Minerals Service; IEA Cooperative Agreement on Enhanced Oil Recovery.

Does the program use strong financial management practices?

Explanation: DOE received a clean audit in FY2001 and FY2002 with no known program deficiencies. Several computer based project management controls are in place to assist in financial management. Systems exist both on the financial side and the project management side. In addition, individual contract specialists keep detailed files of primary records.

Evidence: DOE annual Performance and Accountability reports; contract files.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Top-to-Bottom review will be implemented throughout Fossil Energy. As a result, there has been increased use of R&D investment criteria and internal/external project review, and improved performance measures and implementation of performance tracking systems (JOULE/PRoMIS). These measures will continue to improve management performance.

Evidence: Fossil Energy Top-to-Bottom Review (2002); Budget documents; JOULE results; President's R&D Investment Criteria.

Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained?

Explanation: Approximately 90% of program projects are selected on a competitive basis.

Evidence: Information on percentage of funds earmarked, and subject to competitive review.

Does competition encourage the participation of new/first-time performers through a fair and open application process?

Explanation:  

Evidence:  

Does the program adequately define appropriate termination points and other decision points?

Explanation:  

Evidence:  

If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables and required capability/performance characteristics and appropriate, credible cost and schedule goals?

Explanation:  

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: Incremental oil production of about 2% of domestic oil consumption over the 22 years of the study period (2.3 billion barrels additional 1978-2000).

Evidence: NRC/NAS Report: "Energy Research at DOE Was it Worth It?" (2001)

Does the program (including program partners) achieve its annual performance goals?

Explanation: Changes to correct deficiencies include the establishment of the JOULE performance tracking system resulting in a 100% performance score FY 2003 year-to-date. Based on the AEO 2003 price track, the Oil program will develop technologies that will be used to increase domestic oil supplies in an environmentally friendly manner and to contribute to the Nation's energy security by adding 60 million barrels* in economically recoverable oil resources in FY 2005.

Evidence: Oil and Gas Environmental Program Metrics: 2000 Analysis and Results; JOULE database; Environmental reverse metrics; Computer model results; Scored Met Goal in the 2002 GPRA activities and have met all goals in FY2003 Joule system through the 3rd quarter; Report on Analysis of Field Applications Technology.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: The program has not demonstrated improved efficiencies or cost effectiveness in achieving program goals.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation:  

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Incremental production attributed to program efforts estimated by NAS/NRC. However, NAS/NRC also stated that it is difficult to accurately attribute DOE contributions versus private industry contributions.

Evidence: NRC/NAS report: "Energy Research at DOE Was it Worth It?" (2001)

If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules?

Explanation:  

Evidence: