Detailed Information on the
Land Protection and Restoration Research Assessment

Finalize ambitious, long-term outcome performance measures that assess the utility of the program's research products and services with respect to the outcome goals of its clients.

Develop and implement a protocol for more frequent review and use of financial and performance tracking data to improve budget-performance integration.

Reassess meaningfulness of current efficiency measure in light of recent National Academy of Sciences (NAS) report on efficiency measurement.

Measure: Percentage of Land research publications rated as highly cited publications.

Explanation:This metric provides a systematic way of quantifying research performance and impact by counting the number of times an article was cited within other publications. The criteria and the "highly cited" (top 10% of field) data rankings for this metric are provided by Thomson's Essential Science Indicator (ESI). Each analysis will evaluate the WQRP publications from the last ten year period, and will be timed to match the cycle for the expert peer review panel (BOSC).

Measure: Percentage of Land publications in "high impact" journals.

Explanation:High impact journals are an indication of quality and influence. This measure evaluates the percentage of WQRP publications that are accepted within these prestigious journals and their subsequent impact on the field. The criteria and the 'impact factor' data rankings for this metric are provided by Thomson's Journal Citation Reports (JCR). Each analysis will evaluate the WQRP publications from the last ten year period, and will be timed to match the cycle for the expert peer review panel (BOSC).

Measure: Percentage of planned outputs delivered in support of the mitigation, management and long-term stewardship of contaminated sites long-term goal.

Explanation:Annual research outputs are included in the program's Multi-Year Plan (MYP). Outputs in support of this long-term goal include reports, technologies, methods, and models related to the characterization and remediation of contaminated sites. Additional details are described in the MYP.

Measure: Percentage of planned outputs delivered in support of the manage material streams, conserve resources and appropriately manage waste long-term goal

Explanation:Annual research outputs are included in the program's Multi-Year Plan (MYP). Outputs in support of this long-term goal include reports on technologies, methods, and models to manage material streams and reduce uncertainty in assessments. Additional details are described in the MYP.

Measure: Average time (in days) for technical support centers to process and respond to requests for technical document review, statistical analysis and evaluation of characterization and treatability study plans.

Explanation:The dates of requests, due dates, response time, and customer outcome feedback will be tabulated for the Engineering, Ground Water, and Site Characterization Technical Support Centers.

Is the program purpose clear?

Explanation: The purpose of the Land Research Program is to provide timely, cutting edge, problem-driven research products to support sound science decisions by EPA offices engaged in activities to preserve land quality and remediate contaminated land for beneficial reuse. Research is federally authorized for the EPA programs associated with these activities, i.e. waste management and corrective action programs, the Superfund program and the Oil Spills program. Federal legislation describes the purpose of these research programs.

Evidence: Legislation under the Resource Conservation and Recovery Act (RCRA) authorizes research programs to support scientific needs (Section 6902 of RCRA 42 USC 82 Subchapter I, Sec. 6902 (a)). The Superfund research program was formalized in 1986 with the passage of the Superfund Amendments and Reauthorization Act (Section 9660: Research, Development and Demonstration (b)). Oil Spills program research is authorized in Oil Pollution Act of 1990: Subchapter IV-Oil Pollution Research and Development Program. The mission and objectives of this EPA research program are also discussed in the Agency's Strategic Plan (Goal 3, Objectives 3.1, 3.2 and 3.3).

Does the program address a specific and existing problem, interest, or need?

Explanation: Many contaminated sites that pose threats to human health and the environment present technically challenging issues for remediation. Current cleanup approaches at these sites are often unsuccessful and/or costly to implement and maintain. The Land Research Program addresses the public need for improved tools and methods to clean up contaminated sites, address materials management and resource conservation issues and to characterize and mitigate oil spills. Even if existing methods and technologies were capable of characterizing and remediating all contamination at these sites, the return on research investment can be substantial if research continues to find lower cost approaches.

Evidence: Cleaning Up the Nation's Waste Sites: Markets and Technology Trends - 2004 Edition (http://www.clu-in.org/download/market/2004market.pdf) provides estimates on number of sites and costs for remediation of sites in many of EPA's cleanup programs. Using current rates of identifying and cleaning up sites, the report estimates that the cleanup market will continue for about 3 decades. Reviews of contaminant remediation at specific sites or in general have been performed by the National Academy of Science and EPA's independent Science Advisory Board. Both groups support the need for research on topics contained in this research program.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Land Research Program conducts research that is specific to the needs of EPA's cleanup and resource management programs. The program also provides site-specific technical support to EPA's regional staff, a population not served by other programs, to address immediate site-specific questions through: technical support centers, advisory groups, the remedy review board, and regulatory workgroups. There are cleanup research programs in other federal agencies and state groups. However, duplication of effort is limited through formal coordination with these groups to communicate results, leverage expertise, and collaborate on complex issues.

Evidence: In a 2005, EPA's Science Advisory Board reviewed the program's research plans. They state: "We note in particular the remarkable coordination of the program's research with that of the relevant program offices and other institutions and are encouraged by the judicious use of leveraging opportunities to significantly stretch limited resources to meet more of the Agency's needs." Collaboration examples include (1) Memoranda of Understanding between EPA and the National Institute of Environmental Health Sciences; (2) joint planning groups with the Departments of Defense and Energy such as the Strategic Environmental Research and Development Program and the Environmental Security Technology Certification Program and (3) cross-agency collaboration on specific projects such as multi-media environmental modeling.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The program, including its structure and design, is periodically reviewed by independent panels convened by EPA's Office of Research and Development. Overall, the findings indicate that the Land Research Program's Multi-Year Plan for research achieves its stated purpose of providing a roadmap and framework to achieving EPA's long-term research goals related to the restoration and preservation of land.

Evidence: EPA's independent Board of Scientific Counselors conducted a review of the program in December 2005, including program design. They found that the Land Research Program has a logical and comprehensive design for producing knowledge, know-how and decision-support tools to address and mitigate known current problems. The review also highlights a concern that there may be a significant imbalance in internal and external research, with internal applied research and technical support by EPA staff being the dominant component. EPA's independent Science Advisory Board reviewed the program's multi-year research plans and found that they are programmatically and scientifically sound (EPA-SAB-05-009).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The Land Research Program effectively allocates resources to fund specific activities as established in its Multi-Year Plan (MYP). The MYP is based on the research priorities declared by the clients of the program (i.e., EPA's Office of Solid Waste and Emergency Response and the Regional offices). Products and technical support are targeted to address site-specific cleanup issues for EPA's Regional Program Managers. Research activities also support EPA's regulatory assessment and decision-making processes.

Evidence: The document "Introduction to Research at the Environmental Protection Agency" describes how EPA's Office of Research and Development develops its research agenda, including the Land Research Program. An example of how the program's products are targeted and delivered to specific cleanup problems is provided in "Contaminated Sediment Research Multi-Year Implementation Plan, 2005"

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has developed two new output measures that provide a systematic way of quantifying research performance and impact. The first measure counts the number of times an article reporting research results or products from the program was cited in other publications. The second measure evaluates "high impact," the percentage of program publications that are accepted in relevant prestigious journals and their subsequent impact on the field. "High impact" journals are an indication of quality and influence. Each analysis will evaluate the program's publications from the last ten-year period.

Evidence: For the first measure, the criteria and the "highly cited" rankings are provided by "Thomson's Essential Science Indicator (ESI)". For the second measure, the criteria and the "impact factor" rankings are provided by "Thomson's Journal Citation Reports (JCR)". The program has made significant progress on development of an outcome measure and received provisional OMB approval subject to the establishment of an acceptable evaluation methodology and targets and baselines. The Agency has committed to including these measures in its 2007 GPRA documents.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has obtained quantitative, verifiable baselines for its long-term output measures. Targets have been developed based on data gathered on citations of its research products. However, targets are not considered ambitious because the measures are outputs that have been put in place while the program completes development of its outcome measures.

Evidence: Targets are set at a 1% increase every two years for both measures.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: In their Multi-Year Plan, the Land Research Program sets out an annual schedule for completion of a set of research outputs and technical support activities. The program has two new annual output performance measures that assess the percentage of these activities that are achieved each year. Each annual measure ties directly to one of the program's two research goals and they directly support the outcome measures that are currently under development.

Evidence: The program's two research goals distinguish research in support of (1) mitigation, management and long-term stewardship of contaminated sites and (2) resource conservation and waste management. The Agency has committed to including these measures in its 2007 GPRA documents.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Baselines and targets for the annual milestone measures are created through the multi-year planning process. A limited number of research and technical support activities is negotiated with regional and program office clients. The Multi-Year Plan (MYP) is updated every 3-4 years with new outputs put into place for out-years (3-4 years in advance).

Evidence: The measure is calculated in a given year as percent completed relative to the number proposed in the MYP (previous commitments are not dropped from the MYP, even in cases of delays or non-completion). Targets of less than 100% are ambitious because output negotiations assume full utilization of available resources, and these conditions may change with time (e.g., key personnel may leave for other positions, or resources limitations may affect specific programs) and significant coordination across all organizational elements is required to meet all outputs on schedule.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program is largely carried out by in-house researchers supported by contractors and, to a lesser extent, cooperative agreement partners. All investigators are made aware of the Multi-Year Plan expectations intramurally through EPA's Office of Research and Development and the Land National Program Director. These intramural partners regularly measure and report on their performance through the program's Integrated Resource Management System (IRMS). For extramural partners, the program's goals are incorporated in the statements of work included with contract documents. Major contractors commit to work toward the annual or long-term goals of the program and regularly report their performance status through formal progress reports.

Evidence: Internally, many of the senior researchers in supporting EPA labs are involved in the Multi-Year Plan planning process, as they serve on the various teams and workgroups that communicate with the broad research community. Language contained in example extramural contracts relates contractor performance to program goals. The program also provided example progress reports from some of its major contractors.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The program is assessed every 4-5 years by the Board of Scientific Counselors (BOSC). The BOSC is a distinguished body of scientists and engineers who may be drawn from academia, industry, non-EPA government or state agencies, and the environmental community. The BOSC was established under the Federal Advisory Committee Act (FACA) to provide unbiased and independent counsel to EPA's Office of Research and Development on the management and operation of its programs. These reviews address not only the scientific relevance and quality of the program but also evaluate program performance and improvements made since the previous review. The Land Research program uses feedback from the BOSC review and report to improve its program design, management, and performance.

Evidence: The BOSC Charter, review schedule and past reports are available online at: http://www.epa.gov/osp/bosc/charter.htm. The Report of the December 2005 Land Research program review is available online at: http://www.epa.gov/osp/bosc/pdf/land0603rpt.pdf

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The Land Research Program's budget and performance goals are clearly presented as a stand-alone Program-Project in EPA's budget execution system. The program does not generate any costs or requirements that must be absorbed by other programs. Annual budget requests to Congress demonstrate the linkage between overall resources requested and annual and long-term performance goals. However, the program has not provided evidence of a system for budget-performance integration that would enable assessment of its performance-resource mix.

Evidence: EPA's annual Congressional Justification includes a stand-alone description of the program which describes how program resources will be used to achieve program goals. The Agency accounts for all direct and indirect costs, including work year costs, for this program in the annual operating plan and in the annual budget materials provided to OMB. The program's operating plan links its total resources to its annual performance targets and the Agency's long-term goals. The program's Integrated Resource Management System tracks performance, but is lacking as an integrated budget-performance system because it does not include full resource accounting.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Land Research program has taken meaningful steps to correct its strategic planning deficiencies, specifically with respect to performance measurement. The LRP has made significant progress on development of outcome-oriented long-term goals around which the Land Research Multi-Year Plan for research is structured. The program has also established recurring program-level independent, expert reviews that incorporate the R&D investment criteria of relevance, quality and performance. There have been several strategic shifts in the research program due to recommendation from independent reviews. The program was included in an Agency-wide review of the Superfund program and helped close out relevant recommendations from that study.

Evidence: Based on advice from EPA's Science Advisory Board and others, the program restructured its Multi-Year Plan for research from a risk paradigm framework to a problem-focused framework that was more accessible to clients and fostered cross-lab/center planning and collaboration. The Superfund 120-Day study (available at www.epa.gov/superfund/action/120day/index.htm) cited two recommendations for Superfund research that address strategic planning and communication of research. The program worked jointly with its program partners to address these recommendations and these deficiencies were corrected in July 2005.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The Land Research Program supports technology demonstration activities. Although the program assesses the benefits of some of its activities with respect to its own performance objectives, it has not performed a comprehensive comparison of its program benefits with alternatives, in accordance with federal research and development program investment criteria.

Evidence: The Superfund Innovative Technology Evaluation (SITE) program produces an annual report for Congress that includes an assessment of program benefits. The report is not validated or reviewed by an independent entity. There is no cost-benefit analysis of alternative technologies or approaches that address similar cleanup problems.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The program follows an internal prioritization process to guide budget requests and funding decisions. External advisory bodies such as the EPA Science Advisory Board and the Board of Scientific Counselors are consulted and their recommendations are considered in priority-setting. The Multi-Year Plan, which guides programmatic research and resource allocation, explicitly states the program's goals and identifies a focused set of priorities. The program's priorities are strongly influenced by client needs. The EPA program offices rank their research needs which are then compared to the current research program and used to adjust the program's funding decisions.

Evidence: The program demonstrates the linkage of current priorities, submitted by the EPA program offices that they serve, to a subset of activities covered in their Multi-Year Plan (MYP) by inclusion of these priorities in an appendix of the MYP.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Land Research program collects performance information quarterly from support contractors and EPA's program and regional offices. The program compiles this information in their Integrated Resource Management System (IRMS). Performance information is also routinely tracked and reported to Agency headquarters and the public. Milestone status information is used to inform the annual planning process and to update the Multi-Year Plan, but the program has not demonstrated how it informs day-to-day program management to improve performance.

Evidence: Sample quarterly reports from the Integrated Resource Management System show how milestones that support the annual performance measures are tracked. The process for collecting this information is described in quarterly memos issued to key program partners. The Land Research program annually has progress reviews to report on research performance and accomplishments including information from program partners. This information is used to revise program priorities and address resource issues in subsequent years. For example, in the FY 2005 and FY 2006 contingency planning, funds were shifted or recommended for disinvestment to reflect management decisions and program and regional office priorities.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The Land Research Program incorporates program performance into personnel performance evaluation criteria. Laboratory/Center Directors are identified as the managers who are responsible for program results. These senior managers are held accountable for specific performance standards related to program goals through mid-year and end-of-year performance reviews. Contractors are explicitly held accountable for deliverables, costs, and schedules in evaluation criteria and in the statements of work. The program's project officers are responsible for seeing that agreements are awarded and managed according to government regulations.

Evidence: Sample personnel performance evaluations incorporate program performance into personal performance evaluation. Sample contracts include evaluation criteria and prior performance information. Technical evaluation criteria for contract assessment require contract awards to consider past performance.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: The Land Research Program (LRP) has consistently obligated its funds according to its operating plan with minimal unobligated balances remaining at year-end. LRP management ensures that funds are spent for their intended purpose by allocating resources to the subordinate line organizations at the activity and object class level. LRP's partners are support contractors and collaborators. Assurance that funds are spent timely and for intended purposes is provided by project officer review of invoices to confirm that they meet contract or IAG requirements. During budget execution, LRP reports and tracks obligations and expenditures in the Agency's Integrated Financial Management System (IFMS) against the Operating Plan. However, the program does not establish prospective schedules for obligations that correspond to the resource needs of the program plan.

Evidence: As of March 31, 2006, the Land Research Program had obligated 93% of its total two-year FY2005/2006 resources. Partner funds are tracked according to EPA's Policy on Compliance, Review, and Monitoring, which requires submission of annual progress reports and compliance with federal requirements. Project Officers use the EASYLite (Electronic Approval System (http://oasint.rtpnc.epa.gov/fmc2_prv/easylite.welcome) that allows them to approve invoices on line. This system gives instant validation of account totals, ensures account balances can not be exceeded, and indicates the amount actually paid by the Treasury. Project officers review and maintain copies of the invoices and confirmation of payment (via emails) from the finance center and documentation of payments are maintained on line in the EASYLite system.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The Land Research Program uses several strategies to improve cost-effectiveness in program execution including regular analysis of technical support and administrative operations (e.g., use of electronic methods, competitive sourcing) and an annual planning process to identify candidate activities for improved efficiencies or cost effectiveness in programmatic areas. Additionally, the program assesses the cost savings achieved by clients in their application of selected technologies that were developed or demonstrated with program funding. The Land Research Program has an output efficiency measure (with baselines and targets) in place that tracks the average time for technical support centers to process and respond to technical document review, data analysis and evaluation of characterization and treatability study plans. Technical support addresses Superfund site-specific issues and provides site managers with sound science to support their assessments and decisions.

Evidence: The Land Research Coordination Team (RCT) meets regularly to review the Multi-Year Plan. The extent to which parts of the program can be combined or eliminated, thereby resulting in increased cost-effectiveness, is a criterion used in the RCT planning process. In one example of measuring technology cost effectiveness, the program compared the application costs of landfill covers designed through the Evapotranspiration Cover (ET) technology program to conventional covers to track cost savings at landfill sites. An example of administrative efficiency review is the Total Cost of Ownership (TCO) initiative (additional information available at: http://intranet.epa.gov/ordintra/orma/cio/tco/index.htm) which will consolidate computer infrastructure and maintenance. As part of EPA's Office of Research and Development (ORD), the program actively participates in competitive sourcing. ORD recently completed A-76 competitions at four of its labs/centers and within its management office during FY05. The Agency has committed to including its efficiency measure in forthcoming GPRA documents, including the 2008 Performance and Accountability Report.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Land Research Program actively coordinates research agendas with other Federal research organizations. The program coordinates its work with other agencies including the Department of Defense (DoD) in its Strategic Environmental Research and Development Program (SERDP) and the Environmental Security Technology Certification Program (ESTC), the Department of Energy (DOE), National Institute of Environmental Health Sciences (NIEHS), Department of the Interior (DOI), and state groups to communicate results, leverage expertise, and collaborate on complex issues. The program coordinates with its key EPA partners in other programs and the Regions through the Research Coordination Team which plans and implements the research agenda.

Evidence: The program provided several examples of active collaborative efforts on topical and project-specific research (contaminated groundwater and sediment remediation, multi-media modeling, oil pollution research). SERDP/ESTC planning and project selection ensures that the research programs are complementary rather than duplicative. Characterization and remediation methods for non-aqueous phase liquids (NAPLs) have been a strong focus for all of the organizations. A report, SERDP/ESTCP Expert Panel Workshop on Research and Development Needs for Cleanup of Chlorinated Solvent Sites, describes efforts undertaken by multiple agencies and academia to address recalcitrant NAPL sites. The EPA's independent Science Advisory Board commented that the program's research complements work being done at other institutions: "The Agency has clearly established a range of official and substantive research collaborations with a number of government and private interests. The financial leveraging and opportunities for enhanced diffusion of scientific information afforded by these collaborative relationships have enabled ORD to maintain technically viable research programs in an environment of declining budgets."

Does the program use strong financial management practices?

Explanation: The Land Research program follows EPA's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. Agency officials have a system of controls and accountability (EPA's Resources Management Directives System), based on GAO, Treasury and OMB guidance as well as generally accepted accounting practices (GAAP), to minimize improper payments. The program is served by Funds Control Officers (FCOs) that have documented experience and/or training in EPA's budget execution and financial management systems. The program has no material weaknesses as reported by the Office of the Inspector General (OIG) and has procedures in place to minimize erroneous payments.

Evidence: EPA's Annual Reports and Financial Statements, including audit opinions, are available at: http://www.epa.gov/ocfo/finstatement/finstatement.htm. In their latest FMFIA report (FY 2005), the Office of Research and Development (the office that houses the Land Research program) certified that management controls were adequate and reported no material weaknesses. The program provided training records for their Funds Control Officers that demonstrate competence in financial management.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The Land Research Program, housed within the Office of Research and Development (ORD), has taken several steps to address management deficiencies. In 2005, ORD began implementing its Management Multi-Year Plan (MMYP). The MMYP includes three comprehensive management initiatives intended to support the substantive activities in ORD's scientific Multi-Year Plans (MYP). The Land Research Coordination Team regularly addresses management issues, through progress reviews with the program office and the Deputy Assistant Administrator. Key management elements of the program have received independent, external review by a Board of Scientific Counselors (BOSC), an independent review panel convened regularly by EPA's Office of Research and Development.

Evidence: The Management Multi-Year Plan addresses three initiatives including (1) provision of exceptional administrative services, (2) development and retention of a talented and diverse workforce and (3) effective development of ORD's research programs. An example management issue addressed through the Land Research Coordination Team review process is action on shifts in funding to support the program's tech support centers. The funding shift action is reflected in the 2006 Congressional Justification. In 2006, the program received supplemental funds for a pilot program for quick turnaround, priority research projects and successfully implemented the program within the short timeframe allotted.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The internal program funds are distributed to high priority areas through the Research Coordination Team composed of client representatives and managers from the Office of Research and Development (ORD). However, a description of the allocation process has not been provided. A pilot program for competitive funding was tried in 2006, but there is no indication it will be continued. Internal non-competitive funding may be justified because EPA's research facilities may be able to offer unique capabilities to ORD, however, the program has not provided a sufficient explanation of the unique capabilities of these labs and centers to justify the allocation of funds through this process. External reviews show positive findings regarding the program's use of planning and peer review to generate quality products, however the reviews do not speak to any funding allocation process that ensures that funds for any particular project are awarded to the most capable entity (internal or external).

Evidence: ORD's Fact Sheet "Introduction to Research at the Environmental Protection Agency" describes the activities of the Research Coordination Team including development of the Multi-Year Plan and prioritization process for projects. It does not address how funding is allocated or researchers are selected for each project. A 2006 report of review findings by EPA's independent Board of Scientific Counselors stated "The Subcommittee believes that ORD's Land Research Program continues to generate high quality products and outputs. Quality is assured, in part, by identifying projects most useful to the clients; such prioritization program is achieved through various means including the involvement of senior management and liaisons from client offices to ORD. The Subcommittee found that the routine application of peer reviews by the ORD throughout the Land Program helps to maintain high quality output". In 2006, a small amount of funding for short term research was competed within the ORD laboratories.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Land Research Program is currently using bibliometric analyses as a proxy measure for research performance. The analysis conducted for 2005, to establish a baseline, found that over a quarter (25.3%) of the land research publications are highly cited papers. Nearly a quarter (24.3%) are classified as "high impact." The next analysis will be conducted in 2008.

Evidence: The program provided detailed backup for its analyses including tabulated listing of citations that met the "highly cited" and "high impact" criteria.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The program has shown progress in delivering its planned outputs in support of the annual measure associated with resource conservation and waste management, meeting its target of 100% in 2005 and 2006. The program did not meet its 2005 or 2006 target of delivering 100% of its planned outputs in support of the annual measure associated with long-term stewardship of contaminated sites. Delays in delivery were primarily the result of cost extensions for grants and loss of staff to the Homeland Security Research Center. The program improved its performance in 2006 by meeting 96% of its planned outputs. Because the Multi-Year Plan includes outputs from program's partners (contractors), the data reflect partner's annual performance as well.

Evidence: EPA's Integrated Resource Management System (IRMS) tracks the status of all products and services which together comprise the annual performance measures. IRMS reports for 2003-2006 show the final year-end status from which actual results are tallied.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Performance data for the program's new efficiency measure shows an average response time baseline and aggressive improvement targets over time in response to customer requests. These improvements enable regional staff to receive answers to their site specific questions sooner, and result in more expeditious delivery of research products and data to support site specific decisions. There is evidence that the landfill alternative caps program is achieving improved cost efficiencies over conventional landfill covers. Management initiatives resulted in IT savings being redirected to research and workforce efficiencies.

Evidence: The Evapotranspiration Cover (ET) technology is the technology of choice at an increasing number of Federal, state and local government landfill sites since 2001. Besides the cost savings, ET covers also consistently perform at a level equal to or greater than conventional covers and have more environmental benefits. Competitive sourcing and the Total Cost of Ownership Initiative were executed via the Management Multi-Year Plan, which resulted in IT funding savings.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Several federal agencies operate research departments that serve in-house programs and/or perform activities that, like the Land Research Program, support waste management and cleanup research. Specific parts of the program have been favorably evaluated alongside other federal research programs, for example, comparing the deployment level of technologies investigated by these programs. Additionally, a 2005 report by the National Academies of Science reviewed multi-agency practice and research for oil spill dispersants and commented favorably on EPA's program. These assessments are focused too narrowly or not at all on the performance of the Land Research Program.

Evidence: The Government Accountability Office used a technology deployment metric to compare technology demonstration programs in EPA, DOE and DOD (report available at: http://www.gao.gov/archive/1998/rc98249.pdf). The program's "Monitored Natural Attenuation of organic contaminants in ground water" protocols were utilized by 36% of the sites versus 17% using state guidance and 7% using Nuclear Regulatory Commission documents. NOAA recently commissioned a National Academy of Sciences review of the state of practice and research for oil spill dispersants. The 2005 report, Oil Spill Dispersants: Efficacy and Effects, essentially provides an external peer review of the multi-agency program.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: EPA's independent Board of Scientific Counselors (BOSC) reviewed the Land Research Program in 2005. The charge questions to the reviewing subcommittee incorporated the federal R&D criteria. The panel found that the program generates high quality products and conducts appropriately focused multi-disciplinary research. Additionally, the panel noted that the Land Research Program has a logical and comprehensive design for producing knowledge, know-how and decision-support tools to address and mitigate known current problems. In its 2005 program review, EPA's independent Science Advisory Board (SAB) found that the program's Multi-Year Plans for research are programmatically and scientifically sound. Neither the BOSC nor the SAB have evaluated the effectiveness of the program in terms of outcome performance such as utility and use of it products by decision-makers.

Evidence: The Land Program Review Subcommittee of the ORD's Board of Scientific Counselors (BOSC) conducted a program review December 13th - 15th, 2005. Their draft report presents addresses a series of charge questions that were developed by the EPA in consultation with the BOSC. These questions were structured around the Office of Management and Budget's Program Assessment Rating Tool (PART) process related to Relevance, Quality, Performance, and Leadership (OMB Circular M-03-15). The findings of the Science Advisory Board's review, are summarized in the 2005 cover letter for the "Advisory on the Office of Research and Development's Contaminated Sites and RCRA Multi-Year Plans" (May 23, 2005 letter to Stephen L. Johnson, Administrator).