Detailed Information on the
National Ambient Air Quality Standards Research Assessment

Convene annual program reviews in which extramural expert discipline scientists and clients will assess the state of ORD science, ensure progress toward outcome goals, and determine the need for strategic mid-course adjustments to maximize program efficiency and assist with outyear planning.

Develop an annual measure that more directly demonstrates progress on toward the long-term goal of reducing uncertainty in identified research areas of high priority.

Reassess meaningfulness of current efficiency measure in light of recent National Academy of Sciences (NAS) report on efficiency measurement.

The program must develop at least one efficiency measure that adequately reflects the efficiency of the program.

Improve multi-year plan (MYP) and financial data tracking systems and procedures to better and more transparently integrate grantee and program performance with financial information.

Develop and implement adequate methods for determining progress on the program's two new long-term measures (uncertainty and source-to-health linkage measures) as well as for the new annual measure (customer survey measure).

Measure: Utility of ORD's research for assessing the linkage between health impacts and air pollutant sources and reducing the uncertainties that impede the understanding and usefulnes of these linkages.

Explanation:As science identifies causative agents in particulate matter and copollutants that profile sources, source attribution (as primary causitive elements or or as contributors to transformation products) can be used to develop more effective standards and controls. This can be achieved by developing hypotheses that assess direct and indirect sources linkages. Success for this measure will be verified through use of independent expert review. Using a well-defined, consistent methodology, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Meets Expectations, and Not Satisfactory.

Measure: Percent progress toward completion of a hierarchy of air pollutant sources based on the risk they pose to human health.

Explanation:A multi-city approach to determining linkages between pollutant sources and health outcomes will ensure that program clients target air pollutant strategies most effectively and efficiently to best protect human health and the environment. Development of a source hierarchy represents an important effort in reducing uncertainty. Percent completion will be assessed by independent expert review.

Measure: Utility of ORD's research for reducing uncertainty in the science that supports standard-setting and air quality management decisions.

Explanation:In keeping with the regulatory cycle, the advancing science aligned with the National Research Council's particulate matter Topic Areas will be evaluated for quality and relevance in National Ambient Air Quality Standards development and their utility to the State Implementation Plan process. Success for this measure will be verified through use of independent expert review. Using a well-defined, consistent methodology, the BOSC provides a qualitative rating and summary narrative regarding the performance of each Long-Term Goal. Rating categories include: Exceptional, Exceeds Expectations, Meets Expectations, and Not Satisfactory.

Measure: Percent variance from planned cost and schedule (New measure, added August 2007)

Explanation:This measure captures the ability of the program to increase cost effectiveness based on the extent to which it delivers annual research outputs relative to the amount of funds spent. Using an approach similar to Earned Value Management, the data are calculated by: 1) determining the difference between planned and actual performance and cost for each Long-Term Goal, 2) adding these data together to generate program totals, and 3) dividing the Earned Value of all work completed by the Actual Cost of all program activities

Measure: Percentage of program outputs appearing in the Office of Air and Radiation's National Ambient Air Quality Standard Staff Paper.

Explanation:The primary outputs of the program's research are publications, as well as tools and models, that are used by clients in the field to implement the National Ambient Air Quality Standards. In the Office of Air and Radiation Staff Paper, which is not yet final, program publications are expected to comprised 40% of the total numbers of papers cited. This figure was calculated as the simple number of program-supported publications cited in the Staff Paper divided by the total number of papers cited overall in each document. The program publications cited were deemed appropriate science for informing the regulatory assessment and decisions under consideration.

Measure: Percentage of program publications rated as highly cited papers.

Explanation:This metric provides a systematic way of quantifying research performance and impact by counting the number of times an article is cited within other publications. The "highly cited" data are based on the percentage of all program publications that are cited in the top 10% of their field, as determined by "Thomson's Essential Science Indicator" (ESI). Each analysis evaluates the publications from the last ten year period, and is timed to match the cycle for independent expert program reviews by the Board of Scientific Counselors. Note that prior to FY 2007 the data points for this measure were derived using a bibliometric analysis methodology that evaluated publications produced on a yearly basis, rather than produced in a ten-year window. In FY 2006, ORD and OMB agreed to 1) use a ten-year window as a consistent methodology across ORD and 2) assess the measure biannually.

Measure: Percent of planned actions accomplished toward the long-term goal of reducing uncertainty in the science that supports the standard-setting and air quality management decisions.

Explanation:Beginning in FY 2008, this measure will track the program's success in completing its planned outputs on time. Prior to FY 2008, the measure tracked success in completing both planned outputs and planned actions in response to independent review recommendations.

Is the program purpose clear?

Explanation: The purpose of the Office of Research and Development's National Ambient Air Quality Standards (NAAQS) Research Program is to support EPA's mandated responsibilities under the Clean Air Act to review and set national air quality standards. This includes performing investigations and research concerning specific problems of air pollution and to issue to regions, States, and Tribes (as well as appropriate air pollution control agencies) information they need to develop appropriate and effective air pollution control strategies. This assistance takes many forms, including recommended control techniques and cost estimates of installation and operation, associated energy requirements, emissions reduction benefits, and the anticipated environmental impact of emission control technologies. The program is designed to cover research related to all six NAAQS "criteria" pollutants (carbon monoxide, lead, nitrogen dioxide, sulfur dioxide, particulate matter, and ozone) as needed and depending on the cycle of standards revision. Currently, the program focuses almost entirely on particulate matter (PM) research, with ozone O3 research limited to atmospheric modeling where its presence and involvement is essential to accurate predictions of ambient concentrations in developing State Implementation Plans. The PM element of the program specifically provides (1) advances in scientific understanding with reduction of uncertainty regarding the adverse health effects associated with patterns of human exposure to PM that are needed to review and revise (if necessary) the NAAQS every five years; and (2) the tools and data to predict, measure, and reduce emissions and ambient air levels of PM to implement the NAAQS nationally and regionally to improve public and ecological health.

Evidence: Clean Air Act as amended (42 USC 7408) Section 103, Section 108(a)-(b); National Research Council (NRC) report I 2001 (p. 1) (one of four NRC reports titled Research Priorities for Airborne Particulate Matter, published in 1998, 1999, 2001 and 2004); NRC Clean Air Act Advisory Committee Report on Air Quality Management 2004; Revised 2003 PM Multi-Year Plan.

Does the program address a specific and existing problem, interest, or need?

Explanation: Air pollution continues to be a national issue. Particulate matter in particular continues to be a contributor to the health and environmental risks associated with air pollution. Uncertainties in PM effects and exposure still exist which need to be addressed to ensure adequate protections. The Program focuses on the highest priority gaps in scientific knowledge that need to be investigated to prepare and complete the mandated cyclic NAAQS assessments of PM health effects and exposure. The Program also provides innovations and improvements to models, tools and related databases that are necessary to enable EPA's regulatory offices, states, and tribes to develop more efficient and effective plans to implement promulgated daily and annual PM NAAQS.

Evidence: "Mounting Evidence Indicts Fine-Particle Pollution", Science March 25, 2005, Vo. 307, pgs 1858-61; Clean Air Act as amended (42 USC 7408, Section 108(a)-(b)); NRC Research Priorities for Airborne Particulate Matter report IV, 2004 (p. 22); EPA Air Trends Report - The Particle Pollution Report.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The NAAQS research program is the only authorized program designed and tasked to provide the required state of the science summary (known as a "Criteria Document") which forms the foundation for the mandated cyclic reviews and revisions of the National Ambient Air Quality Standards. It is also the only comprehensive research and development program with an established combination of in-house facilities and staff and extramural resources that support the full range of research questions including how PM causes adverse health effects, who is most susceptible to these effects, how PM levels in the ambient air can be predicted, and how to measure the important attributes of PM and precursor emissions to the extent needed to achieve the Agency's desired outcomes. The federal Committee on Environment and Natural Resources (CENR) oversees related research efforts and helps coordinate the more narrowly-focused investments of other agencies with the breadth of the EPA program. EPA co-chairs the Air Quality Research Subcommittee that meets bimonthly to coordinate the federal air pollution research effort.

Evidence: CENR Reports: Atmospheric Particulate Matter Research: Inventory of Federal Research Programs (Sept 1998); Strategic Research Plan for Particulate Matter (Dec 2002); NRC Reports I-IV: The reports acknowledge the work being conducted by others at the Federal, state, and local levels, and by private and foreign organizations, but address nearly all comments toward the EPA program, highlighting that the bulk of the effort is conducted or funded by EPA and that there is little duplication with other work. For example, from NRC I: Chpt. 6: p.106: "The research funds being invested by EPA in particulate-matter exposure and health-effects research are by far the largest single commitment of public or private resources available to address key particulate-matter research questions related to protecting public health.") Subsequent NRC documents affirm this investment and the quality and advancements resultant of the Program investment.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The NAAQS research program is currently limited in its ability to maximize efficiency and effectiveness because the statutory structure of the NAAQS authority dictates a single-pollutant focus. According to the National Research Council (NRC), this results in a scientifically-artificial separation of the research focus on one pollutant instead of air pollution generally and does not allow adequate recognition of pollutants' interrelationships in determining their risks to health. Within the current design limitations, the program takes advantage of the different capabilities of internal and external researchers by use of external grants and interagency agreements that support work by the most capable organizations in a particular area.

Evidence: NRC Research Priorities for Airborne Particulate Matter Report IV, 2004, pg 12, notes scientifically-artificial separation of research focus and inadequate recognition of interrelationships when assessing health risks. NRC Air Quality Management System study, p 76 - better strategic planning of research is needed, pg 80-83 - limitations of single pollutant approach; Draft Peer Review of the Board of Scientific Counselors: Particulate Matter and Ozone Research Program.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The main beneficiaries or "customers" of the program are the NAAQS standard-setting program in EPA's Office of Air and Radiation and the states. These clients use the NAAQS research products for the development of standards and for State Implementation Plans as well as for assessing levels of attainment with the NAAQS. Currently, the program directs over 90% of its resources to particulate matter research. This is in line with the research needs identified by the scientific, regulated, and regulatory communities. The program is designed to shift resources toward other pollutants as necessary to support the required five-year review and revision of the national ambient air quality standards. To ensure resources are devoted to the issues of highest priority, a Research Coordination Team (RCT) has been established through which progress on key activities is reviewed via weekly calls and ad hoc meetings. This team includes staff from the NAAQS research program, staff from their customer programs, and regional people. The RCT is also involved in the annual planning and strategic planning processes for the program and participates in the development of grant Request For Applications. In addition, the Office of Research and Development established a National Program Director position to more clearly coordinate and direct the activities of the NAAQS research program.

Evidence: In the report titled "The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science to Achieve Results (STAR) Research Grants Program", April 2003, the National Academies of Science favorably reviewed EPA's competitive research grants program, finding that it has yielded significant new findings and knowledge critical for decision-making process. Particulate Matter OAR Staff Paper (Draft 2/2005) demonstrates usefulness and appropriate targeting.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The program has proposed two new long-term goals (LTGs) that reflect its purpose. Both measures rely on expert external review (by the Board of Scientific Counselors - BOSC) to determine adequacy of progress on goals. One LTG measures quality and relevance of advances in science and reducing uncertainties in key research areas (for particulate matter, these were defined by the National Research Council) that affect the development and implementation of standards (NAAQS). The National Research Council links reducing uncertainties in particulate matter science as necessary to support revisions in the standards. The program generates outcome oriented products upon which the logic and rationale for improvements in the regulatory process and potential needs for new standards to protect health are built. The 2nd LTG measures progress on and reductions in uncertainty in research on links between where and how pollution is generated and the health effects of that pollution ("source to health linkages"). Research outcomes in this area can support the development of more targeted, effective and efficient control strategies and standards which would result in improved health outcomes. These new goals will be incorporated in GPRA documents as well as the program's multi-year plan. In addition the program employs an existing long-term output measure that tracks the citation rate for particulate matter-related program products, which helps to provide some indication of the usefulness of the program's research results.

Evidence: The 2005 Board of Scientific Counselors (BOSC) Report on the Particulate Matter and Ozone Research Program indicates support for focusing on the areas addressed by these new measures. Also, the 2004 NRC Research Priorities for Airborne Particulate Matter Report IV demonstrates continuing need to focus on certain priority areas covered in these measures.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program is proposing to measure "excellent", "adequate", or "inadequate" levels of achievement for both long-term outcome goals through an independent review process. OMB and the program as still developing the definitions of these levels as well as how these levels will be measured by the independent review board. A quantified method for calculating the ratings been proposed but further work is needed. Therefore, it is not possible to assess whether the proposed targets (levels) are adequately ambitious. The baselines for both measures will be defined as the relevant levels of uncertainty and gaps in knowledge in various science areas needed to advance the level of understanding and strength of the risk assessment. Using an independent review process for these measures is considered an adequate method for evaluating progress.

Evidence:

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has proposed two annual output measures that meet PART guidance. The first measure involves surveying clients to assess annual progress in meeting their pre-defined needs. It will provide information on progress toward both new long-term goals. The second measure is directly tied to progress on the source-to-health linkage long-term goal. It measures annual progress in developing a hierarchy of sources based on risk levels. An additional measure is under development that will more directly demonstrate progress on reducing uncertainty in identified research areas of high priority. In lieu of this measure, two annual output measures have been included in this PART; one tracks citation rates using a formal bibliometric analysis and the other tracks the percentage of internal program milestones completed. When the improved annual measure is implemented, these two measures will no longer be included in the PART but will be maintained as internal program performance measures.

Evidence: BOSC 2005 Report indicates program is focusing on the right areas. New measures are in line with the directions reviewed in the BOSC report.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: Though numeric targets have been established for the proposed annual goal that will involve surveying clients, the content that these targets will measure has not been defined and baseline information is not available. There is no existing information to which the targets can be compared to estimate their aggressiveness. Targets have been established for the source-to-health linkage annual measure but the baseline has not.

Evidence:

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Program partners include grantees (Particulate Matter Research Centers) and contractors as well as other federal agencies (the National Oceanic and Atmospheric Administration-NOAA) and non-government institutions (the Health Effects Institute (HEI)). Solicitations do not explicitly include the program goals and it is unclear how other partners, such as HEI, commit and work toward the goals of the program. Requests for Application's (RFA's) define research questions that the program needs assistance addressing but there is no defined connection in the RFA that demonstrates how that work will clearly make progress toward an identified program goal. Interagency Agreements with federal partners that provide necessary support functions or data have been established but do not define linkage to program goals. Goals are also not explicitly defined in contractor agreements.

Evidence: The current Request for Applications (RFAs) for the Particulate Matter Research Centers do not explicitly list program goals. ORD does recommend priorities to partner organizations, such as HEI, but no evidence is available that those organizations specifically commit to and work toward the goals of the NAAQS research program. The program has and Interagency Agreement with NOAA and its statement of work includes why the partner's assistance is needed: "In order to define and understand the impact of exposure to various air pollutants on health and welfare, a knowledge of the atmospheric pollutant transport, transformation and deposition cycle from source to receptor is needed." But, no clear link is made to the goals of the NAAQS research program to demonstrate how NOAA commits to helping them achieve program goals. Sample Contracts provided include a contract for calibration services and a contract for the development and delivery of molecular primer. These are not mission contractors that perform tasks critical to direct progress on program goals (i.e. ongoing analysis or systems support). The examples do not reflect links to defined program goals.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The program has undergone multiple independent evaluations that covered the program's design, activities, progress, and management. Since 1998, the National Research Council (NRC) has produced a series of four reports that reviewed the state of particulate matter science, identified areas of deficiencies, and made recommendations for how to adequately address the continuing PM research needs. EPA's program has been the main focus of these reviews and the program has adjusted its efforts in response to recommendations in these reports. Also, the program has established a periodic independent expert evaluation process whereby EPA's Board of Scientific Counselors (BOSC), which will complete a full evaluation of the program every three to four years. The BOSC recently completed its first evaluation of the program's quality, relevance, and performance in March 2005 and the results were positive.

Evidence: The NRC reports titled "Research Priorities for Airborne Particulate Matter", published in 1998, 1999, 2001 and 2004 and the 2000 and 2005 BOSC Reports combined met the scope, quality, independence, and frequency requirements for this question and provided useful information on management and planning.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Congressional Justifications and budget submissions to OMB incorporate program budget and performance data at an aggregate level and in a disconnected manner. At this aggregate level it is not possible to determine how changes in funding for NAAQS research activities impact progress toward established goals. It is also not possible to clearly identify how grantee funds fit with the overall budget or how the grantee accomplishments contribute to program goals. For day to day financial management, the program uses the Office of Research and Development's Integrated Resource Management System (IRMS) to track resource and performance information. This system does display a hierarchy of performance, linking high-level milestones and objectives to goals but it does not link financial data to these lower levels of performance. It is tracked at the aggregate program project level or by object class, not by programmatic activity, and it is not linked to specific performance targets. It is not possible to determine how changes in funding levels would quantifiably impact progress in meeting the short and long term targets established for those goals. It is also not clear how the program tracks grantee or contractor performance in their budgeting system and it is not evident that indirect costs are adequately identified and reported on.

Evidence: Annual Congressional Justification Multi-year plans define program expectations and performance plans. Reports from the Integrated Resource Management System (IRMS) show resources in multiple formats, including at the long-term goal level, but reports do not define how funding is linked to program activities or annual goals. IRMS also does not show how grantee funds contribute to goals versus intramural funds.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program uses the Office of Research (ORD) and Development's Multi-Year Plan (MYP) process to review strategic directions and goals, and identify and remedy strategic deficiencies. The program revises its multi-year plan regularly which helps ensure that program direction adequately focuses on meeting major research needs and is outcome-oriented. ORD implemented a new National Program Director position for the NAAQS research program with clear responsibility to better direct and coordinate program planning and management. The program's establishment of a Research Coordination Team (RCT), which ensures client participation, was another step taken to address the lack of adequate customer influence in the planning process. The RCT's involvement in the planning process ensures that the program is able to meet NAAQS standard-setting priority needs and make adjustments as needed. For example, in 2003, $3M was shifted from health to implementation research so that the NAAQS standard-setting program (which is this program's primary customer - the program that this research program is statutorily designed to directly support) could meet the deadline for their attainment assessment. Also, the Program responded to 2005 OMB PART and recommendations made by external independent reviewers by revising its long-term goals to be more outcome-oriented, instituting an independent expert review process, and targeting appropriate science to reduce uncertainties in the programmatic agenda of the client offices. External organizations have indicated that better strategic planning for air quality-related research is needed.

Evidence: The 2005 BOSC Report on particulate matter indicates the program is incorporating and aligning with recommendations for improvements: (p. 8) -"... the cross-cutting research issues, recognition of the value of the one atmosphere concept, and the need to develop clear targets and associated measures provide the foundation for the PM-O3 MYP." NRC Air Quality Management System study, p 76, highlights the need for better strategic planning of air quality-related research. NRC report "Research Priorities for Airborne Particulate Matter" published in 2004 report specifically highlights alignment of research priorities. Lastly Implementation Steering Committees have been formed in the research Labs in response to NRC concerns.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: There are no other programs suitable for comparing program benefits. In addition, this program conducts mainly basic research. Basic research programs do not have "alternatives" like applied R&D programs do and there the program is not required to compare the potential benefits of its proposals with alternatives.

Evidence:

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The NAAQS research program focuses on priorities which are based on two factors: the required five-year review cycle for the national ambient air quality standards and the identified need for research on specific NAAQS pollutants. As standards come up for review, the program shifts efforts and resources to support the development of the statutorily-required state of the science document, commonly referred to as the Criteria Document, which forms the basis for starting the NAAQS review. Where EPA and external independent groups have identified air pollution concerns that require focused research assistance, the program has established R&D efforts to align with and address those concerns. Priorities are determined by evaluating the short term needs of clients (particularly the NAAQS standard-setting program in EPA's Office of Air and Radiation) and the longer term directions (such as those recommended by the National Research Council), and are generally described in the program's multi-year plan. For Particulate Matter in particular, the program directions are derived from the broad research priorities outlined by the National Research Council, and specific project areas are prioritized. Decisions are based upon a judgment of what work will be most effective in meeting needs as defined by program goals, after comparing the expected impacts of funding or not funding competing projects. Particular consideration is given to research that addresses multiple questions, that addresses time-sensitive needs, or leverages resources from other agencies or organizations.

Evidence: Beginning in the 1998, the program significantly shifted its focus to particulate matter (PM) research in response to needs identified by external bodies. The program's multi-year plan provides evidence of prioritization and goal-setting. Other internal priority-setting is done to address resource and policy constraints, including the establishment of a "discretionary priority list" to ensure key issues are addressed. External identification of priority research areas and the close alignment with EPA-funded activities can be found in the four NRC reports entitled "Research Priorities for Airborne Particulate Matter".

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The status of intramural research is reviewed and grantees are required to report progress but there evidence or examples were not available that adequately demonstrated how the program used this information to adjust program priorities or resource allocation. Also, the program has collected all the baseline data necessary to set adequate performance targets. The program uses results from external reviews to set longer-term program direction.

Evidence:

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The program incorporates program performance into personnel performance evaluation criteria. Management is accountable for specific performance standards relating to program goals. The program also monitors progress against GPRA targets, including mid-year reviews with the Deputy Administrator. For contracts and grantees, statement of work, deliverables, costs, and schedules are written into award terms. All ORD Project Officers (POs) are responsible for seeing that the agreement is awarded and managed according to government regulations in a way that gives value to the government and public.

Evidence: Performance standards are used for all executive managers and examples were provided. Project Officer Training is also conducted to ensure proper management. The program provided sample contracts and grants agreements that outline requirements.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: It is not clear that the program establishes obligation schedules for grantees or that procedures are in place to compare expenditures with intended use. Also, insufficient evidence was available to demonstrate that the program takes action to correct problems with grantee funding. For the federal program funding, obligations and expenditures are tracked in the Agency's Integrated Financial Management System (IFMS) against the initial allocation plan. IFMS allows for the reporting and tracking of obligation rates at the program/project level of the strategic architecture. Project managers monitor obligation rates to ensure the proper utilization of resources. Funds Certifying Officials perform monthly reconciliations to monitor commitment, obligation and expenditure data for accuracy and prepare routine status of funds reports. ORD-wide status of funds reports are provided to senior management on a monthly basis.

Evidence: The program follows agency-wide guidance on the roles and responsibilities for funds control, including the monitoring and tracking of resource obligations.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program does not have an approved efficiency measure and no evidence was available that demonstrates a "regular procedures" in place to improve efficiency and cost-effectiveness. The program has taken individual actions to address efficiency and cost-effectiveness. One example of the program's efforts to achieve efficiency gains is its participation in the Office of Research and Development's Total Cost of Ownership (TCO) initiative, which builds upon guidance from the Federal Chief Information Officer's (CIO) Council and best industry practices to develop more effective, efficient ways to provide quality desktop and server management. This initiative should help with efficiency gains but it is not clear how the program's participation on this is integrated with its regular procedures. There are three components of the TCO initiative which should help ORD realize cost savings while enhancing the efficacy of conducting its science and research. The three components are Desktop Replacement, Network Operations Center, and Consolidated Call Center.

Evidence:

Does the program collaborate and coordinate effectively with related programs?

Explanation: The program works through the Committee on Environment and Natural Resources (CENR) to coordinate activities with other agencies that have related activities. EPA also collaborates with external organizations, such as NARSTO (formerly known as the North American Research Strategy for Tropospheric Ozone) and other national and international groups to ensure that it has identified and is taking advantage of opportunities for collaboration and coordination. Grant solicitations are developed with client involvement and in consultation with other federal agencies, which has lead to joint research solicitations.

Evidence: The CENR Air Quality Research Subcommittee developed an overarching research strategy in 2002 to coordinate efforts (available on the web). The program's role in this subcommittee is discussed in its 2003 Multi-Year Plan (page 8). EPA and the Department of Health and Human Services, National Institute of Environmental Health Sciences issued a joint Request For Applications on cardiovascular research related to PM (RFA-ES-03-010). In this joint effort, project and goal leveraging was used to develop a complimentary program of activity in this new area of PM research.

Does the program use strong financial management practices?

Explanation: The program has no material weaknesses as reported by the IG and has procedures in place to minimize erroneous payments. The program follows EPA's financial management guidelines for committing, obligating, reprogramming, and reconciling appropriated funds. The existing system of controls and accountability are based on GAO and other principles to ensure that improper payments are not made. The Office of Research and Development has trained Funds Control Officers (FCOs) who are responsible for certifying the availability of funds as to the correct purpose, time and amount. FCOs enter each commitment into the Agency's Integrated Financial Management System and perform monthly reconciliations to ensure the accuracy of commitment, obligation and expenditure data. In addition, resources are monitored via routine Status of Funds reports that are provided to the managers within their Labs, Centers and Offices.

Evidence: EPA's Annual Plan for FY2005, Records Schedule from Budget Automation System (BAS), Unqualified audit opinion on EPA FY04 and 03 financial statements, IFMS/IRMS reconciliation reports, and Funds Control Officer (FCO)??Job Qualifications and Training.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: EPA has instituted an independent external review process that will examine management and performance of the research program on a recurring basis. In addition, specific actions have been taken to address and otherwise ensure regular identification of deficiencies. The program will be implementing a Management Multi-Year Plan which comprises three initiatives: provide exceptional administrative services; attract, develop and retain a talented and diverse workforce; and design, evaluate, and communicate program actions effectively. Workgroups have been formed to develop action plans for each of the three areas. This effort is focused on implementing efficient and effective practices to support EPA research. A succession plan is also being developed to deal with staff and management turnover. To improve program coordination and oversight, a permanent National Program Director for Air Research has been appointed.

Evidence: Draft Management Multi-Year Plan and information on workforce planning activities was provided. Internal guidance for writing Multi-Year Plans was also made available. The results of the PM BOSC review also indicated improvements in the NAAQS research program.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: It is not clear that program's grant renewal process includes sufficient amounts of outreach to encourage new applicants or that it holds renewal applicants to adequately high standards of performance for all grantees. The bulk of particulate matter grant funds have gone to the same research centers over time. A limited amount of information on the renewal process was available. NAAQS research grants are awarded through the Office of Research and Development's competitive STAR extramural grants program, using external scientific peer reviewers to rate applications based on scientific merit. Only applications rated as "excellent" or "very good" (usually 10-20% of proposals) are considered for funding based on relevance to EPA programmatic priorities. EPA's process for soliciting and selecting proposals is articulated on its website.

Evidence:

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: Grant project officers monitor grantee performance, including submission of annual progress reports and compliance with federal requirements. Grantees provide a list of publications, presentations and other activities on an annual basis and at the end of their grant period. Particulate matter Research Centers hold annual reviews conducted by external scientific advisory committees; project officers conduct a site visit in conjunction with this review. The centers' Directors and EPA NAAQS program senior scientists meet annually to discuss progress and mutual/complementary research plans for the coming year. Workshops have been held to bring together grantees and Agency scientists to inform one another of scientific plans and results as well as upcoming policy considerations. A subcommittee of the Board of Scientific Councilors convened in March 2005 to review the entire integrated program (intramural and extramural) for progress and management.

Evidence: An interim review of PM research centers was conducted in 2003 and published in Environmental Health Perspectives (Lippmann M, et al., Vol. 11, No. 8, June 2003). The report was titled "The U.S. Environmental Protection Agency Particulate Matter Health Effects Research Centers Program: A Midcourse Report of Status, Progress, and Plans". The program follows EPA's Policy on grant compliance, review, and monitoring (EPA 5700.6) and provided evidence of post-award monitoring reports, documentation of post-award monitoring and grantee financial status reports.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: An annual progress report is submitted by each grantee and posted on EPA's National Center for Environmental Research (NCER) website. Reports from the particulate matter centers and individual grantees are distributed to EPA staff to disseminate to interested parties. These reports include summaries of progress in relation to project objectives as well as publications of research results. Grantees also present results at PM-related scientific conferences held annually. The PM Centers have organized several workshops on various PM topics that include participants from outside EPA and the Centers. Progress is further disseminated through members of the PM Centers' external scientific advisory committees and external reviewers of NCER's PM programs. Summaries of PM research accomplishments are posted on the NCER website. The program does well in disseminating the grantee research results to the scientific community.

Evidence: Grant Reports, proceedings from workshops, and other results information are available at the NCER website. Information on particulate matter research centers funded by STAR grants, including annual progress reports are also available on the web. National Academy of Sciences (NAS) in its report "The Measure of STAR: Review of the U.S. Environmental Protection Agency's Science to Achieve Results (STAR) Research Grants Program", April 2003, stated "The STAR program has been commendably aggressive in experimenting with innovative approaches to communicating the results of its funded research to a wide variety of users and audiences, but its success in these efforts has been uneven" (Executive Summary, pp. 9).

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: Adequate evidence of the following was not available: a full accounting of how both intramural and extramural funds are allocated, the program's internal processes for awarding intramural funding, and the unique capabilities where funds are not awarded competitively.

Evidence:

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: Because the baselines and targets for the new long-term goals (reduction in uncertainty and source-to-health linkage measures) have not been fully developed, progress on those measures can not be determined. For the long-term measure relating to citation rate in the key customer's product - the Office of Air and Radiation Staff Paper, progress has been made since the 1996 baseline. The program had exceeded its 2005 target by 16% and is well on its way to meeting the 2010 target.

Evidence: The program provided data supporting the calculation of progress for this measure as evidence. Also, there is evidence that the program has made progress reducing uncertainty even though adequate targets have not been established for the new uncertainty reduction measure. The "Draft Report on the Particulate Matter and Ozone Research Program Review", EPA Board of Scientific Councilors, August 11, 2005 indicates the program has made progress on several fronts, including [p. 4 "The ORD PM & O3 Research Program has resulted in significant reductions in scientific uncertainty in critical areas: (1) the distribution and dosimetry of inhaled fine and ultrafine particles, (2) the relationship of ambient, fixed-site PM monitoring to real-world human exposures, (3) the identification of susceptible subpopulations, (4) the identification of biologically plausible mechanisms of PM toxicity (including cardiovascular effects), (5) the validity of PM epidemiological studies, including in particular confounding and misclassification of exposure, as well as (6) improved emissions monitoring and air quality modeling." - numbers have been added for clarity.].

Does the program (including program partners) achieve its annual performance goals?

Explanation: Progress was made on two output annual measures. The program was a few percent short of their target on the amount NAAQS research program publications rated as highly cited papers but the program exceeded its target for the percent of planned actions that were accomplished.

Evidence: The program provided data demonstrating progress on these goals. A bibliometric analysis was provided along with reports from the Office of Research and Development's Integrated Resource Management System. The reports outlined progress on individual pre-established program milestones including status information, major products resulting from accomplishment of the milestone, and the expected impacts of the results. In addition, EPA FY 2002-2004 Annual Reports discuss completion of work products.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: There is no approved efficiency measure for this program and there is no quantified, year-to-year evidence that demonstrates improved efficiency or cost-effectiveness each year.

Evidence:

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other programs addressing NAAQS research possessing the scope of purpose and goals of the EPA program. Other Agencies or groups undertake related research projects for different purposes and often have more narrowly-focused investments. Independent private efforts that perform similar research are often undertaken to augment or otherwise support EPA's efforts. The EPA program is unique in its integration and leveraged approaches to PM research. Where international programs have potential overlap with pieces of the EPA program, there exist efforts to compliment the programs for value added. As such Memorandums of Understanding (MOU's) have been developed with the National Institute for Public Health and the Environment (RIVM) of the Netherlands and the National Research Center for Environment and Health (GSF) of Germany.

Evidence: MOU's with RIVM of the Netherlands and GSF of Germany are available.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The program has been evaluated by an NRC Committee on an on-going basis - four volumes have been produced from 1998-2004. In each case, the committees have agreed that the program is making progress and is providing the necessary information to achieve strategic goals. The program uses feedback from these reviews to improve its program implementation, measurement and management. But, work remains and there are areas where improvements are needed.

Evidence: NRC reports entitled: Research Priorities for Airborne Particulate Matter, I-IV (1998, 1999, 2001 and 2004). EPA's NAAQS research program largely follows the research portfolio set out in the first and second reports. Reports III and IV outline progress on the 10 key research areas established in the earlier reports and discuss EPA-specific contributions. In addition, the recent Board of Scientific Councilors 2005 Draft Report indicates effectiveness - (p. 5) "The Subcommittee finds that the particulate matter-ozone [NAAQS Research] program directly addresses NRC (and OMB) concerns in terms of the Agency's long term goals, the plans to meet these goals, and the ways to measure progress toward these goals. The??program has resulted in dramatic reductions in scientific uncertainty in critical areas??" Also on p. 5: "The current??program appears to provide an exceptional blend of research outputs targeted at uncertainty reduction and outcome-directed research to assist OAR [Office of Air and Radiation] in protecting public health."