Detailed Information on the
Fundamental Science and Engineering Research Assessment

The program will strengthen its performance goals and improve how it monitors performance against those goals.

The program will ensure increased timeliness of yearly project reports from investigators.

The National Science Foundation will assess potential improvements to the merit review process it uses to make funding decisions.

Measure: External validation by the Advisory Committee for GPRA Performance Assessment that NSF has demonstrated "significant achievement" in providing leadership in identifying and developing new research and education opportunities within and across science and engineering fields.

Explanation:Assessment by the external Advisory Committee for GPRA Performance Assessment of "significant achievement" in providing leadership in identifying and developing new research and education opportunities within and across science and engineering fields.

Measure: External validation by the Advisory Committee for GPRA Performance Assessment that NSF has demonstrated "significant achievement" in encouraging collaborative research and education efforts across organizations, disciplines, sectors and international boundaries.

Explanation:Assessment by the externa Advisory Committee for GPRA Performance Assessment of "significant achievement" in encouraging collaborative research and education efforts across organizations, disciplines, sectors and international boundaries.

Measure: External validation by the Advisory Committee for GPRA Performance Assessment that NSF has demonstrated "significant achievement" in enabling people who work at the forefront of discovery to make important and significant contributions to science and engineering knowledge.

Explanation:Assessment by external Advisory Committee for GPRA Performance Assessment of "significant achievement" in enabling people who work at the forefront of discovery to make important and significant contributions to science and engineering knowledge.

Measure: Increase the percentage of proposals submitted to the Research Grants program from academic institutions not in the top 100 of NSF funding recipients.

Explanation:Because the Fundamental Science and Engineering category no longer exists under NSF's new Strategic Plan, data on the measures associated with the PART Program can no longer be tracked. However, because this PART Program corresponds to the Research Grants portfolio, the Foundation has revised the measure so that it applies to that portfolio, which represents the majority of the individual investigator awards made by the Foundation.

Measure: For 70 percent of proposals submitted to the Research Grants program, be able to inform applicants about funding decisions within six months of proposal receipt or deadline, or target date, whichever is later, while maintaining a credible and efficient merit review system.

Explanation:Because the Fundamental Science and Engineering category no longer exists under NSF's new Strategic Plan, data on the measures associated with the PART Program can no longer be tracked. However, because the PART Program corresponds to the Research Grants portfolio, the Foundation has revised the measure so that it applies to that portfolio, which represents the majority of the individual investigator awards made by the Foundation.

Is the program purpose clear?

Explanation: The purpose of NSF's investments in Fundamental Science and Engineering (FS&E) is to "support the best new ideas generated by scientists and engineers working at the forefront of discovery." Broadly across the fields of science and engineering (NSF FY 2003-2008 Strategic Plan). FS&E is NSF's largest investment category. It comprises the broad, core set of research activities that ensure the vitality of a broad array of scientific and engineering fields needed for the U.S. to maintain leadership in science and engineering. This statement of purpose is derived directly from the statutes that govern the Foundation. The NSF Act of 1950, as amended, authorizes NSF to initiate and support basic scientific research and research fundamental to the engineering process. In the NSF Authorization Act of 2002 it is noted that Congress finds (1) The National Science Foundation has made major contributions for more than 50 years to strengthen and sustain the Nation's academic research enterprise that is the envy of the world. (2) The economic strength and national security of the United States and the quality of life of all Americans are grounded in the Nation's scientific and technological capabilities. NSF's investments in FS&E support these major contributions and promote continued improvement in quality of life.

Evidence: Evidence of the program purpose can be found in: NSF FY 2003-2008 Strategic Plan, page 16 (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201); National Science Foundation Act of 1950, Functions of the Foundation (frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=browse_usc&docid=Cite:+42USC1862); NSF Authorization Act of 2002, P.L. 107-378, Section 2 (frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=107_cong_bills&docid=f:h4664enr.txt.pdf).

Does the program address a specific and existing problem, interest, or need?

Explanation: NSF's investments in Fundamental Science and Engineering address a specific and existing need as spelled out in the NSF Act of 1950, as amended, and reiterated more recently in Paragraphs 4 and 6 of Section 2 (Findings), of the NSF Authorization Act of 2002. Section 2, Paragraph 4 notes that the research and education activities of the National Science Foundation promote the discovery, integration, dissemination, and application of new knowledge in service to society and prepare future generations of scientists, mathematicians, and engineers who will be necessary to ensure America's leadership in the global marketplace. Section 2, Paragraph 6 states that the emerging global economic, scientific, and technical environment challenges long-standing assumptions about domestic and international policy, requiring the National Science Foundation to play a more proactive role in sustaining the competitive advantage of the United States through superior research capabilities.

Evidence: Evidence of the specific and existing need addressed by FS&E can be found in the NSF Authorization Act of 2002, P.L. 107-378, Section 2 (http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=107_cong_bills&docid=f:h4664enr.txt.pdf).

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: FS&E is not redundant or duplicative of other efforts as it provides the principal source of Federal support for basic research at colleges and universities in many disciplines. NSF is the only Federal agency charged with promoting the progress of science and engineering research and education in all fields and disciplines. As a result of this, NSF is in a unique position to lead many interagency efforts that fall within FS&E. To avoid duplication between Federal and non-Federal organizations, NSF routinely partners with others in the support of research of mutual interest. For example, NSF leads interagency working groups for the National Nanotechnology Initiative (NNI) and Networking and Information Technology Research and Development (NITRD) efforts and collaborates with other Federal agencies in activities such as those related to climate change science programs.

Evidence: NSF has specific, statutory authority to evaluate the status and needs of the various sciences and engineering and to consider the results of this evaluation in correlating its research and educational programs with other Federal and non-Federal programs. See National Science Foundation Act of 1950, as amended, under the "Functions of the Foundation" (42 U.S.C. 1862, Sec. 3. (a)(5)); (frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=browse_usc&docid=Cite:+42USC1862); Information on the NNI is available at www.nano.gov/; Information on NITRD is available at www.nitrd.gov/; Information on climate change science programs is available at www.usgcrp.gov.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: There have been no identified major design flaws that would limit the effectiveness or efficiency of FS&E. NSF's investments in Fundamental Science and Engineering rely upon the competitive merit review process, NSF Program Officers, and external Committees of Visitors (COVs) to ensure effectiveness and efficiency. Merit review by peers has been recognized as a best practice for administering Research & Development (R&D) programs. Independent reviews by COVs and other external groups (e.g., Advisory Committees, National Science Board, National Academies of Science / National Research Council, President's Council of Advisors on Science and Technology) provide additional scrutiny of the portfolio and goals and results. FS&E program components make improvements based on recommendations received from the independent reviewers. This follows the guidance provided in the R&D Criteria, as outlined in the Office of Management and Budget/Office of Science and Technology Policy Guidance Memo. No strong evidence exists that an alternative design would better achieve FS&E's purpose.

Evidence: Evidence of the effectiveness of FS&E's design can be found in: Report to the NSB on the NSF Merit Review Process - FY 2004 (www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf); August 2004 OMB/OSTP Guidance Memo (www.ostp.gov/html/m04-23.pdf); COV Reports (www.nsf.gov/od/oia/activities/cov/covs.jsp).

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: NSF employs several mechanisms to ensure effective targeting, such as outreach activities, use of the World Wide Web, targeted e-mails, and clear statements of purpose, so that FS&E resources address its purpose of supporting the best new ideas generated by scientists and engineers working at the forefront of discovery. NSF is interested in engaging the nation's intellectual talent and in supporting the best ideas, so all program announcements and solicitations are available online to ensure open access and to inform individuals interested in proposing ideas of opportunities to do so. To ensure program focus, all announcements and solicitations contain clear statements of a program's purpose and context. Targeted outreach is accomplished through MyNSF, an electronic communications system that alerts self-identified people of specific opportunities. NSF Program Officers also conduct outreach activities at professional conferences and during visits to academic institutions. Several dozen NSF staff also participate in outreach at NSF's bi-annual Regional Grants Conference. Finally, the merit review process ensures that funding is awarded to proposals that best address the program's purpose.

Evidence: Evidence of effective targeting includes: MyNSF, (formerly NSF's Custom News Service) is available at www.nsf.gov/mynsf/; Program solicitations for all NSF programs, including those within FS&E can be found at www.nsf.gov/funding/; Report to the NSB on the NSF Merit Review Process - FY 2004 (www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf); Overhead transparencies used at the most recent Regional Grants Conference are located at www.nsf.gov/bfa/dias/policy/outreach.jsp.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Fundamental Science & Engineering has three specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program; these are listed in the 'Measures' tab. These are drawn from the objectives set forth in the NSF FY 2003-2008 Strategic Plan, and they encompass support of the best new ideas generated by scientists and engineers working at the forefront of discovery.

Evidence: Evidence of specific long-term performance measures that reflect the purpose of the program can be found in the Measures Tab; and the"Ideas" section of the NSF Strategic Plan: (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201; page 15).

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Fundamental Science & Engineering has ambitious targets and timeframes for its long-term measures, which include verifiable qualitative assessments by external experts that the program enables people who work at the forefront of discovery to make important and significant contributions to science and engineering knowledge, encourages collaborative research and education efforts, and provides leadership in identifying and developing new research and education opportunities. The assessment by external experts ensures that the goals and timeframes for these activities are appropriately ambitious and that they promote continuous improvement. The primary mechanisms for external evaluation are the annual Advisory Committee for GPRA Performance Assessment (AC/GPA) and the Committee of Visitors (COV) process. Other external guidance includes third party program assessments and Principal Investigator (PI) meetings.

Evidence: Ambitious targets and timeframes for long-term measures can be found in the Measures Tab; FY 2004 AC/GPA Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216; page 27); and FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par; page II-59).

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Fundamental Science & Engineering has two specific annual performance measures, shown in the Measures Tab, which can demonstrate progress toward achieving the program's long-term goals and the agency's strategic goals. One is an efficiency measure; the other measures broadening participation, which increases the pool of ideas and thus improves FS&E's ability to identify the best new ideas of people who work at the forefront of discovery in science and engineering.

Evidence: Specific annual performance measures demonstrating progress toward achieving long-term goals may be found in the Measures Tab.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: FS&E's baselines and ambitious targets for its annual measures are shown in the Measures Tab.

Evidence: Baselines and targets for annual measures can be found in the Measures Tab; NSF's Enterprise Information System; FY 2004 Performance and Accountability Report summary table (www.nsf.gov/publications/pub_summ.jsp?ods_key=par ; Sec. II, p. II-6).

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: Fundamental Science &Engineering obtains ongoing commitment toward its goals from all its partners by aligning program descriptions program announcements and program solicitations with those goals, using the merit review process to select proposals that demonstrate commitment to the goals, and requiring grantees to submit satisfactory template-based annual and final progress reports, subject to NSF program officer approval, as a prerequisite for continuation and/or renewal support. Continuing support (i.e., continuing grant increments) is based upon required template-based annual progress reports submitted by grantees that are subject to review and approval by NSF Program Officers before additional funds are released. To receive further support (subsequent awards), all applicants are required to include in their new proposals a report on the results of previous NSF support, which is then considered in the merit review process; in addition, a template-based final progress report must be submitted after an award ends, and no subsequent awards can be made to an applicant unless the final progress reports for all previous awards have been approved by a Program Officer.

Evidence: Evidence that all partners commit to and work toward the annual and/or long-term goals of the program include awardee project reports and Grant General Conditions (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf98gc1a). There are numerous program announcements that are issued annually by various science and engineering disciplines throughout the Foundation (www.nsf.gov/funding/). All program announcements reference NSF's two merit review criteria (www.nsf.gov/pubs/1999/nsf99172/nsf99172.htm), which are aligned with the FS&E goals. Examples of language from representative proposal solicitations or program announcements are: from NSF 05-572, Opportunities for Promoting Understanding through Synthesis (OPUS) -- "Synthesis is an essential component of scientific inquiry. Synthetic works??those that integrate information from numbers of studies??have historically been influential in describing the state of a field and in spawning new research directions. To encourage synthetic studies, the Ecological Biology, Ecosystem Science, and Population and Evolutionary Processes Clusters in the Division of Environmental Biology are establishing OPUS ?? ;" from NSF 05-570, DDDAS: Dynamic Data Driven Applications - "The research scope described here requires strong, systematic collaborations ?? Consequently, most projects proposed in response to this solicitation are expected to involve teams of researchers. Following merit review of the proposals received, projects will be selected for support by NSF, the National Institutes of Health (NIH) and the National Oceanic and Atmospheric Administration (NOAA) ... . This solicitation seeks to catalyze multidisciplinary research enabling DDDAS ?? NSF and the other sponsor agencies welcome proposals for collaborative projects involving both universities and the private commercial sector ?? International collaborations are also encouraged;" from NSF 04-586, Microbial Observatories (MO) and Microbial Interactions and Processes (MIP) - "Both MO and MIP proposals must describe how the work will make a substantial impact on scientific understanding of the diversity of microorganisms and microbial communities and their role in diverse environments ?? Proposals in either the MO or MIP category that show evidence of collaborative arrangements between academic and/or commercial groups to conduct more detailed investigations on particular microbes or microbial communities also are encouraged ?? MO and MIP encourages laboratory-to-laboratory interactions between U.S. and foreign organizations or institutions ??" Other evidence may be found in the Report to the NSB on the NSF Merit Review Process - FY 2004 (www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf).

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evaluations of Fundamental Science & Engineering component activities are conducted regularly by Committees of Visitors (COVs), the Advisory Committee for GPRA Performance Assessment (AC/GPA), Directorate and other Advisory Committees (ACs), and on an as-needed basis by other independent entities to inform and support program improvements, evaluate effectiveness and relevance, and influence program planning. On an annual basis, the AC/GPA determines whether the breakthroughs, advances and other results reported from FS&E demonstrate significant achievement towards the long-term outcome goals; each activity at NSF, including the FS&E component activities, is reviewed on a triennial cycle by a COV; and ACs generally meet several times a year. NSF's approach to evaluation was recently highlighted by GAO as an "evaluation culture--a commitment to self-examination, data quality, analytic expertise, and collaborative partnerships." Advisory Committees review Directorate performance and COV reports, and the Advisory Committee for GPRA Performance Assessment assesses performance on an NSF-wide basis for the Strategic Goals. FS&E sponsors workshops and PI meetings that help define emerging areas and knowledge gaps, in order to be responsive to the NSF mission. NSF staff and external experts conduct site visits for major activities. NSF also supports focused assessments by the National Research Council, the World Technology Evaluation Center and other organizations, as appropriate. All these activities inform NSF senior management and contribute to development of plans for the agency. NOTE: The weight of this question has been increased to reflect the importance NSF places on the conduct of independent evaluations to support program improvements and evaluate effectiveness.

Evidence: Evidence of independent evaluations begin of sufficient scope and quality may be found GAO's recent report, Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity GAO-03-454 May 2, 2003 (www.gao.gov/new.items/d03454.pdf). Other evidence includes COV reports and NSF responses (www.nsf.gov/od/oia/activities/cov/) and AC reports, including the Advisory Committee for GPRA Performance Assessment (AC/GPA) Report www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: NSF's performance structure provides the underlying framework for NSF's annual budget request. Each major NSF organization - i.e. the directorates - ties its budget directly to NSF's performance framework. This is further documented in the performance summary included in each organization's chapter of the budget, which ties the budget directly to the PART activities. With respect to presenting the resource needs in a clear and transparent presentation, the NSF budget displays resource requests by structural component and by performance goal. This presentation is based on consultations over the past year with key Congressional and OMB staff, and it also incorporates recommendations from the 2004 report on NSF by the National Academy of Public Administration. The purpose of this presentation is to highlight the matrix structure that NSF employs, with the major organizational units each contributing to the goals and investment categories established in the NSF Strategic Plan. This revised presentation contains additional information on the portfolio of investments maintained across NSF.

Evidence: Evidence showing that budget requests are tied to accomplishment of the annual and long-term goals of the FS&E program include the Executive Branch Management Scorecard (www.whitehouse.gov/results/agenda/scorecard.html); FY 2006 Congressional Justification (www.nsf.gov/about/budget/fy2006); NSF Budget Cost Performance Integration Plan (Internal Document); and National Science Foundation Governance and Management for the Future, National Academy of Public Administrators, Order Number 04-07 (www.napawash.org/resources/news/news_4_28_04.html, 71.4.192.38/NAPA/NAPAPubs.nsf/17bc036fe939efd685256951004e37f4/23f8c16a35c7eb6485256e85004b4a4f?OpenDocument).

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Although strategic planning deficiencies per se have not been noted, the Committees of Visitors (COV) process and Directorate Advisory Committees (ACs) provide valuable constructive feedback on an ongoing basis concerning areas where strategic planning can be strengthened, and in response the agency takes steps to address those issues.

Evidence: Evidence demonstrating FSE's strategic planning may be found in COV reports and NSF responses (www.nsf.gov/od/oia/activities/cov/); Directorate Advisory Committee minutes (for example: www.nsf.gov/bio/advisory.jsp; www.nsf.gov/geo/advisory.jsp; www.nsf.gov/cise/advisory.jsp); and NSF Strategic Plan FY 2003 - FY 2008 (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201).

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: NSF's investments in Fundamental Science and Engineering address unique national STEM research needs that are not under the purview of the more mission-specific federal, state or local agencies. No other program of this scope or objectives exists in the Federal Government. FS&E comprises the broad, core set of research activities that ensure the vitality of a broad array of scientific and engineering fields needed for the U.S. to maintain leadership in science and engineering. The Office of Science and Technology Policy, the National Science and Technology Council, the National Science Board, OMB, the Congress, and other policy-making bodies regularly review NSF's investments in FS&E in the context of the overall Federal investment in science and engineering. In areas where research activities may overlap, NSF coordinates its activities with that of the agencies in order not to duplicate efforts and to ensure that each agency supports those efforts most appropriate to its mission.

Evidence: Evidence of coordination include the High Energy Physics Advisory Panel (coordinates various physics activities between NSF and DOE); Astronomy and Astrophysics Advisory Committee (coordinates various NASA and NSF activities); and Ocean.US (www.ocean.us/).

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: In formulating budget requests for the various activities within Fundamental Science & Engineering, a prioritization process is used that develops both NSF's overall highest priorities and priorities for the various component activities within FS&E; factors considered in developing priorities for component activities include: NSF's highest funding priorities (listed in the FY 2006 Budget Request -- especially strengthening the core and addressing major national challenges identified by the Administration); needs and opportunities identified by Committees of Visitors and the Advisory Committees; new frontiers and topics of major impact that are identified by the scientific community, e.g., through workshops; and important emerging areas for which we receive numbers of highly ranked proposals. Senior management integrates that information, prioritizes budget requests within and among programs, and determines funding levels for FS&E component activities, which are reviewed by the National Science Board. FS&E relies on the merit review process to prioritize proposals within each component activity for funding decisions; final funding decisions also include consideration of NSF's core strategies and maintenance of a diverse portfolio.

Evidence: Evidence demonstrating a prioritization process to guide budget requests and funding decisions may be found in NSF Strategic Plan FY 2003 - FY 2008 (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201); Congressional Budget Justification (www.nsf.gov/about/budget/fy2006/toc.htm; Overview, p. 1); COV reports (www.nsf.gov/od/oia/activities/cov/); National Science Board Reports, minutes and agendas (www.nsf.gov/nsb/); Funding decisions: Grant Proposal Guide (www.nsf.gov/publications/pub_summ.jsp?ods_key=gpg).

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: FS&E regularly collects timely, high-quality performance data relating to key program goals through Committee of Visitors reports and other mechanisms, and uses this information to make corrective actions, adjust program priorities, make decisions on resource allocations and make other adjustments in management actions. GPRA and PART performance data are verified and validated by an independent, external consulting firm. Performance information is collected from NSF grant recipients via project reports as well as via site visits. The information is shared with and reviewed by key program partners within NSF as well as with programs at other agencies, when relevant and appropriate. In addition, COV reviews and recommendations are utilized to improve program performance. Process-related or quantitative goals such as dwell time are monitored via the agency's Enterprise Information System (EIS).

Evidence: Data and information are included in COV reports and NSF responses (www.nsf.gov/od/oia/activities/cov/); AC meeting minutes, and the annual AC/GPA report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); Awardee project reports; Enterprise Information System (EIS) - GPRA module; and NSF GPRA and PART Performance Measurement Validation and Verification: Report on FY 2004 Results Site visit reports (internal documents).

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: NSF awardees and contractors must meet reporting and financial record keeping requirements, and are held accountable for cost, schedule and performance results; NSF managers monitor results and take corrective action when necessary. To receive further support (subsequent awards), all applicants are required to include in their new proposals a report on the results of previous NSF support. Such past performance is then considered in the merit review process. The efforts of NSF staff are reviewed by their supervisors and by Committees of Visitors. Individual staff performance plans are directly linked to NSF's strategic goals. Subgrantees are similarly held accountable by NSF by grantees and contractors.

Evidence: Evidence demonstrating that federal managers and program partners are accountable for cost, schedule and performance results may be found in COV reports (www.nsf.gov/od/oia/activities/cov/); Awardee project reports; NSF Grant General Conditions (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf98gc1a); Federal Cash Transaction Reports;and annual performance evaluations of NSF staff/program officers.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: NSF routinely obligates its funds in a timely manner, and they are monitored to assure they are spent for the intended purposes; funds appropriated for FS&E are mostly two-year funds, but NSF obligates over 99% of this support within the first year it is appropriated. Provisions for automatic awarding of continuing grants are made at the beginning of each fiscal year; each funding increment is subject to the approval of an annual progress report by the program officer. NSF also has pre- and post-award internal controls to reduce the risk of improper payments. Beginning in FY 2004 NSF has incorporated erroneous payments testing of awardees into its on-site monitoring program. This will provide NSF with more information about the usage of NSF funding by awardees.

Evidence: Data and information demonstrating that funds are obligated in a timely manner and spent for the intended purpose are included in the PriceWaterhouse Coopers NSF FY 2001 Risk Assessment for Erroneous Payments; data on NSF Carryover, presented in the NSF Budget Request to Congress (www.nsf.gov/about/budget/); Risk Assessment and Award Monitoring Guide; clean opinion on financial statements for past 7 years; and Federal Cash Transaction Reports.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: NSF is a leader in the vigorous and dynamic use of information technology to efficiently and effectively advance the agency mission through the use of IT improvements that permit more timely and efficient processing of proposals. In FY 2004, NSF processed nearly 44,000 electronic proposals. NSF has established a time to decision goal to ensure that Principal Investigators who apply for grants are given a decision on the funding in a reasonable amount of time. Monthly reports on progress are sent to managers and results are available to all staff through the agency's Enterprise Information System. Next generation e-capabilities (in the planning and implementation stage) will be influenced by eGov activities, and NSF will continue its leadership in the Government-wide Grants Management Line of Business Strategy. NSF continues to investigate ways of making systems even more efficient such as broadening the use of letters of intent and proactively coordinating program deadlines.

Evidence: Evidence of procedures to measure and achieve cost effectiveness may be found in theMeasures Tab. Data and information on time-to-decision are located on pages I-20 and II-89 in FY 2004 Performance and Accountability Report (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=par). Data and information on organizational excellence is located on page II-85 in FY 2004 Performance and Accountability Report (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=par). Other evidence may be found in COV reports (http://www.nsf.gov/od/oia/activities/cov/); the NSF FY 2003-2008 Strategic Plan (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201); NSF Grant Proposal Guide (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=gpg); and program solicitations (http://www.nsf.gov/funding/).

Does the program collaborate and coordinate effectively with related programs?

Explanation: NSF promotes partnerships, including collaboration with other agencies, industry, national laboratories, and other countries for programs of mutually related interest. NSF routinely partners with others in the support of research of mutual interest. For example, NSF leads interagency working groups for the National Nanotechnology Initiative (NNI) and Networking and Information Technology Research and Development (NITRD) efforts and collaborates with other Federal agencies in activities such as those related to climate change science programs. NSF regularly shares information with other agencies and participates in coordination activities through OSTP and National Science and Technology Council [e.g., Multi-Agency Tissue Engineering Science (MATES)]. Policy guidance provided by the National Science Board incorporates perspectives from related programs and investments. Split funding across organizational lines is common, and standard operating procedures are in place.

Evidence: Data and information demonstrating that the program collaborates effectively with related programs are included in management plans, internal administrative manuals, and in program solicitations (http://www.nsf.gov/funding/). Information on USGCRP is available at http://www.usgcrp.gov. Information on the NNI is available at http://www.nano.gov/. Examples of interagency joint grant announcements include: Joint DMS/BIO/NIGMS Initiative to Support Research in the Area of Mathematical Biology, NSF 04-572 (http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04572); and Maize Genome Sequencing Project: An NSF/DOE/USDA Joint Program, NSF 04-614 (http://www.nsf.gov/pubs/2004/nsf04614/nsf04614.htm). Information on NITRD is available at http://www.nitrd.gov; and information on NITRD interagency collaboration and coordination is documented in the "Supplement to the President's Budget for FY2006" available at http://www.nco.gov/pubs/2006supplement/, which provides overviews of dozens of collaborative activities of NSF with DoD, DoE, and other NITRD agencies. Additional collaborative efforts are documented in the annual "Blue Book," the High End Computing Revitalization Task Force (HECRTF) report, and Interagency Coordination Report all available at http://www.nco.gov/pubs/.

Does the program use strong financial management practices?

Explanation: The strong financial management practices of FS&E are consistent with practices that led to NSF being the first federal agency to receive a "green light" for financial management on the PMA scorecard. NSF continues to maintain a green rating. NSF has received a clean opinion on its financial audits for the last 7 years. The NSF is committed to providing quality financial management to all its stakeholders. It honors that commitment by preparing annual financial statements in conformity with generally accepted accounting principles in the U.S. and then subjecting the statement to independent audits. As a federal agency, NSF prepares the following annual financial statements: Balance Sheet, Statement of Net Cost, Statement of Changes in Net Position, Statement of Budgetary Resources, and Statement of Financing. Supplementary statements are also prepared including Budgetary Resources by Major Accounts, Intragovernmental Balances, Deferred Maintenance, and Stewardship Investments.

Evidence: Data and information demonstrating strong financial practices are included in the Executive Branch Management Scorecard (www.whitehouse.gov/results/agenda/scorecard.html); in the results of NSF financial audits; and in performance and management assessments (www.whitehouse.gov/omb/).

Has the program taken meaningful steps to address its management deficiencies?

Explanation: NSF receives and acts on advice on how to improve program management from external experts including Committees of Visitors (COVs), which regularly provide feedback on programmatic and management-related concerns. The COVs provide external program assessments that are used in program management. COVs conduct detailed reviews of the materials associated with individual proposal actions and have traditionally assessed the integrity and efficiency of the processes for proposal review. Each COV addresses management issues through a series of questions in Section A5 of the COV template. NSF staff, in turn, respond to any management deficiencies identified through an agency response to the COV report and outline the steps the agency will take to address any issue. For example, the Chemistry COV observed that "diversity is still problematic for chemistry as for many of the sciences." As a result, the CHE Division has been using a multi-faceted strategy to broaden participation. At the undergraduate level the Division has encouraged its community to engage first- and second-year college students from the full spectrum of postsecondary institutions in cutting-edge research. A Joint Subcommittee comprising members of the MPS Advisory Committee and the Advisory Committee of the Education and Human Resources Directorate endorsed the idea of engaging students in MPS research areas within the first two years of college and of reaching out to two-year institutions as an important element of such an effort. The Undergraduate Research Center (URC) program's first awards were made in FY2004, comprising both planning and full grants through a partnership with the Office of Multidisciplinary Research (OMR) and the EHR directorate. The NSF response is updated annually. In addition, the NSF response, as well as the initial COV report, is reviewed by directorate Advisory Committees. The Foundation also conducts an annual review to assess administrative and financial systems and procedures to ensure that effective management controls are in place, and that any deficiencies are identified and addressed.

Evidence: All COV reports and NSF responses may be found at www.nsf.gov/od/oia/activities/cov/covs.jsp. Subchapter 300 of the Proposal and Award Manual provides additional information regarding the Committees of Visitors.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Grants are awarded based on NSF's competitive, merit review process that includes external peer evaluation using two standard NSB-approved criteria. A qualified expert serving as an NSF program officer carefully reviews all proposals. Additionally, each proposal is usually reviewed by 3-10 persons outside NSF who are professionals in the particular field represented by the proposal. All such activities are reviewed by NSF's Committees of Visitors. Competitive merit review, with peer evaluation, is NSF's accepted method for informing its proposal decision process. The NSB-approved criteria address the "Intellectual Merit" and the "Broader Impacts" of the proposed effort. Some solicitations contain additional criteria that address specific programmatic objectives. NOTE: The weight of this question has been increased to 20% to reflect the relative importance of merit review in assuring the relevance, quality, and performance of NSF's investments.

Evidence: Evidence demonstrating that grants are awarded through a clear competitive process are included in the NSF FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); NSB Policy on Recompetition (www.nsf.gov/nsb/documents/1997/nsb97224/nsb97224.txt); Report to the NSB on the NSF Merit Review Process - FY 2004 (www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf); NSF Merit Review Criteria (www.nsf.gov/pubs/1999/nsf99172/nsf99172.htm); and COV Reports and NSF responses (www.nsf.gov/od/oia/activities/cov/).

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: FS&E uses a multifaceted array of oversight practices including merit review, appropriate grant mechanisms, site visits, project reports, an IT-enabled grants management system, and a risk-based monitoring program that provide sufficient knowledge of grantee activities. Review and Award: The NSF merit review process provides a high degree of assurance that awardees are technically qualified and have resources available to successfully undertake the work they have proposed. Proposals include results from prior NSF support and reviewers are asked to comment on the quality of the prior work in their evaluations. This process helps assure the quality of future grantee activities based upon past performance. At the award phase, the most appropriate funding instrument is determined, as are grant terms and conditions that ensure appropriate reporting on progress and expenditure of funds. Grant Oversight: NSF's award oversight is tailored for each award, and could consist of any combination of the following: regular reports from grantees on progress, desk-reviews, site visits, meetings with project staff, and interim reviews by special panels. Continuing support (i.e., continuing grant increments) is based upon required template-based annual progress reports submitted by grantees that are subject to review and approval by NSF Program Officers before additional funds are released. Award Monitoring and Business Assistance Program: Annually, awardees undergo a risk based monitoring process, the Award Monitoring and Business Assistance Program (AMBAP). The AMBAP provides focused assurance that each awardee institution maintains sufficient knowledge of all NSF project activities. Through AMBAP, NSF has increased the number of staff devoted to post-award administration, including a dedicated outreach manager to interact and communicate with awardees. Internal Improvements: NSF has implemented and is continuously improving an IT-enabled grants management system to support all awards and ensure effective post-award management. NSF uses technology and innovative practices, such as teleconferencing, videoconferencing, and reverse site visits to enhance performance oversight. Further, NSF systems track fund expenditures quarterly at the level of individual grants. This information is available electronically to all NSF staff. Further, NSF is in the process of updating the Proposal and Award Manual and the Grant Policy Manual, and developing system requirements for a new comprehensive project report tracking system. Federal Requirements: Finally, NSF adheres to the oversight standards used by all Federal agencies. Using these standards allows NSF to benefit from awardee compliance with all relevant OMB Circulars regarding annual audits, and compliance with other Federal regulations regarding the use of Federal funds. The Single Audit Act and cognizant audit agencies mandate significant oversight functions for grant and contract recipients. This law and the concomitant oversight audit and review activities provide baseline oversight procedures that govern all grant recipients. As a system, NSF's oversight mechanisms provide sufficient knowledge of grantee activities to monitor and understand how funds are utilized by grantees. However, NSF's capacity to provide oversight is dependent on available resources. While NSF oversight remains sufficient, current resources limit NSF's ability to perform the level of oversight deemed desirable.

Evidence: Data and information demonstrating sufficient oversight practices are included in COV reports (www.nsf.gov/od/oia/activities/cov/); awardee project reports; Report to the NSB on the NSF Merit Review Process - FY 2004 (www.nsf.gov/nsb/documents/2005/MRreport_2004.pdf); Risk Assessment and Award Monitoring Guide; clean audit opinions; President's Management Agenda (PMA) Scorecard for Financial Management (www.whitehouse.gov/results/agenda/scorecard.html); Site visit reports; Trip reports from attendance at professional meetings; and Workshops and grantee meetings as presented in the annual Performance and Accountability report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par).

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NSF collects significant results from NSF-supported research and makes them available on the web in terms that the public can understand through the Discoveries area of the NSF web site, through press releases, through the Performance and Accountability Report, an annual brochure on Performance Highlights, and the Report of the Advisory Committee for GPRA Performance Assessment. Grantees provide project reports to NSF, which are examined, and approved/disapproved by the program directors. NSF Grant General Conditions require that results of NSF-supported research be published in open literature such as peer-reviewed journals. Members of the general public have access to data on the numbers of proposals and numbers of awards as well as, for each award, the name of the principal investigator, the awardee institutions, amount of the award, and an abstract of the project. NSF proactively seeks out noteworthy discoveries and distributes these in general press releases.

Evidence: Evidence demonstrating performance data is collected from grantees includes NSF Discoveries web site (www.nsf.gov/discoveries/); news releases (www.nsf.gov/news/news_list.cfm?nt=2); FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); AC/GPA annual reports (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); FY 2004 Performance Reports and Highlights (www.nsf.gov/about/performance/reports.jsp); FY 2006 Budget Request (www.nsf.gov/about/budget/); NSF Grant General Conditions (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf98gc1a); Highlights of annual meetings/grantees meetings; workshops; and awards database (www.nsf.gov/awardsearch/).

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation:  

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Fundamental Science and Engineering (FS&E) program has demonstrated adequate progress in achieving its long-term performance goals as determined by several external expert panels. NSF relies on external evaluation to determine whether it is achieving its long-term objectives. Since FY 2002, the NSF Advisory Committee for GPRA Performance Assessment (AC/GPA) serves as the focal point for these activities. Input is derived from numerous sources including Committees of Visitors (COVs), grantee progress reports, and summaries of substantial outcomes ("nuggets") from funded research. The AC/GPA has determined that the accomplishments under the Ideas goal have "demonstrated significant achievement" toward annual and long-term performance goals.

Evidence: Evidence that NSF has demonstrated adequate progress in achieving its long-term performance goal is indicated in the FY 2004 AC/GPA report, pages 27-37 at www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216; and in the FY 2004 Performance; and Accountability Report, pages II-26 to II-31 (executive summary) and II-59 to II-73 at (www.nsf.gov/publications/pub_summ.jsp?ods_key=par).

Does the program (including program partners) achieve its annual performance goals?

Explanation: NSF has established a time to decision goal to ensure that Principal Investigators (PI) who apply for grants are given a decision on the funding in a reasonable amount of time. Since 2002, NSF has more than achieved its goal of informing PIs of funding decisions for at least 70% of proposals "within six months of deadline or target date, or receipt date, whichever is later."

Evidence: Evidence showing that the FS&E program meets its annual performance goals can be found in the Measures Tab; COV/AC reports (www.nsf.gov/od/oia/activities/cov/); NSF FY 2003-2008 Strategic Plan (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04201); AC/GPA Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); FY 2004 Performance and Accountability Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); and time to decision data from the NSF EIS System.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: NSF continues to improve efficiency and effectiveness of agency operations through increasing use of electronic business systems and operations. NSF builds on the past successes of FastLane, which accepts proposals from the university community electronically and supports university/NSF business processes. NSF continues improvement notably with the Electronic-Jacket (E-J) that supports merit review, real-time access to proposal and award information, and provides for "shared-work" processes for interdisciplinary proposals that involve more than one organization. E-J has been an essential enabler for NSF to maintain dwell time results in the face of increased numbers of proposals. NSF's Business and Operations Advisory Committee has rated NSF as successful in developing and using "new and emerging technologies fro business application." NSF received green ratings in 2003 and 2004 in e-Government, indicating current success and positioning for future successes in government wide electronic business methods.

Evidence: Evidence demonstrating improved efficiencies in achieving program goals can be found in FY 2004 Performance and Accountability Report, pages I-13, I-19; (www.nsf.gov/publications/pub_summ.jsp?ods_key=par); and in program solicitations (www.nsf.gov/funding/). Efficiencies in proposal processing are demonstrated throughout the agency by the increasing volume of competitive proposal actions over the last five years (21,445 actions in FY 2000; 31,552 actions in FY 2004; an increase of 47 percent) while increasing staffing levels only slightly (6.5 percent).

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: Because of the quality, relevance and performance of NSF programs, aspects of NSF investments and processes are often emulated by programs in other Federal agencies and international governments. NSF uses external expert review, through Committees of Visitors, to ensure the continued high quality of the distinct disciplinary components that make up Fundamental Science and Engineering. In addition, NSF is the only Federal agency charged with promoting the progress of science and engineering research and education across all fields and disciplines. As such, NSF's activities through its investments in Fundamental Science and Engineering provide the principal source of federal support for basic research at colleges and universities in many disciplines. No other entity, government or private, addresses this far-reaching purpose.

Evidence: Evidence of FS&E comparing favorably to other similar programs can be found in the article on the International emulation of NSF. "A Framework for Change?" Science (04/15/05) Vol. 308, No. 5720, P. 342; FY 2004 Performance and Accountability Report www.nsf.gov/publications/pub_summ.jsp?ods_key=par); COV reports (www.nsf.gov/od/oia/activities/cov/); Advisory Committee reports; and the Enterprise Information System (EIS) data (internal).

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Independent reviews by Committees of Visitors (COVs) and other external groups (e.g., Advisory Committees, National Science Board, National Academies (NAS/NAE/NRC), PCAST) find that programs are effective and achieve results. In particular, the most recent evaluation that included the entire Ideas goal, incorporating the Fundamental Science and Engineering portfolio, was the 2004 meeting of the AC/GPA. The AC/GPA wrote: "... demonstrated significant achievement for all indicators ..." for the Ideas goal. The AC/GPA added, "NSF accomplishments in the IDEAS outcome goal have advanced the frontiers of discovery and hold considerable promise for addressing important societal concerns." In reaching this determination, the committee specifically considered indicators that matched the objectives used here for Fundamental Science and Engineering. NOTE: The weight of this question has been increased to reflect the importance of independent evaluation in verifying relevance, quality and performance of NSF's investment in Fundamental Science and Engineering.

Evidence: Evidence demonstrating that independent evaluations are of sufficient scope and quality can be found in the AC/GPA Report (www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf04216); FY 2004 Performance and Accountability Report www.nsf.gov/publications/pub_summ.jsp?ods_key=par); Awardee project reports; and COV reports (www.nsf.gov/od/oia/activities/cov/) and NSF responses.