Detailed Information on the
Defense Basic Research Assessment

Work with the research community and Congress to explain the need to limit claims on research grant funds to proposals that independently can meet the standards of a strict merit-review process.

Continue to emphasize the use of independent review panels in assessing the performance of the program.

Measure: Certification in biennial reviews by technically competent independent reviewers that the supported work, as a portfolio, is of high quality, serves to advance the national security and is efficiently managed and carried out.

Explanation:Review team will be independent, comprised of technically competent reviewers.

Measure: Long-term Measure: Portion of funded research that is chosen on the basis of merit review. Reduce non-merit-reviewed and -determined projects by one half in two years (from 6.0% to 3.0%)

Explanation:The goal is to have research awarded through a merit-review process.

Is the program purpose clear?

Explanation: "The mission of the Defense Science and Technology (S&T) Program is to ensure that the warfighters of today and tomorrow have superior and affordable technology to support their missions and provide revolutionary war-winning capabilities."

Evidence: DoD publishes a Basic Research Plan, which lays out, for the general public and the scientific community, information needed to understand the program and, for researchers, general information that can provide a basis to know if they might be able to contribute to the program. (The purpose of the program can be found on page I-1, section C.)

Does the program address a specific interest, problem or need?

Explanation: Interests/needs are: (1) provide options for new weapon systems; (2) help prevent technological surprise by adversaries; and (3) develop new scientists who will contribute to the DoD mission in the future.

Evidence: The interests are stated in the Basic Research Plan (BRP), page I-1, section C.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: The Basic Research program is designed to ensure that DoD maintains its competitive edge over potential adversaries in the future. DoD names Strategic Research Areas in the Basic Research Plan, which identify, for the research community, areas of interest for which funding is available and which point to general capabilities that will help maintain that compettive advantage. Broad Agency Announcements (BAAs) detail for researchers particular areas of interest by funding Service or Agency, but also allows researchers to propose specific research topics and approaches.

Evidence: Can be found in BRP, Chap VI, "Strategic Research Areas" and Broad Agency Announcements (BAAs), published by the individual Military Services and Agencies for response by research organizations interested in working as grantees.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: DoD tries to maximize its research investments by monitoring technology or products from the private sector or work supported by other organizations (the National Science Foundation, for example), and will use that work unless progress in those areas is inadequate for DoD needs. A DoD process, called the Reliance Process, seeks to eliminate most of the duplication of research responsibilities within the Department. DoD's research organizations meet and compare capabilities and one organization often is chosen to lead research in a given area.

Evidence: Example: The Office of Naval Research's Directive Ser 01/8225. "DoN Science and Technology National Naval Program Guidance" lays out Navy research needs.

Is the program optimally designed to address the interest, problem or need?

Explanation: The DoD procedure is built around announcements of research areas (BAAs, above), and competition (submission of proposals by researchers). Merit review of proposals follows identification of problems and specific research opportunities by proposers, including in-house researchers.

Evidence: Service BAAs. See, for example, www.onr.navy.mil/onr/contracts.htm, click on Contracts and Grants, then click on Solicitations and Business Opportunities, then ONR or NRL.

Does the program effectively articulate potential public benefits?

Explanation: In addition to publications and testimony laying out the benefits for national security (and to the Nation more generally), DoD organizations hold many conferences at which they make known the Department's sponsorship of research in certain general areas of investigation, along with the potential public benefits.

Evidence: See BRP 1-1 C. Also the BAAs themselves communicate with the relevant scientific public about potential benefits and needs.

If an industry-related problem, can the program explain how the market fails to motivate private investment?

Explanation: Not industry-related. DoD's programs are designed to benefit a National need that is not already addressed by the private sector.

Evidence:

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: DoD's Defense Technology Area Plan, which addresses applied research in a manner similar to the way that the BRP addresses basic research, has goals for the various technologies supported by DoD. Basic research feeds into those technologies and hence contributes to the meeting of the goals. Because basic research by its very nature is a long-term and speculative activity, the long term goals cannot be highly specific without risk of making the program too conservative, thereby limiting its potential payoff.

Evidence: Defense Technology Area Plan lays out goals by area. Also, Service or Agency BAAs articulate thrusts.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: DoD uses semi-annual reviews by outside review panels to assess the health of the Basic Research program. They assess program content, management abilities, program results.

Evidence: Instructions given to the independent reviewers prior to the start of their reviews. (Note that the instructions addressing R&D PARTs require a unique process-focused interpretation of annual goals as applied to basic research programs.)

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: Increasing numbers of recipients of Basic Research funds have sought Congressional earmarking assistance, thus circumventing in whole or in part, the merit-based award process and DoD has not been notably successful in stopping them. Such circumvention of the merit-based process has been opposed by professional societies, as they recognize that in the long-term, the health of the research enterprise is greatly diminished by awards based on geography or influence. This is in contrast to the situation at NIH, where past leadership and the affiliated research community has been highly successful in nearly eliminating earmarks.

Evidence: Increasing earmarks/directed adds solicited by universities and other research organizations and supported by Congress without independent technical review and merit-based awards.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: DoD works with many other agencies as well as industry in coordinating and executing their program. DoD has entered into agreements with other Federal agencies to fund and manage selected initiatives and the Department also works through such mechanisms as the National Science and Technology Council (NSTC) to coordinate various R&D efforts. The record isn't perfect, however, with some degree of parochialism in research agendas on the part of the support organizations.

Evidence: Various Memoranda of Understanding or Memoranda of Agreement with other related Federal agencies document cooperation with other entities. Also, NSTC membership includes DoD as one of the many departments that coordinate in the planning and execution of research and development projects.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: Reviews of the Basic Research program are conducted at least once every three years by outside review panels made up of academics, industrial researchers and researchers from other agencies.

Evidence: DoD provides instruction pamphlets and evaluation forms to its independent reviewers.

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: The basic research budget programs are divided into a few distinct programs or collection of related projects which are traceable in R&D tables, and show up as line items in the President's Budget.

Evidence: DoD budget documents, such as the Research, Development, Test and Evaluation Program listings (R-1s) are used to track funding allocations through budget preparation, Congressional approval and budget execution processes.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: Recommendations from the independent review groups are reviewed and often implemented.

Evidence: In 2000/2001, the independent panel reviewing the DoD Basic Research program recommended that the DoD policy on Intellectual Property Rights (IPR), as it pertains to results from industry-university collaborations, be clarified and be made uniform for DoD-sponsored research. The recommendation resulted in new IPR guidelines first transmitted in Jan., 2002.

Is evaluation of the program's continuing relevance to mission, fields of science, and other "customer" needs conducted on a regular basis?

Explanation: The review of specific research areas is part of preparation for program announcements as well as review panel findings and the Technology Area Reviews and Assessments (TARAs) findings reflected in recommendations under II-7.

Evidence: TARA recommendations. IPR memo Jan 31, 2002 and New IPR Guidelines.

Has the program identified clear priorities?

Explanation: The Basic Research Plan lists a number of Strategic Research Areas, including biomimetics, nano-science, smart materials and structures, information technology, human centered systems and compact power. These strategic research areas are reflected in the BAAs of the Military Services and Agencies.

Evidence: BRP, Chap VI , "Strategic Research Areas", and Chapter IV, "Basic Research Areas", and the BAAs that identify these areas to the research community.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Researchers are required to document their results in technical publications or summaries. Independent review panels for each of the Services, made up of academic researchers, industrial researchers and scientists from other Federal agencies, review these materials, and materials and managers of the research program, by technical area and assess the quality of the research programs and make recommendations to improve the program.

Evidence: Information from the Air Force Scientific Advisory Borad of the Air Force Office of Scientific Research (AFOSR) independent review.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: The research offices within each Service are expected by the parent organizations to stick to budgets and schedules, as funds are needed for other operational programs. Non-performing programs are likely to have funds taken away to meet more immediate needs.

Evidence: Many programs have been restructured or funding reduced when financial reports indicate a pattern of slow performance or if the parent organization has reason to believe that the program is inadequately focused on the needs of the warfighter.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: DoD carefully monitors expenditures by each military Service. Each services spends funds for basic research in a timely way.

Evidence: Obligation reports prepared by the financial processing centers of the Department show obligation rates in excess of 90% in the first year of availability.

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: There are no formal procedures in place that provide incentives to managers. The most significant incentives are informal. If the research program doesn't perform, program managers lose money to nearer-term demands. In addition, managers must use competitive award processes mandated by regulation and statute, which contibute to program efficiencies.

Evidence: This response, which refers to formal incentives and procedures, was de-weighted to reflect the significant informal procedures, noted to the left, that contribute to efficiency and cost effectiveness.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: Full annual cost budgeting is required of all basic research. The number of programs that DoD funds is a function of the money available.

Evidence: DoD budget documents provided to the Congress each year provide visibility into the Basic Research program.

Does the program use strong financial management practices?

Explanation: DoD's financial management system is being improved, but slowly. No audit reports showing that the Basic Research program is free of internal control weaknesses have been provided.

Evidence:

Has the program taken meaningful steps to address its management deficiencies?

Explanation: When deficiencies have been identified by the DoD Inspector General or independent review panels, DoD has sought to correct the deficiencies.

Evidence: A DoD IG report (98-198) on the University Research Initiative program identified weaknesses in the Army's and Air Force's management control programs. Subsequent actions by the Army and the Air Force have addressed these deficiencies.

Does the program allocate funds through a competitive, merit-based process, or, if not, does it justify funding methods and document how quality is maintained?

Explanation: DoD awards most of its Basic Research funding through grants, which regulations specify should be competitive.

Evidence: DoD's Grant Regulations (DoD 3210.6-R). require the use of competitive procedures, to the maximum extent practicable, in awarding research grants.

Does competition encourage the participation of new/first-time performers through a fair and open application process?

Explanation: There is a significant turnover each year in universities receiving grants from DoD. The big universities (with large numbers of projects supported and receiving significant portions of the overall funding) always receive some of the funding, but there is a not insignificant turn-over on the margins. In addition, Principal Investigators (those charged with heading research projects within the recipient universities or non-profits) and their co-researchers change with time, providing further turnover of research personnel and ideas.

Evidence: Fourteen percent of schools receiving basic research funding in 2001 had not received funding in 1999 (41 of 302). The DoD Grants Regulations (DoD 3210.6-R) require use of merit-based, competitive preocedures to the maximum extent practicable in the award of grants. In addition, DoD maintains a program that specifically targets new or under-represented entrants to encourage increased participation in its basic research grants program.

Does the program adequately define appropriate termination points and other decision points?

Explanation: Generally, 1, 2 or 3 yr. periods for individual grants, as specified in BAAs at program announcement. However, there is no formal definition of termination points with regard to Strategic Research Areas, which one would expect to show a great deal of continuity over time.

Evidence: The BAAs identify the period of time for which grants can run.

If the program includes technology development or construction or operation of a facility, does the program clearly define deliverables and required capability/performance characteristics and appropriate, credible cost and schedule goals?

Explanation: The basic research program does not fund facilities.

Evidence:

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: The Basic Research program contributes significantly to the achievement of the DoD mission. Both external and internal reviews indicate that reviewers believe that the program contributes materially to the Department's capabilities.

Evidence: Internal and external review group reports. For example, the back-briefing to the AFOSR, cited above.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Annual performance goals for the research results of the Basic Research program are not directly measurable. Instead, determination of the value of the program must be based largely on assessment of the soundness of the awards and management processes. For this assessment, we largely rely on in-depth reviews of the programs by independent review panels. Such reviews (usually biennial) generally indicate well executed programs, but not all DoD Services and Agencies measure results equally well. The Army, through the Army Research Laboratory's assessment process, does particularly well. Others lag.

Evidence: External review reports.

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: DoD works very hard to achieve improved efficiencies in its enacted program, and generally does well in executing the approved program. However, it has done less well in convincing Congress and the academic community of the need to support its recommended program than does, for example, NIH, which has an unusually low number of earmarks in its program compared to DoD. This results in a decreasing portion of the research total that is productive to the Department's goals.

Evidence: Increases in directed research of questionable value to the nation's security, as displayed in Congressional conference reports.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: DoD basic research sponsorship is generally considered on par with the best of other Federally-sponsored research and other world-class research. Earmarks, not being chosen on basis of merit to the national security, generally contribute less than the typical research project to fulfilling the Department's mission.

Evidence: Independent review panels, number of Nobel Prize winners supported prior to receipt of their Prizes.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: Independent review panels, associated with the research programs of each of the military Services and agencies, do indicate that the programs are effective and are producing valuable results.

Evidence: Reports from the independent panels.

If the program includes construction of a facility, were program goals achieved within budgeted costs and established schedules?

Explanation: Does not include construction of facilities.

Evidence: