Detailed Information on the
Advanced Technology Program Assessment

Working with the Administration and Congress to terminate this program.

Measure: Cumulative number of publications generated by ATP funded research

Explanation:

Measure: Cumulative number of patents generated by ATP-funded research

Explanation:

Measure: Cumulative number of technologies under commercialization

Explanation:

Is the program purpose clear?

Explanation: ATP is designed for and focuses on promoting private investment in R&D for high-risk, broad impact technology development.

Evidence: The mission and purpose of the ATP is stated in DOC Annual Performance Plans, budget justifications, and associated documents. ATP's purpose derives from its statutory authority: see 15 USC 278n.

Does the program address a specific interest, problem or need?

Explanation: ATP was initially established to address concerns about U.S. competitiveness in the late 1980s and early 1990s. However, one could argue that this concern has lessened in recent years. Studies show that there are many non-governmental entities investing in early-stage technology development, such as corporate research labs, venture capital firms, angel investors, and universities. Given the amounts available from other sources, it is not evident that there is a clear need for federal subsidies for private technology development.

Evidence: Recent work by Lewis Branscomb et al. at Harvard estimates that between $5.4B (conservative estimate) to$35.5B (inclusive estimate) was invested in early stage technology development (Branscomb and Auerswald, Between Invention and Innovation, pre-publication copy available upon request). These estimates include $1.4B to $7.3B in investments from the federal government.

Is the program designed to have a significant impact in addressing the interest, problem or need?

Explanation: ATP is designed to have a targeted impact in an area dominated by private funding. Substantial ATP investments have been made in areas where significant external funding is available, such as biotechnology and information technology. Relative to the other funding sources available for these areas, ATP is only a modest contributor.

Evidence: ATP has met its annual performance goals for new patents filed and new technologies under commercialization (see "program results" below, as well as DOC's budget justification and annual DOC performance plans and reports), but it is not evident that ATP funding was actually needed for individual projects to achieve these results.

Is the program designed to make a unique contribution in addressing the interest, problem or need (i.e., not needlessly redundant of any other Federal, state, local or private efforts)?

Explanation: ATP is tiny fraction of the total amount invested in early-stage technology development. There is overlap with private venture capital and angel investors, as well as with other federal programs, such as the Defense Advanced Research Projects Agency (DARPA), Small Business Innovation Research (SBIR), and Small Business Technology Transfer (STTR).

Evidence: Past GAO studies have concluded that many ATP projects would have been funded with or without ATP participation. Branscomb's study indicates significant investment by other entities, both private and federal, relative to ATP. Of ATP clients surveyed, 75% indicated that the project would have continued in some form without ATP funding.

Is the program optimally designed to address the interest, problem or need?

Explanation: ATP is intended to stimulate highly focused R&D efforts that are identified and led by the private sector but which would not take place without Federal resources. ATP selection criteria currently do not adequately ensure that projects are not duplicative of research occurring in the private sector, nor do they ensure that projects address particular public needs. Commerce continues to work with the Administration and with Congress to reform the program.

Evidence: Past GAO studies have identified ATP projects that are duplicative of research efforts in the private sector. The Administration has submitted reforms, but they are unlikely to be enacted in a form that will transform the program.

Does the program have a limited number of specific, ambitious long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: ATP has one overarching strategic goal that directly reflects the purpose of the program.

Evidence: ATP's overarching goal is "to accelerate private investment in and development of high-risk, broad-impact technologies". See DOC's budget justification and annual DOC performance plans and reports.

Does the program have a limited number of annual performance goals that demonstrate progress toward achieving the long-term goals?

Explanation: ATP's annual performance goals--technical publications, patents, and technologies commercialized--suggest at least some progress toward the program's long-term overarching goal and purpose. However, as with most publicly funded R&D programs, it is difficult to determine whether progress would have occurred without ATP funding. Potential for cost recoupment is another indicator that could be used in assessing long-term progress.

Evidence: See "Program Results" section below; see also DOC's budget justification and annual DOC performance plans and reports.

Do all partners (grantees, sub-grantees, contractors, etc.) support program planning efforts by committing to the annual and/or long-term goals of the program?

Explanation: At the time of the initial award, grantees begin contributing directly to the program's long-term goal and purpose. Grantee technical progress and outputs (and hence contribution to the program goal) are monitored throughout the grant period and for up to six years after ATP funding ends. Under the Terms and Conditions of ATP awards, ATP uses its Business Reporting System to systematically collect data from awardees during and after project completion; these data allow ATP to track and report on output and intermediate outcome performance. These data are supplemented by case studies and special-purpose surveys.

Evidence: Data from ATP's Business Reporting System are used for reporting results on key ATP performance measures; measures and results are presented in annual DOC Performance Plans and Performance Reports. Data also are analyzed and presented in special ATP progress reports (for example, see Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies, US DOC/TA/NIST, report #6491, April 2000). Data on technical performance and technology diffusion are collected systematically in Technical Quarterly and Final Reports.

Does the program collaborate and coordinate effectively with related programs that share similar goals and objectives?

Explanation: While ATP is structurally different than other Federal technology programs such as the Small Business Innovation research program (SBIR) and the Defense Advanced Research Projects Agency (DARPA), there is opportunity for collaboration with these programs. OMB has asked Commerce to evaluate options for using ATP as a competitive source for other agencies as an alternative to SBIR.

Evidence: There is no evidence of a strong track record of collaboration with these other programs.

Are independent and quality evaluations of sufficient scope conducted on a regular basis or as needed to fill gaps in performance information to support program improvements and evaluate effectiveness?

Explanation: Regular external review and oversight are provided by the ATP Advisory Committee (which meets 2 to 3 times per year) as well as by the NIST Visiting Committee on Advanced Technology (which meets quarterly). ATP also has been the subject of many external program evaluations and reviews, including 29 GAO and OIG audits from 1993 to the present that have evaluated virtually all aspects of the program.

Evidence: See annual reports of the ATP Advisory Committee and the NIST VCAT. ATP Advisory Committee reports and minutes are available at www.atp.nist.gov/atp/adv_com/reports.htm. See also NRC, The Advanced Technology Program: Challenges and Opportunities, 1999 (available at books.nap.edu/catalog/9699.html); NRC, The Advanced Technology Program: Assessing Outcomes 2001, available at www.nap.edu/catalog/10145.html; and numerous GAO and OIG audits (references available upon request).

Is the program budget aligned with the program goals in such a way that the impact of funding, policy, and legislative changes on performance is readily known?

Explanation: ATP's performance measures show the impact of changes in funding levels.

Evidence: DOC budget justifications and annual performance plans and reports show the relationship between funding levels and performance measures over time.

Has the program taken meaningful steps to address its strategic planning deficiencies?

Explanation: NIST as a whole has developed a new Institute-wide long-term strategic planning process; the process includes new mechanisms for aligning Operating Unit plans with the NIST-wide plan. The Administration has also proposed ATP reforms.

Evidence: NIST's external advisory bodies routinely observe and comment on any deficiencies associated with NIST's strategic planning processes, and NIST responds to these observations. For example, the Visiting Committee on Advanced Technology has reviewed and commented favorably on NIST-wide strategic planning efforts in recent meetings (held quarterly).

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: ATP's Business Reporting System and Technical Quarterly Reports collect data systematically; data are reported regularly in annual performance plans and reports; current projects are evaluated regularly and performance factors are used to make continuation and termination decisions and to review program design and project management processes.

Evidence: Program performance data collected through the Business Reporting System are presented in DOC budget justifications and annual performance plans and reports. Data collected through Technical Quarterly Reports includes proprietary data and are not publicly available.

Are Federal managers and program partners (grantees, subgrantees, contractors, etc.) held accountable for cost, schedule and performance results?

Explanation: ATP program management is accountable for programmatic and administrative performance. Grantees are held accountable for--and continuation / termination decisions are made on the basis of--cost, schedule, and performance results.

Evidence: See annual reports of the ATP Advisory Committee and the NIST VCAT; internal program reviews also focus on accountability for programmatic and administrative performance.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: NIST as a whole manages its resources carefully and the ATP program typically has a limited amount of unobligated funds at year end, excluding adjustments for changes in the status of grants made in prior years (which may result from program management practices). NIST's strong budget and accounting systems include rigorous internal reviews and external audits to ensure that funds are expended as intended. In addition, ATP grantees are audited on a regular basis to ensure funds are spent appropriately.

Evidence: SF-132 (apportionment schedule) and SF-133 (report on budget execution). Internal processes include rigorous quarterly financial reviews. See the NIST-audited Annual Financial Statements and numerous GAO and OIG reviews. Audits of ATP grantees are conducted by external, independent auditors following Government Auditing Standards (these audits contain proprietary data and are not publicly available).

Does the program have incentives and procedures (e.g., competitive sourcing/cost comparisons, IT improvements) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: ATP's administrative costs have been held at the lowest possible level over the course of the program, and comply with appropriation guidance. ATP program management continuously reviews administrative procedures to identify and implement measures that will improve program efficiency and effectiveness.

Evidence: Examples of recent efforts to improve administrative efficiency and effectiveness include: electronic submission of proposals; rolling submissions over the fiscal year; and a gated approach to proposal review. Administrative costs are tracked in the NIST accounting system; data can be provided, if needed, on administrative costs per FTE, per grant, etc. or a similar ratio.

Does the agency estimate and budget for the full annual costs of operating the program (including all administrative costs and allocated overhead) so that program performance changes are identified with changes in funding levels?

Explanation: NIST's budget request and prior year budget data reflect the full annual costs of operating ATP, including direct and indirect costs. Out-year targets for quantitative performance measures are based in part on resource inputs; variation in input levels directly affect estimated performance.

Evidence: Total program costs are presented in NIST's budget justification and annual financial statements. NIST's internal accounting system reports can provide costs by object class. Overhead is applied uniformly per full-cost accounting procedures that are specified in Chapter 8.07 of the NIST Administrative Manual. DOC annual performance plans show the impact of proposed funding levels on ATP's performance measures.

Does the program use strong financial management practices?

Explanation: NIST maintains financial management oversight. NIST has a long history of unqualified financial audits and provides accounting services for several other DOC bureaus.

Evidence: See NIST's audited Annual Financial Statements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Regular program oversight is obtained through several channels: the NIST Visiting Committee on Advanced Technology; ATP's external Advisory Committee; internal NIST program reviews. Many of the 29 GAO and OIG audits have focused on program management assessment.

Evidence: ATP has made numerous changes in program management in response to recommendations produced by these review mechanisms.

Are grant applications independently reviewed based on clear criteria (rather than earmarked) and are awards made based on results of the peer review process?

Explanation: All grantees are selected through ATP's rigorous Source Evaluation Board (SEB) analysis and review process, which combines appropriate technical and economic expertise for peer review of all proposals. Reviews are based on clear and transparent criteria and debriefings are made available to all proposers.

Evidence: The SEB-based selection process has been carefully designed based on extensive stakeholder input and numerous external reviews by GAO. The process also has been reviewed by the NRC; see NRC, The Advanced Technology Program: Challenges and Opportunities, 1999 (available at books.nap.edu/catalog/9699.html); and NRC, The Advanced Technology Program: Assessing Outcomes 2001 available at www.nap.edu/catalog/10145.html).

Does the grant competition encourage the participation of new/first-time grantees through a fair and open application process?

Explanation: ATP receives proposals through open competitions in response to broadly advertised notices. Public conferences are held to explain the application process and include appropriate time for audience questions regarding the competition process.

Evidence: ATP's most recent notice of availability of funds appeared in the Federal Register on April 18, 2002 (67 FR 19160-19164); see also numerous GAO reviews of ATP's selection process.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ATP project managers closely track projects during the grant period, review performance, and recommend termination of funds for underperforming projects; ATP's Business Reporting System systematically gathers data on grantee activities and performance, including data for the Composite Ranking System for completed projects.

Evidence: DOC's budget justification, annual DOC performance plans and reports, and individual ATP reports present data from the Business Reporting System and the results of ATP's new Composite Performance Rating System.

Does the program collect performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: ATP's Business Reporting System routinely and systematically gathers data on grantee activities and performance, including data for the Composite Ranking System for completed projects; aggregate data is presented in public reports.

Evidence: DOC's budget justification, annual DOC performance plans and reports, and individual ATP reports present data from the Business Reporting System and the results of ATP's new Composite Performance Rating System.

Has the program demonstrated adequate progress in achieving its long-term outcome goal(s)?

Explanation: ATP has one overarching outcome goal: to accelerate private investment in and development of high-risk, broad-impact technologies. ATP's annual performance measures suggest some progress over time, and while there are many unknowns, special economic studies demonstrate the net public benefits that derive from specific ATP projects. However, effects are difficult to measure and it is hard to obtain the data necessary to identify the actual impact of ATP funding.

Evidence: Data are presented in DOC's budget justification and annual DOC performance plans and reports. External reviews by the National Research Council and numerous economic impact studies attempt to identify economic and social benefits associated with ATP projects. In one impact study, the benefits of just a few projects analyzed in depth exceeded program costs to date (studies available at www.atp.nist.gov); however. these are anticipated benefits that have not yet been realized.

Does the program (including program partners) achieve its annual performance goals?

Explanation: ATP routinely meets or exceeds its annual quantitative performance targets. These targets represent progress towards the program's long-term goal.

Evidence: The three measures provided below collectively represent indicators of performance towards ATP's long term goal. See also DOC's budget justification and annual DOC performance plans and reports

Does the program demonstrate improved efficiencies and cost effectiveness in achieving program goals each year?

Explanation: Annual performance improvements do not yet indicate an increasing return on investment over time. Early estimates on recoupment also seem modest in both the near- and long-term. However, performance targets have been achieved at specified budget levels, and program expenditures have leveraged an equal level of private sector R&D investment.

Evidence: Data are presented in DOC's budget justification and annual DOC performance plans and reports.

Does the performance of this program compare favorably to other programs with similar purpose and goals?

Explanation: ATP is structurally different than other Federal technology programs such as the SBIR and DARPA, however, there are enough similarities to warrant comparison. Comparable data is not currently available to directly compare programs, but given ATP's cost-sharing component and rigorous review process, OMB thinks the program does compare favorably with these other programs.

Evidence: Unlike SBIR, ATP is open to companies of all sizes and is available for all technologies, while much of SBIR funding is agency or mission-specific and DARPA is focused exclusively on DoD mission-driven technology interests. Unlike either SBIR or DARPA, ATP is a partnership program that requires cost-sharing from all grantees.

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: External advisory committees and formal evaluation studies conducted by the National Research Council have found the Program to be effective. However, other indicators (such as the difficulty the program would have recouping its costs) raise questions.

Evidence: See the reports of the ATP Advisory Committee (available at www.atp.nist.gov/atp/adv_com/reports.htm) and the NIST VCAT. See also see NRC, The Advanced Technology Program: Challenges and Opportunities, 1999 (available at books.nap.edu/catalog/9699.html); and NRC, The Advanced Technology Program: Assessing Outcomes 2001 (available at www.nap.edu/catalog/10145.html).