Detailed Information on the
Commerce Small Business Innovation Research Program Assessment

Working with the NOAA Research Council to redirect the SBIR Program to be more consistent with the NOAA Strategic Plan.

NIST SBIR - Working with the Small Business Administration to develop long-term outcome-oriented measures

Working with the small business awardees and contract Technical Representatives to more closely monitor the technical and administrative aspects and progress of each contract to ensure timeliness of deliverables.

Working to develop and implement interim annual performance measures to improve processing times of proposals.

Is the program purpose clear?

Explanation: The SBIR Program is legislatively mandated. The overall purpose of the SBIR is to strengthen the role of innovative small businesses in Federally-funded research by stimulating and fostering scientific and technological innovation, increasing the commericalization of Federal R&D, and encouraging participation by minority and disadvantaged persons.

Evidence: In 1982, Public Law 97-219 established a five-year, government-wide, Small Business Innovation Research Program. This Program has been extended 3 times (through September 30, 2008) by Public Law 99-443, Public Law 102-564, and most recently, by Public Law 106-554.

Does the program address a specific and existing problem, interest, or need?

Explanation: The SBIR program is based on two primary assumptions, that small businesses are excluded from the federal R&D procurement process and that there is an under-investment in scientific and technological innovation. Most of NIST's SBIR funding is set aside from the Advanced Technology Program (ATP), which also aims to foster technological innovation. Small businesses are already eligible to apply for ATP funding; in fact, 77% of ATP's FY02 awardees were small businesses. Within NOAA, small businesses are also eligible for R&D funding and contracts from existing programs. For example, commercial organizations are eligible for NOAA's Saltonstall-Kennedy grant program, which provides funding for research areas, such as aquaculture, that are also funded under the SBIR program. Given these existing opportunities, there is no evidence that a need exists at Commerce to address the problems on which the SBIR program is based.

Evidence: See www.atp.nist.gov for descriptions and statistics on NIST's Advanced Technology Program. See www.noaa.gov for descriptions and statistics on non-SBIR funding opportunities.

Is the program designed so that it is not redundant or duplicative of any Federal, state, local or private effort?

Explanation: Per the legislation, any Federal agency with extramural R&D budgets of $100 million or more must reserve 2.5% for SBIR. The law does not allow any flexibility for agencies to fulfill the purposes of the SBIR program through any other means. As explained above, the purposes of the SBIR program are already addressed through existing programs, so to further set aside funding for SBIR is redundant.

Evidence: See Public Law creating the SBIR.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: As mentioned above, the law does not provide flexibility for agencies to fulfill the purposes of the SBIR program though other programs. Such flexibility could allow agencies to more effectively meet their needs while still promoting involvement of small businesses. As a mandatory "tax" on R&D programs, the design also prevents agencies from making discretionary decisions on the appropriate level of funding. The SBIR program takes funding from other R&D efforts and restricts it in a manner that may not be the most effective or efficient means of meeting the agency's mission.

Evidence: See Public Law creating the SBIR.

Is the program effectively targeted, so program resources reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: All SBIR resources are used to fund R&D proposals from small businesses responding to an annual solicitation highlighting NIST-specific R&D needs. Each proposal is evaluated to ensure it meets a defined set of criteria.

Evidence: See NIST SBIR website: www.nist.gov/sbir and Public Laws creating the SBIR.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: To date, Commerce has not developed long-term performance goals for its SBIR program. Commerce uses the SBIR program as a vehicle to receive R&D while supporting the overarching purpose of SBIR. The annual goal of the Commerce program is to ensure the alignment of each SBIR project with bureau missions while complying with SBA issued guidelines and regulations for SBIR programs.

Evidence: No goals or measures are identified in DOC's Annual Performance Plan or Strategic Plan.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: As discussed above, Commerce has not developed long-term performance goals for this program.

Evidence: No goals or measures are identified in DOC's Annual Performance Plan or Strategic Plan.

Does the program have a limited number of specific annual performance measures that demonstrate progress toward achieving the program's long-term measures?

Explanation: The Commerce SBIR program does not have a specific set of performance measures to assess overall progress in the program. However, each individual contract does have performance goals that must be met prior to award.

Evidence: No goals or measures are identified in DOC's Annual Performance Plan or Strategic Plan.

Does the program have baselines and ambitious targets and timeframes for its annual measures?

Explanation: As discussed above, Commerce does not have annual measures for this program.

Evidence: No goals or measures are identified in DOC's Annual Performance Plan or Strategic Plan.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, etc.) commit to and work toward the annual and/or long-term goals of the program?

Explanation: As discussed above, Commerce has not developed long-term goals for this program. However, SBIR clients are required to agree and meet the terms and conditions of their contracts which include the delivery of progress reports and/or other technical deliverables (according to solicited stated R&D goals).

Evidence: Terms and conditions, including deliverables, are clearly stated in each SBIR contract.

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: GAO conducts periodic reviews of the SBIR program. The Commerce IG also audited the program in 2000. In addition, the National Research Council is working on a multi-year, government-wide study to evaluate the effectiveness of the SBIR program.

Evidence: See 1987, 1989, 1995, 1998, 1999 GAO reports and 2000 DOC IG Audit.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: In accordance with the legislation, the annual NIST SBIR budget is 2.5% of its extramural R&D budget; the SBIR program is not funded through specific budget requests and resources are not tied to results.

Evidence: See Public Law creating the SBIR.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: Commerce has not developed long-term goals or measures to evaluate program effectiveness. However, NIST has developed a strategic improvement plan for the SBIR program to improve processes and develop performance measures.

Evidence: No goals or measures are identified in DOC's Annual Performance Plan or Strategic Plan. NIST's Strategic Plan is available upon request.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Each SBIR awardee is required to submit regular deliverables. Once these deliverables are inspected by Commerce and technically accepted, payments are made on the contract. Because these projects are R&D in nature, the process measures are in the form of deliverables which allow Commerce to meet its mission related needs.

Evidence: Commerce maintains records of each accepted SBIR deliverable.

Are Federal managers and program partners (grantees, subgrantees, contractors, cost-sharing partners, etc.) held accountable for cost, schedule and performance results?

Explanation: SBIR awards are made through fixed-price contracts which provide a framework for accountable delivery of the proposed R&D. The terms and conditions of the contracts provide a schedule for the conduct of the R&D and interim progress reports and deliverables. Payments are not issued until Commerce receives and deems the deliverable technically acceptable.

Evidence: The SBIR Program awards contracts as its funding agreement and thus relevant FAR regulations are invoked and observed.

Are all funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: All funds allocated to the SBIR Program are obligated during the current year for the exclusive purpose of funding SBIR contracts. Any funds that remain in the allocation because of insufficiency to fund another complete project are carried over to the following fiscal year allocation.

Evidence: Annual reports prepared for SBA detailing budget expenditures.

Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, approporaite incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: There are no efficiency measures or other indicators available to determine efficiences and cost effectiveness in the program.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: The NOAA Program working with the SBA coordinates its program with all other Federal SBIR agencies. Federal program manager meetings, 3 National Conferences, and numerous program to program communications all aid coordination.

Evidence: SBA Annual Reports, National Conference Procedings, NOAA annual award abstracts

Does the program use strong financial management practices?

Explanation: The SBIR program adheres to established agency financial management practices and principles.

Evidence: See NIST and NOAA audited Annual Financial Statements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Both NIST and NOAA have taken steps to address management deficiencies. The DOC IG audit in 2000 identified suggested program improvements, all of which were adotped. Within NIST, the entire SBIR program is managed by a staff of 1 FTE. The size of the program allows for systematic program management evaluation during the SBIR administrator's annual performance review.

Evidence: 2000 DOC IG Audit; Annual performance reviews.

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: Both NIST and NOAA publish an annual solicitation outlining R&D topic areas. In NIST, applicants submit proposals in the topic area and two experts familiar with the research topic evaluate and score each proposal according to a set of criteria outlined in the annual solicitation. Final selection is based on the technical scores, combined with a priority ranking provided by the laboratory sponsoring the research area. In NOAA, a selection committee evaluates the proposal according to the established criteria and makes the final selection.

Evidence: See the NIST SBIR website at www.nist.gov/sbir and the NOAA SBIR website at www.oar.noaa.gov/ORTA/SBIR for annual solicitations and criteria.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: In NIST, SBIR awards are made through fixed-price contracts which provide a framework for accountable delivery of the proposed R&D. The terms and conditions of the contracts provide a schedule for the conduct of the R&D and interim progress reports and deliverables. Payments are not issued until NIST receives and deems the deliverable technically acceptable. NOAA assigns a technical expert to monitor each contract to ensure adequate progress is being made and that the conditions of the contract are being met.

Evidence: Specific terms and conditions are included with each contract issued.

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: NIST and NOAA publish abstracts of proprosed SBIR R&D along with the client company names and addresses on their websites and in the annual report to SBA.

Evidence: SBA Annual Reports; Awards section of the NIST SBIR website at www.nist.gov/sbir; NOAA Abstracts of Annual Award Winners at www.oar.noaa.gov/ORTA/SBIR.

Has the program demonstrated adequate progress in achieving its long-term outcome performance goals?

Explanation: As discussed in section 2, Commerce has not developed long-term performance goals for its SBIR program.

Evidence: No goals or measures are identified in DOC's Annual Performance Plan or Strategic Plan.

Does the program (including program partners) achieve its annual performance goals?

Explanation: As discussed in section 2, Commerce has not developed annual performance measures to assess the overall SBIR program.

Evidence: No goals or measures are identified in DOC's Annual Performance Plan or Strategic Plan.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

Explanation: There are no efficiency measures or other indicators available to determine efficiences and cost effectiveness in the program.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., that have similar purpose and goals?

Explanation: There is no performance data available to compare Commerce's SBIR program with other agency SBIR programs. GAO and NRC reviews have not made qualitative comparisons of Commerce's program with other agency programs. Some overall reviews of SBIR have compared it to other Federal technology R&D programs, but results have varied.

Evidence:  

Do independent and quality evaluations of this program indicate that the program is effective and achieving results?

Explanation: GAO conducts periodic reviews of the entire SBIR program. However, these reports have not adequately evaluated the effectiveness and results of the program, nor have they evaluated the effectiveness of the program's overall design. The National Research Council is currently working on a multi-year, government-wide study to evaluate the effectiveness of the SBIR program.

Evidence: See GAO reports on the SBIR program. The NRC report is due to Congress in FY05.