Detailed Information on the
Science and Technology: Rapid Prototyping of Countermeasures Assessment

Conduct assessment of program and project milestones

Collect requirements from customers and make adjustments in current and outyear plans.

Develop additional annual and long-term performance metrics and targets that focus on outcomes that accurately measure the program.

Conduct independent assessments of sufficient scope and quality.

Merge all prior rapid prototyping activities to the Transition program.

Develop 5 Year Research and Development Plan.

Conduct assessment of program and project milestones.

Collect requirements from customers and made adjustments in current and out year plans.

Measure: Percentage of technologies reviewed that are prototyped or commercialized.

Explanation:Data and information regarding the results of technology and operational tests (assess how well the device or prototype performs in pilot or field test) will be available at the end of the development effore and/or the operational tests. In some instances, independent testing will occur after the initial technology or operational tests. Data from this testing will be verified through independent tests and/or through observations and documentation of tests.

Measure: Percentage of technologies prototyped in support of DHS missions.

Explanation:The percentage of technologies prototypes or commercialized is derived by the number of prototypes funded through the Rapid Prototyping program and the number that are accepted by the operational end users each year. In FY 2005, the percentage was set at 3% as the baseline for future years.

Is the program purpose clear?

Explanation: The DHS Secretary, acting through the Under Secretary for Science and Technology, shall have the responsibility for??(6) establishing a system for transferring homeland security developments or technologies to Federal, State, local government,and private sector entities"; The mission of the Rapid Prototyping portfolio is to accelerate, through rapid prototyping, deployment of advanced technologies to address urgent user requirements. The Rapid Prototyping portfolio advances the Directorate's mission to conduct, stimulate and enable research, development, test, evaluation and timely transition of homeland security capabilities to Federal, state and local operational end-users. Accelerating the timeline for development and commercialization of relevant technologies will allow the Science and Technology Directorate to ensure that operational end-users are equipped with the most advanced technologies available. To accomplish this mission, the Rapid Prototyping portfolio will seek to transition two types of technologies. The first are those technologies already in existence that require capability modification to address an immediate or emergent threat. These technologies will undergo a rapid re-engineering process and then be transitioned to the private sector or deployed by government entities on an accelerated schedule to provide defensive countermeasures or response capabilities when needed. The second category is comprised of those technologies that have reached a level of functional maturity to make them candidates for accelerated development. These technology opportunities will undergo operational testing and pilot deployment to determine their ultimate applicability.

Evidence: The Rapid Technology Application Program (RTAP) documentation provides additional program purpose and objectives. Reference RTAP program brief (03/08/05), slide 2. Prototypes within 6-18 months, including field testing, customer involvement critical

Does the program address a specific and existing problem, interest, or need?

Explanation: Provide operational end users with the technology and capabilities to detect and prevent terrorist attacks, means of terrorism and other illegal activities. The Nation's technical superiority in science and technology is key to securing the homeland. We will use, leverage and enhance the vast resources and expertise of the Federal Government, private sector, academic community, non-governmental organizations and other scientific bodies. We will develop new capabilities to facilitate the sharing of information and analysis; test and assess threats and vulnerabilities; counter various threats, including weapons of mass destruction and illegal drugs; and mitigate the effects of terrorist attacks. We will also focus our efforts on developing technology to detect and prevent the illicit transport of chemical, biological, radiological and nuclear materials. We will develop and deploy the capabilities, equipment and systems needed to anticipate, respond to and recover from attacks on the homeland.

Evidence: Evidence can be found in: 1) P.L. 107-296 Homeland Security Act of 2002: "SEC. 313. TECHNOLOGY CLEARINGHOUSE TO ENCOURAGE AND SUPPORT INNOVATIVE SOLUTIONS TO ENHANCE HOMELAND SECURITY. (a) ESTABLISHMENT OF PROGRAM.??The Secretary, acting through the Under Secretary for Science and Technology, shall establish and promote a program to encourage technological innovation in facilitating the mission of the Department (as described in section 101). (b) ELEMENTS OF PROGRAM.??The program described in subsection (a) shall include the following components: (1) The establishment of a centralized Federal clearinghouse for information relating to technologies that would further the mission of the Department for dissemination, as appropriate, to Federal, State, and local government and private sector entities for additional review, purchase, or use. (2) The issuance of announcements seeking unique and innovative technologies to advance the mission of the Department. (3) The establishment of a technical assistance team to assist in screening, as appropriate, proposals submitted to the Secretary (except as provided in subsection (c)(2)) to assess the feasibility, scientific and technical merits, and estimated cost of such proposals, as appropriate. (4) The provision of guidance, recommendations, and technical assistance, as appropriate, to assist Federal, State, and local government and private sector efforts to evaluate and implement the use of technologies described in paragraph (1) or (2). (5) The provision of information for persons seeking guidance on how to pursue proposals to develop or deploy technologies that would enhance homeland security, including information relating to Federal funding, regulation, or acquisition.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: DHS Rapid Prototyping (RTAP) program modeled after the Department of Defense's (DOD) TSWG (Technical Support Working Group), but focuses on Homeland Security instead of DOD missions. See RTAP briefing, slides 3-7. Thus, duplication is avoided because coordination to regularly established with TSWG and others to avoid redundant efforts. Moreover, the descrete needs of DHS personnel stand in contrast to DOD missions and while the model for the program is TSWG, the technologies addressed are geared toward DHS gear.

Evidence:  

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The RTAP program process was developed to ensure that rapid prototypes would be fielded per the program's defined objective and performance goal of 6-18 months. Reference RTAP briefing, slide 8. The RTAP Program was modeled after successful, long-running DoD program (TSWG). By developing RTAP, DHS has been able to increase customer integration, cost savings (of administrative fees), and the ability to provide timely and effective products to RTAP participants (reference RTAP briefing slides 3-7). Products are analyzed with celerity and without any obstacles that have been noted by program managers. No regulatory or congressional mandates slow the rapidity of this program.

Evidence:  

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: By design, strong customer involvement drives the RTAP process, thereby maximizing the likelihood that outputs reach intended beneficiaries within 6-18 months. The RTAP requires direct participation from Federal, State, and local customers to determine and maintain relevance to national needs and DHS missions. The RTAP process was designed to ensure that resources are directly used, minimizing administrative support, to support the DHS counter-terrorism mission. Reference RTAP briefings and RTAP Meeting Notebook.

Evidence:  

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Rapid Prototyping Portfolio does not have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program. This portfolio in the S&T Directorate began in FY 2004; consequently historical information prior to FY 2004 does not exist. In FY 2004, long-term performance measures were established to: - Meet the requirements of section 313 of the Homeland Security Act of 2002 - Support Federal DHS missions and State and Local Responders by addressing chemical and biological threats, explosives detection, training technology tools, improvised nuclear device defeat, and investigative and forensic support topics. There was no baseline prior to FY 2004. These long-term measures ensure compliance with statutory requirements and reduce risk by placing needed homeland security capabilities in the hands of operational end-users in a timely manner. Specific projects to reduce risk are prioritized based on the input from and level of support from DHS users and their commitment to participate in field-tests and subsequent deployment. The Technology Clearinghouse, established in FY 2004 in compliance with section 313 of the Homeland Security Act of 2002, provides an information portal to Federal, State and Local Responders regarding homeland security related technologies, products, and capabilities. Support to State and Local Responders is also provided through direct solicitation of user-group organizations and DHS organizational elements. The Rapid Prototyping program is less than two years old. It is expected that the long-term performance measures will be reviewed and revised based on internal and external program reviews to ensure desired outcomes are most effectively met.

Evidence:  

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The Rapid Prototyping Portfolio has ambitious targets and timeframes for its long-term measures. The Rapid Prototyping program is less than two years old. No baseline existed prior to FY 2004. The projected prototyping performance in FY 2005 is 10-11 prototypes, FY 2005 is currently exceeding the target of 3%. Thus, on average of 20 actual prototypes in the 18 months following contract award in FY 2006 and beyond is an ambitious target because fewer projects will be supported per annum than in FY 2004 and FY 2005 because of anticipated funding for Rapid Prototyping. The Technology Clearinghouse content and functionality was assessed in FY 2005 by User and Subject Matter Expert (SME) Groups, these assessments will guide out-year investments in both content and functionality.

Evidence:  

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The Rapid Prototyping Portfolio has a limited number of specific annual performance measures that can demonstrate progress towards achieving the program's long-term goals, they are listed below. No baseline existed prior to FY 2004. To demonstrate progress toward meeting the long-term goal of 20 successful prototypes in 18 months, an average of 14 prototypes need to be developed each year following the initial start-up years of FY05-06, funding permitting. As an efficiency measure, on average, each prototype is to cost no more than a few million dollars to develop. However, because the program is less than two years old, it is expected that revisions to these measures will occur as the baseline is established and existing measures are reviewed and revised based on internal and external program reviews to ensure desired outcomes are most effectively met. Customer surveys of prototype development will be conducted in FY 2006. These surveys will focus on customer satisfaction in having their needs considered, participation and involvement throughout the project's period of performance, and satisfaction with prototype performance, and willingness to deploy. The effectiveness and progress of the Technology Clearinghouse in meeting the statutory requirements of section 313 of the Homeland Security Act of 2002 and the needs of Federal, State and local Responders will be assessed by internal and external program reviews and user and subject matter experts, respectively.

Evidence:  

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The RTAP is an ambitious program. First, it is a new program that in its first year has generated over 25 requirements. This is a lot to accomplish in the first cycle of the program. Second, the target of delivering new technologies 18 months after contract start is very aggressive. Finally, the RTAP program is working in a much more interactive manner with the users than other RDT&E programs. This new way of doing business is a learning experience for all parties involved. The Technology Clearinghouse was established in FY 2004, the first year of the Rapid Prototyping Portfolio, thereby establishing the baseline for this performance measure. In FY 2005, User and Subject Matter Experts (SME) assessed the content and functionality of the Information Portal. Version 1 of the Information Portal will be available in September 2005, meeting an aggressive programmatic target. User and SME group assessments provide the information necessary for continued improvements in content and functionality. The scope of the improvements will be finalized in FY 2006. To demonstrate progress toward meeting the long-term goal of 20 successful prototypes in 18 months, an average of 14 prototypes need to be developed each year following the initial start-up years of FY05-06, funding permitting. As an efficiency measure on average each prototype is to cost no more than a few million dollars to develop. However, because the program is less than two years old, it is expected that revisions to these measures will occur as the baseline is established and existing measures are reviewed and revised based on internal and external program reviews to ensure desired outcomes are most effectively met.

Evidence:  

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) in the Rapid Prototyping Portfolio are committed to and work toward the annual and/or long-term goals of the program. Rapid Prototyping projects are assigned to professional Federal Program Managers, who have responsibility for project technical and financial oversight. In addition, most of the projects have iterative development or intermediate deliverables. This gives the Program Manager additional insight into the progress of the project and its applicability to user need and support to annual and long-term program goals. If needed, the Program Manager is authorized to terminate projects not meeting the goals of the program. The RTAP process requires all Federal, State, and local partners to commit to and work toward the annual goals of the program. The Science and Technology Requirements Council (SRC) Guidance is the starting point of the RTAP Requirements Generation process. The SRC drives top-down, multi-year RDT&E planning and resource allocation across all DHS Directorates. On an annual basis, program accomplishments will be reported through this council to ensure that both long-term and RTAP Program requirements and goals are being met Partners and contractors are held accountable through accepted government contracting requirements. Included in these requirements is the requirement that authorization to pay invoices must be made by the technical contract managers for each project.

Evidence: No evidence submitted.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evaluations of sufficient scope and quality are not conducted as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need. However, as part of the RTAP process, each prototype proposal receives independent reviews on the potential prototype mission and performance requirements from at least one potential customer, as well as from multiple S&T representatives. The Rapid Prototyping portfolio is less than two years old and projects are just beginning to demonstrate results that can be evaluated. Consequently, independent evaluations (independent of a DHS entity) of sufficient scope and quality have not yet been started because they would not yet be able to measure outputs or outcomes of the programs. Regular independent reviews are scheduled to begin in FY 2006 and will be focused on supporting program improvements as well as evaluating effectiveness and relevance to counter emerging terrorism threats. Additionally, this will ensure the efforts are not duplicative with other research within the Department or with other federal agencies. The Rapid Prototyping program is less than two years old and many of the program efforts in initiated in FY 2004 continued into FY 2005. Program evaluations are currently conducted on individual programs by the executing organizations and the portfolio manager. In addition, some programs have established independent reviews. Three principal options exist for conducting independent evaluations beginning in FY 2006: 1.) The Homeland Security Institute, an FFRDC, could be requested and funded to conduct such an evaluation - a determination of whether the Homeland Security Institute is sufficiently independent of DHS is required.; 2.) A request to the Undersecretary of S&T to task the Homeland Security Science and Technology Advisory Committee to conduct an independent evaluation can be made.; 3.) An independent organization could be contracted to perform an independent evaluation. The specific choice will be made in the first quarter of FY 2006. The S&T Directorate's RDT&E process requires that each portfolio conduct and document a risk assessment to help guide the planning. These risk assessments are independently reviewed by the S&T Program Analysis and Evaluation (PA&E) staff and the Office of Comparative Studies (OCS). In addition, the Comparative Studies office is responsible for conducting independent risk assessments. Although these responsibilities are through contacting mechanisms to others outside of DHS, within the S&T Directorate they provide an independent evaluation of all programs which is reviewed by the S&T Senior Management, and who subsequently provide management guidance to each program. Independent input on the threats, risks, vulnerabilities etc is also provided by the Information Analysis component of DHS.

Evidence:  

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The RTAP program develops its program budget using annual performance targets and budget resources and presents this information in the DHS Five Year Homeland Security Program (FYHSP). The FYHSP captures and expresses the impact of funding, policy, or legislative decisions on the RTAP program. The RTAP program reports all direct and indirect costs which are captured in the DHS Federal Financial Management System (FFMS). This system reports costs that are used to assess overall RTAP performance results. The Rapid Prototyping budget for FY05 and FY06 supports the goal of achieving at least 20 successful prototypes in 18 months, and does account for the fact that not all prototype development efforts will be successful.

Evidence:  

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The RTAP program is a major, new initiative developed to supplement the strategic planning conducted within the S&T Requirements Council. Its focus of providing rapid prototypes to Federal, State, and local end-users was developed and designed to compliment the longer-term Research, Development, Testing and Evaluation (RDT&E) being conducted within the Department of Homeland Security. See RTAP briefings.

Evidence:  

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The DHS RTAP review process, coupled with the DOD TSWG process, are specifically intended to ensure that only those prototypes for which there is not a current program in either DoD or DHS are selected for development. Comparisons of the potential benefits of projects and project alternatives begin with RTAP Requirements Generation Panels, to reduce a collection of general needs to a prioritized list of hard, specific requirements that can be accomplished in time with funds available. The analysis of the Requirements Generation Panels are performed by the actual customer and S&T experts, ensuring credibility throughout the Requirements Generation process. See RTAP briefings and notebook.

Evidence:  

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The RTAP process has a built-in prioritization process to select those prototypes for development. This process is: ?? Clearly documented and independently moderated meetings between potential customers and prototype developers in the S&T Directorate are used to identify Rapid Prototyping priorities and guide RTAP program planning ?? The resulting priorities are used to down select and fund specific rapid prototyping projects ?? As part of the SRC process, the DHS Under Secretaries provide their program goals, objectives, and activities to the Science and Technology Council (SRC) on an annual basis. RTAP is in the process of identifying its first set of current priorities. See RTAP briefings and RTAP Meeting Notebook.

Evidence:  

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: For the Homeland Security Advanced Research Projects Agency (HSARPA) programs, including RTAP, weekly reports are provided to the HSARPA Deputy Director detailing: 1) The date of program origination; 2) the date of funding obligation; 3) the amount of funding that left DHS; 4) the amount of funding obligated (if available); and 5) the amount of funding expended (if available). Furthermore, each prototype development program under RTAP will be assigned an HSARPA program manager to closely monitor the program's progress. Each RTAP prototype development program is expected to deliver a working prototype which meets the customer-agreed-upon performance requirements within 6-18 months of contract award, and with no more than a few million dollars total funding.

Evidence:  

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: All partners are held accountable for requirements generation, cost, schedule, and performance results. The RTAP program: ?? Identifies the program managers and associated team members in each countermeasure or mission portfolio are who are accountable for program results (see roster in RTAP notebook) ?? Establishes clearly defined or quantifiable performance standards & performance requirements for those managers as defined in the RTAP notebook ?? Works with customers to establish specific performance standards for each countermeasure or mission related program If funding expenditure and/or other data indicate that an RTAP program will exceed the 6-18 month development goal, or exceed its funding allocation, the program will be considered for termination and no further funding will be allocated to the program. See RTAP briefings and notebook.

Evidence:  

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Prior to the RTAP program, the Rapid Prototyping portfolio funded programs through the DoD TSWG process. Over 90% of FY03/04 funding for DHS-sponsored TSWG programs has been obligated for rapid prototyping programs to serve DHS missions (see DHS-TSWG Awards spreadsheet). The RTAP program has just begun the requirements generation process, and will likely begin to award contracts for FY05 programs in 4QFY05. See RTAP briefings and notebook.

Evidence:  

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: In developing the RTAP process, regular procedures were developed and are now in place and are being utilized to develop requirements and competitively source projects to enable the marketplace to drive efficiencies and cost effectiveness. One efficiency measure -- the per unit cost of an RTAP program -- is estimated at $3-4M dollars maximum. The cost and performance goals for RTAP compare favorably to similar programs in the DoD TSWG program. For example, in FY00 TSWG funded approximately 200 programs for $70 million, an average of $3 million per program (see DoD Special Briefing on the Technical Support Working Group dated Nov. 29, 2001.) Efficiency is built-in to the RTAP program by requiring that each prototype development program be completed within 6-18 months and cost no more than a few million dollars. Each program will be competed via a Broad Agency Announcement competitive solicitation process.

Evidence:  

Does the program collaborate and coordinate effectively with related programs?

Explanation: The closest related program to the DHS RTAP program is the DoD's TSWG program, which RTAP is modeled after. DHS personnel serving on the TSWG will also serve in the RTAP program, to help ensure the programs are complementary, not duplicative. Furthermore, with the startup of the RTAP program, resource allocations will change, with the S&T Directorate significantly reducing its funding contribution to the TSWG in order to fund more DHS-relevant RTAP programs. See RTAP briefings.

Evidence:  

Does the program use strong financial management practices?

Explanation: The DHS IG is initiating an audit of DHS's FY05 consolidated financial statements, to be performed by an independent accounting firm. See attached memorandum dated Feb 9, 2005 for Adm. Loy from R.L. Skinner. Also see the answer to question 3.1 above.

Evidence:  

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The RTAP program as designed includes a methodology to control and correct management deficiencies. Furthermore, every prototype development program will be assigned a government program manager who will maintain ownership of, and carefully monitor, the prototype's development progress. Any prototype development program within RTAP which fails to develop a workable prototype that meets customer requirements within the 6-18 month timeframe for a cost of no more than a few million dollars will be reviewed and will likely be terminated with no further funding expended.

Evidence:  

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The RTAP program uses a BAA competitive solicitation process.

Evidence:  

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Rapid Prototyping Portfolio has demonstrated adequate progress in achieving its long-term performance goals. This is the first year for RTAP so no projects have been completed. However, RTAP is making progress in its planned program execution. The requirements panels were selected. The panels produced many well-specified requirements. Based on funding, the list was narrowed and sent to the Under Secretaries for validation. In all cases but one, the requirements were validated. The exception was the Commandant of the U.S. Coast Guard who stated that his two top priorities were not covered. He asked for replacement of those topics for which the USCG is the main user with his two priorities. We are currently in the process of finalizing the replacement topics. The next steps are solicitation, evaluation, contracting, and execution. Prior year funding from rapid prototyping was used to support projects through TSWG. Many of these projects have shown great promise and have been funded for later phases and greater than 5% of the funded projects have been prototyped to date, exceeding the FY 2005 target. The first milestones of the RTAP process, customer/developer prioritization and requirements generation meetings - have been successfully completed on schedule. The program is on track. The Technology Clearinghouse was established in FY 2004, thereby meeting the requirement in Section 313 of the Homeland Security Act of 2002. User and Subject Matter Expert (SME) groups assessed content and functionality needed in FY 2005 and this guided the development of Version 1 of the Information Portal which will be released to the public in September 2005. Additional User and SME group assessments will be conducted in FY 2006 to guide further development of content and functionality and provided customer satisfaction results.

Evidence:  

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Rapid Prototyping Portfolio (including program partners) appears to acheive its annual performance goals. The Rapid Prototyping program is less than two years old. A two year assessment of effectiveness will be conducted at the end of FY 2005 that will provide the baseline for futures annual measures. The percentage of capabilities developed and subsequently adopted for use will be tracked and additional annual measures established. Individual project performance is monitored and also contributes to meeting annual performance goals. The FY 2006 Congressional Budget Justification also indicated performance goals and historical performance in meeting annual goals. The development of actual prototypes has exceeded the established target of 3% prototyped in FY 2005 (currently exceeding 5% and expected to be near 10% by the end of FY 2005). The FY 2004 goal of establishing the Technology Clearinghouse was met. The FY 2005 goal of conducting User and SME group assessments of the Information Portal and implementing Version 1 with public release by end of FY 2005 will be met.

Evidence:  

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The new efficiency measure of 6-18 months development time at a few million dollars cost are new being implemented rigorously in the RTAP program, whereas they were not explicitly part of the Rapid Prototyping portfolio earlier. See RTAP briefings.

Evidence:  

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The cost and performance goals for RTAP compare favorably to similar programs in the DoD TSWG program. For example, in FY00 TSWG funded approximately 200 programs for $70 million, an average of $3 million per program (see DoD Special Briefing on the Technical Support Working Group dated Nov. 29, 2001.)

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The Rapid Prototyping program is less than two years old and many of the program efforts in initiated in FY 2004 continued into FY 2005. Program evaluations are currently conducted on individual programs by the executing organizations and the portfolio manager. In addition, some programs have established independent reviews. The RTAP program has not existed long enough to undergo any independent evaluations regarding effectiveness and results. Independent evaluations (independent of a DHS entity) of sufficient scope and quality have not yet been conducted on a regular basis to support program improvements since the first results of projects initially funded are just now being realized. These evaluations have not yet been conducted because the programs within the Rapid Prototyping portfolio are new and are just beginning to demonstrate results that can be effectively evaluated. Independent reviews that will focus both on technical quality and managerial effectiveness are scheduled to begin in FY 2006.

Evidence: