Detailed Information on the
Science and Technology: SAFETY ACT Assessment

Develop Five-Year Research and Development Plan.

Develop FY 2010 Spendplan.

Increase outreach efforts to broaden the base of applicants.

Develop SAFETY Act program portion of the S&T five year research and development plan.

Measure: Number of applications for SAFETY Act coverage submitted.

Explanation:The goal is to continue to increase the number of applications OSAI receives each year by 10%. By continuing to increase the number of applications the SAFETY Act program will continue to be effective in enabling the widespread commercial availability of effective anti-terrorism technologies. The number of applications is also an indicator of long-term success; it is a reflection of the homeland security market's desire to develop and deploy anti-terrorism technologies and the necessity of a program that will enable this. Potential spike up in applications may reflect external concerns such as FY 2008 Presidential elections.

Measure: Number of SAFETY Act "transition" (new, highly innovative) technologies awarded, i.e. a technology or service that is a new entrant into the homeland security arena that is emerging from a developmental status toward widespread commercial availability.

Explanation:In order to stay up to date with the continually changing nature of terrorism, OSAI will seek out those evolving technologies that can serve a homeland security mission and provide coverage to enable their transition into the commercial market, at a rate of 20% a year. A "transition" technology is defined as any technology that is awarded Developmental Testing and Evaluation (DT&E) Designation, and those that can be considered new and innovative (I.E. a new technological application in the homeland security arena). OSAI is actively seeking out these technologies in an effort to address the ever-changing nature of terrorism. The SAFETY Act program is the only federal program that attempts to help industry transition these developmental technologies into the commercial marketplace.

Measure: Number of eligible SAFETY Act awardees that apply for renewal of their coverage.

Explanation:The goal is to achieve renewal requests from at least 50% of those awardees who are eligible to apply for renewal during that fiscal year, starting in FY 2009 (the year that the first SAFETY Act awards will expire). Achieving the renewal target will provide a measure of the program's continued value to the applicants and their customers. Those applicants that apply for renewal are those that consider the SAFETY Act program to be an effective risk and litigation management tool and are willing to put in the significant time and effort to attempt to renew their coverage. Renewals are considered to have a direct correlation with the continued effectiveness of the program and serve as an indicator of value.

Measure: Number of outreach initiatives and events completed.

Explanation:The more exposure potential applicants have to the program, the more likely they are to apply, thus increasing the number of full applications.

Measure: Number of Designations and Certifications awarded.

Explanation:This will enhance widespread commercial availability of effective anti-terrorism technologies.

Measure: Average processing time for SAFETY Act application.

Explanation:This measure shows the efficient use of resources and prompt delivery of benefit to applicants and their customers. The program efficiently process in a timely fashion even with continued increases in application rates.

Measure: Average processing cost per application.

Explanation:This measure reflects the efficient use of resources. The program has reduced the cost of processing applications to accomodate the increases in application rates.

Is the program purpose clear?

Explanation: The Office of SAFETY Act Implementation (OSAI) was created to implement the Support Anti-terrorism by Fostering Effective Technologies Act of 2002 (the SAFETY Act). The Act was created by Congress to ensure that the threat of liability does not deter potential developers, manufacturers or sellers of anti-terrorism technologies from developing, deploying, and commercializing technologies that could save lives. To accomplish this purpose, the Act creates certain liability limitations for claims arising out of, relating to, or resulting from an act of terrorism where "Qualified Anti-Terrorism Technologies" (QATTs) have been deployed. OSAI evaluates the efficacy of the potential QATTs to determine if they could effectively provide an anti-terrorism capability. If OSAI deems the anti-terrorism technology a QATT, it will determine the appropriate level of liability protections based upon a set of established criteria.

Evidence: Subtitle G of Title VIII of the Homeland Security Act of 2002, P.L. 107-296. SAFETY Act Final Rule (6 CFR Part 25).

Does the program address a specific and existing problem, interest, or need?

Explanation: The SAFETY Act Implementation program addresses the need to protect developers, manufacturers and sellers of qualified anti-terrorism technologies from liability risks that could deter the development, deployment and commercialization of effective anti-terrorism technologies. Technologies developed for military use that could be modified for homeland use were not being deployed due to liability insurance concerns. In addition, manufacturers and sellers of anti-terrorism technologies feared costly legal claims that could result if their technology was deemed to be the proximate cause of losses sustained by third parties as a result of an Act of Terrorism. Similar concerns were voiced by potential customers. The program provides a means for manufacturers and sellers of effective anti-terrorism technologies to obtain certain liability protections that would be applicable not only to the seller of the technology but to every entity that purchases the technology, as the SAFETY Act permits claims to be filed only against the seller of a QATT. These incentives will help encourage the deployment of new anti-terrorism technologies to the market and consequently will make America safer.

Evidence: Liability protections under the SAFETY Act include Designation: limiting the liability of a seller of an anti-terrorism technology to the amount of liability insurance DHS requires the seller to carry; Certification: total liability immunity for the seller (enabling the seller to assert the Government Contractor Defense for claims arising from acts of terrorism); and Developmental Testing and Evaluation Designation (DT&E): limited liability protection for developers of anti-terrorism technologies that show potential but for which additional evidence of effectiveness in an operational setting is needed. Recent court precedent ($2 billion court judgment in 2005 against the N.Y. Port Authority from the 1993 World Trade Center terrorist bombing), ongoing litigation against various commercial air carriers and security firms from the 9/11/2001 terrorist attacks, and the 2006 claims settlement ($4.15 billion) against the N.Y. Port Authority for damage to the Twin Towers incurred on 9/11/2001 underscore the validity and currency of these liability concerns. Subtitle G of Title VIII of the Homeland Security Act of 2002, P.L. 107-296. SAFETY Act Final Rule (6 CFR Part 25).Willis, Inc. Insurance Brokerage Fact Sheet, TRIA S 101 (b) (2)

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The SAFETY Act Implementation program is the only program that evaluates and qualifies effective anti-terrorism technologies for liability protections. Congress was clear, both in the text of the SAFETY Act and in the Act's legislative history, that it can and should be a critical tool in expanding the creation, proliferation and use of anti-terrorism technologies. The SAFETY Act applies to a broad range of technologies, including products, services and software, or combinations thereof, as long as the Secretary of the Department of Homeland Security, as an exercise of discretion and judgment, determines that the technology merits SAFETY Act protections. SAFETY Act liability protections apply only to claims related to Acts of Terrorism as defined and declared by the Secretary of the Department of Homeland Security.

Evidence: Subtitle G of Title VIII of the Homeland Security Act of 2002, P.L. 107-296. SAFETY Act Final Rule (6 CFR Part 25).

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The SAFETY Act Implementation program processes have been independently reviewed and evaluated by the Homeland Security Institute (HSI), MARSH Insurance group and most recently, Booz Allen Hamilton (BAH). Recommendations have been reviewed, analyzed, and implemented to the maximum extent possible. OSAI strives to continue to find ways to maximize the results of these reviews. Following the introduction of the SAFETY Act Final Rule in June of 2006, OSAI has made significant progress towards program efficiency and effectiveness. Prior to the implementation of the Final Rule, there had been a significant backlog of applications, some more than a year old. The Final Rule allowed the new management team, with the help of the results from both the HSI and MARSH reviews, to address these deficiencies (there is currently no backlog of applications and the average processing time is 113 days). The Final Rule introduced the Pre-Application, which provides an opportunity for an applicant to consult with OSAI prior to submission of a full application. The purpose of this consultation is to address the applicant's questions regarding the application process as well as the criteria the Department will use to evaluate the QATT. In addition, OSAI has made significant improvements to the website, including the ease of the application kit, as well as the availability of the helpdesk (there is both a 24-hour number as well as an electronic email option). A list of supporting information, namely a user guide is available which provides the step-by-step registration and application submission process and a SAFETY Act 101 briefing which highlights the different levels of protections and the supporting documentation needed for each.

Evidence: HSI's SAFETY Act Program Assessment review, Section 5, pg 63-95 MARSH's Process for Enhancing SAFTY Act Insurance Analysis, Section 7, pg 33-34 BAH's Review and Audit, pg. 3 OSAI year end accomplishments and report card 2008 DHS Congressional Justification

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The SAFETY Act Implementation program is specifically designed for the sole purpose of assisting developers, manufacturers, and sellers of anti-terrorism technologies with obtaining sufficient liability protections so that they will not be deterred from developing and widely deploying their technologies throughout the United States. The SAFETY Act Implementation program accomplishes this through extensive outreach and educational efforts geared toward potential applicants, with special emphasis on reaching those business sectors who can provide effective technologies for known anti-terrorism capability gaps. The SAFETY Act is a voluntary program, and involves an evaluation of evidence of the technical efficacy of the applicant's described anti-terrorism technology. As such, procedures have been implemented to enable the applicants to submit a complete application that is capable of being evaluated in a timely manner (120 calendar days), as set forth in the SAFETY Act's Final Rule. Pre-submission consultations are strongly encouraged to provide a screening service for those interested in applying, and to answer any questions prospective applicants may have before their application is submitted. Workshops and a 24/7 help desk are available to assist the applicant with any question that may arise during the application process. The SAFETY Act program is designed to ensure that any unintended subsidies are avoided.

Evidence: OSAI year end accomplishments and report card, FY 2008 DHS Congressional Justification, BAH's Review and Audit

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The purpose of the SAFETY Act program is to encourage the development and deployment of anti-terrorism technologies by providing limited liability protection to the sellers and manufacturers of effective anti-terrorism technologies. The measures used to track the success of the SAFETY Act program are: ?? Number of applications for SAFETY Act coverage submitted; ?? Number of SAFETY Act "transition" technologies, i.e. a technology or service that is a new entrant into the homeland security arena that is emerging from a developmental status toward widespread commercial availability and; ?? Percentage of eligible SAFETY Act awardees that apply for renewal of their coverage.

Evidence: Internal weekly, monthly, and quarterly status reports; SAFETY Act Congressional Justification FY08 and FY09; Annual program report card; Hill brief to staffers.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: For the FY 2007 - FY 2012 timeframe, OSAI has established the following targets for its long-term measures: ?? Increase the number of applications by 10% each fiscal year from the FY 2006 base (program received first applications at the beginning of FY 2004). ?? Increase the number of "transition" (new, highly innovative) technologies by 20% each fiscal year from the FY 2006 base. ?? Achieve renewal requests from at least 50% of those awardees who are eligible to apply for renewal during that fiscal year, starting in FY 2009 (SAFETY Act awards are for a minimum of 5 years, FY 2009 is the year that the first SAFETY Act awards will expire). By continuing to increase the number of applications the SAFETY Act program will continue to be effective in enabling the widespread commercial availability of effective anti-terrorism technologies. In order to stay up to date with the continually changing nature of terrorism, OSAI will seek out those evolving technologies that can serve a homeland security mission and provide coverage to enable their transition into the commercial market. Achieving the renewal target will provide a measure of the program's continued value to the applicants and their customers. Due to an intense managerial focus in FY 2007 to stabilize and improve program execution, these are ambitious targets for the future of the program. OSAI will strive to grow the program by at least 10% a year while maintaining the cost of application flat. OSAI strives to enable "effective" anti-terrorism technologies; if this measure percentage is too high, OSAI cannot ensure that these new technologies will serve as effective terrorism deterrents. This is also true for the long-term performance percentage of transition technologies. It is essential that innovative technologies, although they are in a developmental stage when they receive coverage, serve a homeland security purpose and can and should be transitioned into the commercial marketplace. Finally, if the SAFETY Act program is expected to continue to serve an essential homeland security mission, at least 50% of previous applicants should apply for renewal of their coverage. Significant retention of applicants would demonstrate that awardees believe that the program is providing value. This is also an ambitious target because as a result of the ever-changing nature of terrorism some technologies have the potential to become obsolete with the passage of time.

Evidence: Internal weekly, monthly, and quarterly status reports; SAFETY Act Congressional Justifications; Annual program report card; SAFETY Act Presentation brief; Transition Technologies snapshot.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The annual performance measures used to demonstrate progress toward achieving the aforementioned long-term goals are: ?? Number of outreach initiatives and events completed (the more exposure potential applicants have to the program, the more likely they are to apply, thus increasing the number of full applications). ?? Number of Designations and Certifications awarded (this will enhance widespread commercial availability of effective anti-terrorism technologies). The efficiency measures used will be: ?? Average processing time for SAFETY Act application (efficient use of resources and prompt delivery of benefit to applicants and their customers). ?? Average processing cost per application (efficient use of resources). In FY 2007, OSAI made dramatic improvement in each of these measures: the average processing time was reduced by 31% - putting it below the 120 days expected by Congress, and the average cost of processing an application was reduced by 41%. These reductions were achieved as a result of intense managerial focus on stabilizing and improving processes and procedures and program execution. Having maintained these levels for approximately one year, OSAI does not believe it is realistic to achieve further significant reductions in these measures without compromising the quality of application reviews. Nonetheless, in spite of the trend of many applications being more technically complex and broad in scope, thus requiring a more expansive review, OSAI intends to use the FY 2006 levels as the baseline for measuring and maintaining program efficiency and cost effectiveness through FY 2012. This will be an ambitious undertaking.

Evidence: Internal weekly, monthly, and quarterly status reports; SAFETY Act Congressional Justifications; Annual program report card; SAFETY Act Presentation brief.; Outreach Activities report.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The SAFETY Act program is a voluntary program, as such, outreach efforts to educate the sellers and manufacturers of the benefits of the SAFETY Act program is essential in ensuring continued success. Over the FY 2007 to FY 2012 timeframe: ?? Increase the outreach events by 5% each fiscal year. The more exposure potential applicants have to the program the more likely they are to apply, thus increasing the number of full applications. ?? Increase the number of SAFETY Act Designations and Certifications awarded by 5% each year. This ensures an increase in the number of potential overall awards as well as the number of potential "transition" technologies. ?? The goal is to continue to maintain the average processing time under 120 days without sacrificing evaluation quality. ?? OSAI intends to adhere to the long-term goal of increasing the applications for the program by 10% a year while keeping the average cost to process an application at a level at least as low as that achieved during FY 2007 (during which OSAI achieved a substantial decrease - 41% -- in average cost over FY 2006). In FY 2007, OSAI received a large increase in the number of applications, most likely due to the implementation of the Final Rule. OSAI intends to continue to steadily increase the number of applications, through increasing outreach efforts, while keeping both the average processing time and the average cost per application static. With the increase in applications, OSAI can expect to have to utilize more resources to ensure full review and complete reviews. Thus, by keeping with the congressionally mandated timeline of 120 days and maintaining the average processing cost achieved at the FY 2007 level, OSAI will increase overall program efficiency. These percentage increases are considered extremely ambitious. OSAI was able to achieve a significant reduction in these measures in FY 2007 as a result of intense managerial focus on stabilizing and improving processes and procedures and program execution. Having maintained these levels for approximately one year, OSAI does not believe it is realistic to achieve further significant reductions in these measures without compromising the quality of application reviews. Nonetheless, in spite of the trend of many applications being more technically complex and broad in scope, thus requiring a more expansive review, OSAI intends to use the FY 2006 levels as the baseline for measuring and maintaining program efficiency and cost effectiveness through FY 2012. Therefore these targets represent an intense effort required on the part of OSAI.

Evidence: Internal weekly, monthly, and quarterly status reports; SAFETY Act Congressional Justifications; Annual program report card; SAFETY Act Presentation brief; Outcome measures as identified in 2.1; Output measures as identified in 2.2; Efficiency measures as identified in 2.3.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The Institute for Defense Analyses (IDA), the primary entity providing contractor and program support to the Office of SAFETY Act Implementation, is a program partner. IDA conducts the technical and economic reviews and provides administrative processing support for each SAFETY Act application and reports the results of each review to OSAI. IDA tracks and provides frequent reports to OSAI on their performance as it relates to accomplishing the SAFETY Act's stated efficiency goals (average processing times, average cost to process an application). In addition, on November 7, 2007, the FAR (Federal Acquisition Regulations) Council published an interim FAR rule addressing the use of the SAFETY Act in Federal procurements. A DHS department-wide Management Directive "Ensuring Maximum Utilization of the SAFETY Act" is undergoing final review and will be issued to raise awareness and encourage use of the SAFETY Act as appropriate in Departmental procurements and programs.

Evidence: Weekly reports of SAFETY Act stats; monthly financial reports; monthly processing time reports, Management Directive 143-01.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: The SAFETY Act program contracted the Homeland Security Institute (HSI) to analyze and evaluate the application processes and identify areas of improvements in FY 2006. HSI was asked to evaluate the major aspects of the program and recommend changes to enhance its success. A three part study was conducted which included a review of current OSAI processes, interviews with participants in the process, and an analysis of the procedures and management practices DHS uses to make Designation and Certification decisions. Together with OSAI, HSI developed a set of six study questions to guide the evaluation. Marsh and McLennan, one of the nation's largest insurance brokerage firms, was also contracted in FY 2007 to provide an independent evaluation of the insurance liability methodology and processes used to assess the risk associated with anti-terrorism technologies to develop the insurance requirement for applicants. Recommendations were reviewed, analyzed and implemented to the maximum extent possible. In FY 2008, Booz Allen Hamilton (BAH) was contracted to audit the results of implementing the recommendations made by HSI and to identify those areas that warrant further attention. The purpose was to assist OSAI in addressing whether and to what extent the program is achieving its goals and objectives, BAH sought to achieve two objectives in their evaluation; 1) examine HSI's SAFETY Act Program Analysis and determine if OSAI has implemented HSI's recommendations and 2) recommend measures to close the capability gap and improve program results. Booz Allen Hamilton concluded that OSAI has made significant improvements to its processes, procedures and productions including the implementation of many of HSI's recommendations.

Evidence: Homeland Security Institute's SAFETY Act Program Analysis Report, Nov 2005; Marsh's Process for Enhancing SAFETY Act Insurance Analysis, Nov. 2006; Booz Allen Hamilton Audit Report, March 2008.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The operations of the SAFETY Act program allow an independent third party to provide an efficacy evaluation of a qualified anti-terrorism technology. If funding is not sufficient, the SAFETY Act program will not be able to meet the output, outcome and efficiency measures, and therefore be ineffective in ensuring the continued development and deployment of anti-terrorism technologies. The primary component in formulating the program's budget request for a given fiscal year is the forecast of number of applications for that fiscal year and the estimated average cost to process each application. The estimated average cost to process an application is based on the requirement to employ sufficient assets to enable the processing time measure (120 days), a Congressional expectation, to be met. In addition, the SAFETY Act program reports indirect costs associated with the management of the program. The program submits budget requests that link direct costs with management and administration (indirect) costs to more effectively reflect the amount it takes to accomplish measures and achieve long-term goals.

Evidence: Budget justifications; internal weekly, monthly, and quarterly status reports; SAFETY Act Congressional Justifications; Annual program report card; SAFETY Act Presentation brief; Outcome measures as identified in 2.1; Output measures as identified in 2.2; Efficiency measures as identified in 2.3.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The SAFETY Act program has established ambitious, long-term performance goals (Q2.2) and a limited number of annual performance measures (Q2.3) that demonstrate progress toward achieving the long-term goals, is collecting data related to these goals and measures, and is using the data collected to ensure effective and efficient operational performance. OSAI has implemented an execution plan employing analytics to drive management decisions, priorities, resource allocations, and actions.

Evidence: Division of Transition Strategic Goals and Objectives; Science and Technology Goals and Objectives; SAFETY Act leadership's performance plans.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Office of SAFETY Act Implementation (OSAI) collects various weekly progress reports and monthly financial reports from its key partner, the Institute for Defense Analysis (IDA), which assists in carrying out the program's day to day operations. These reports include information to track the status of applications from receipt to award, processing times, help desk activities, trends, status of funds, etc. that helps OSAI track its progress against its annual goals and measures. These reports provide synopsis of the actual activity for the week, the year to date status of the program operations and projected year end results. The information is systematically collected through specialized software called the SAFETY Act Business Process Management tool which is interfaced with the SAFETY Act web site. Program managers use this to make management decisions such as adjusting program priorities and re-allocating resources.

Evidence: Weekly S&T application status reports, TESSA weekly incidence response report, OSAI weekly SA stats report, Web trends for SAFETY Act, monthly financial reports and status of funds.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: The SAFETY Act program is led by a Director and a Deputy Director, both career federal employees, whose employee performance evaluations are tied to meeting the program goals and objectives, including the performance measures discussed in Section 2. OSAI holds weekly status meetings with senior management from DHS and IDA to review program progress, identify potential problems, and coordinate with constituents to ensure attainment of output, outcome and efficiency measures cited above. Renewal of IDA's contract is contingent upon successful performance of their program responsibilities.

Evidence: Weekly S&T application status reports, TESSA weekly incidence response report, weekly SA stats report, Web trends for SAFETY Act, monthly financial reports and status of funds. In FY 07, the OSAI received the Department of Homeland Security Secretary Team Award, and the Science and Technology Directorate's Under Secretary Award for Program Management. In FY 08, OSAI managers received an On the Spot Award for planning and conducting a highly successful SAFETY Act workshop outreach event.

Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose and accurately reported?

Explanation: Federal funds are obligated and awarded in a timely manner and consistently spent only for their intended purpose. At the end of FY 2006, OSAI's obligation rate of FY 2006 funds was 19%. With the implementation of the Final Rule and the new management team in 2007, OSAI was able to obligate and award the remaining balance from FY 2006 and obligate 100% of FY 2007 funds by the end of the fiscal year. As of April 2008, OSAI obligated 95% of its FY 2008 appropriated funds. The awards are promptly entered into the financial management system. The SAFETY Act Implementation program reviews the status of funds and program execution reports on a weekly basis. The information tracked is: status of purchase requisitions (PRs), commitments, obligations and expenditures. The Office of SAFETY Act Implementation (OSAI) collects various weekly progress reports and monthly financial reports from its key partner, the Institute for Defense Analyses (IDA), which assists in carrying out the program's day to day operations. The monthly financial report includes a line item report on program costs and expenses for evaluating applications, IT, and overhead. The report depicts expenses for each period, year-to-date, and projections for expenditures in future months of the fiscal year. This enables OSAI leadership to validate appropriateness of funds spent and stay well within the funding limit allocated for the SAFETY Act program. By monitoring the financial report monthly, OSAI is able to track any significant changes in spending and address the issue well before it can become a problem. In addition, OSAI is able to forecast changes in workload which would require a re-allocation of resources and report these projected changes to our program partners in a timely manner to ensure sufficient obligations in the future.

Evidence: Transition PR tracking reports, Transition Execution reports, Weekly S&T application status reports, weekly SA stats report, Web trends for SAFETY Act, monthly financial reports and status of funds.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The SAFETY Act Implementation program has made significant strides in its efforts to encourage the development and deployment of anti-terrorism technologies, in both the internal and external processing of applications. OSAI recognizes that as the program continues to grow and mature it will gain greater visibility. The number of applicants for SAFETY Act protections continue to increase, straining the personnel and systems resources. OSAI has been proactive in seeking effective ways to deal with this anticipated workload increase. In an effort to continue operational excellence, OSAI is in the process of planning a complete overhaul of the IT system. The new system will provide greater tracking and reporting capability as well as ease of use, both to the applicant as well as OSAI, thus increasing overall process and cost efficiency. The IT improvements also include a transition into the DHS IT staging environment which will provide accredited and certified systems and lower operating cost.

Evidence: Weekly S&T application status reports, weekly SA stats report, Web trends for SAFETY Act, monthly financial reports and status of funds.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Before an applicant's technology is awarded SAFETY Act benefits by the Under Secretary for Science and Technology (S&T), each application must undergo at least three technical evaluations and two economic reviews by subject matter experts along with a legal review by the S&T Office of General Counsel, a review by the S&T Test & Evaluation and Standards Division, and final review by the S&T Director of Transition. All of these reviews must be completed within the 120 day time frame. To be effective in meeting this time line, OSAI co-ordinates and collaborates with DHS components, other Government agencies, and procurement officers to gather evidence as to the efficacy of the technology and to ensure that an awareness exists as to how the application may impact other Departmental programs and policies. OSAI also works with procurement officers to educate them on the SAFETY Act so that they can consider it in the selection requirements for anti-terrorism procurements. Examples include: Transportation Security Laboratory - for their input and test results for private baggage screening technologies and airport screening services; DNDO - for nuclear detection technologies; Department of Energy (DOE) and National laboratories - for assistance in Chem/Bio, Explosive, and Cyber detection technologies.

Evidence: DHS Management Directives, Federal Acquisition Regulations, Procurement Pre-Qualification Request Procedures (on www.safetyact.gov), S&T transmittal review and routing procedures.

Does the program use strong financial management practices?

Explanation: The Office of SAFETY Act Implementation (OSAI) collects various weekly progress reports and monthly financial reports from its key partner, the Institute for Defense Analysis (IDA), which carries out the program's day to day operations. These reports include information to track the status of applications, number of new applications, trends, financial requirements related to conducting reviews of SAFETY Act applications, status of funds, etc. that helps OSAI track its progress against its annual goals and measures. Weekly status meetings are held with senior management to review progress, identify potential problems, co-ordinate with constituents, etc. to ensure time lines and resources are on schedule to meet program goals, objectives and measures. The S&T budget office also keeps track of the status of funds and program execution and reports this information on a weekly basis to senior leadership. The information tracked is: status of purchase requisitions (PRs), commitments, obligations and expenditures. Through careful review of these reports, OSAI is able to identify a change in a spending trend line and address the issue well before it can become a problem. In addition, OSAI is able to forecast changes in workload which would necessitate a re-allocation of resources and discuss these projected changes with program partners in a timely manner to ensure sufficient and accurate obligations in the future.

Evidence: Weekly S&T application status reports, weekly SA stats report, Web trends for SAFETY Act, monthly financial reports and status of funds.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The SAFETY Act program contracted the Homeland Security Institute (HSI) to analyze and evaluate the application processes and identify areas of improvements in FY 2006. March & McLennan, Inc., a leading national insurance broker, was also contracted in FY 2007 to provide an independent evaluation of the insurance liability methodology and processes used to assess the risk associated with anti-terrorism technologies. The program reviewed, analyzed and implemented the recommendations to the maximum extent possible. In FY 2008, Booz Allen Hamilton was contracted to audit the results of implementing the HSI recommendations and to identify those areas that warrant further attention. The results indicated that OSAI has made significant improvements to the program's processes and procedures including the implementation of the feedback from the HSI review. In addition, OSAI has implemented an execution plan employing analytics to drive management decisions, priorities, resource allocations, and actions.

Evidence: Homeland Security Institute's SAFETY Act Program Analysis Report, Nov. 2005; Marsh's Process for Enhancing SAFETY Act Insurance Analysis, Nov. 2006; Booz Allen Hamilton Audit Report, March 2008.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The SAFETY Act program intends to continue to increase the number of applications received, specifically those that can be considered "transition" technologies, as well as ensuring that continued renewal of those QATTs that already have SAFETY Act coverage. The Final Rule, Independent evaluation and analysis of the SAFETY Act program, processes and procedures, and customer feedback identified the program deficiencies that have previously hindered our progress in achieving the long term performance goals. Standard operating procedures have been put in place to address the identified weaknesses and processes are in place to constantly re-evaluate to maintain efficacy of the program and to accomplish the program objectives. During FY 2007, a new management team arrived at OSAI and placed intense emphasis and sustained managerial focus on stabilizing and improving OSAI's processes, procedures and program execution. The resultant significant decrease in process time has allowed for more expeditious reviews of applications, most well within the 120 timeline, as well as the opportunity to reach out to companies to increase the number of applications that were received. During FY 2007, applications increased 63%, processing time was reduced by 31%, and average cost to process an application was reduced by 41% from FY 2006 levels. As a result of a re-tooled and vigorously implemented outreach program, applications from small companies (typically those with the most innovative technologies), increased 225% (from 44 to 101). Anticipating customer demand, a day-long SAFETY Act workshop on how to prepare a successful application, was held in November 2007 in the metro DC area, which attracted a capacity crowd of 250 from across the country, with 70 prospective applicants requesting and receiving individualized pre-application consultations. Recently, OSAI was notified that it was a 2008 nominee for the Homeland Security Medal, one of the Service to America Medals sponsored by the Partnership for Public Service.

Evidence: In FY 07, the OSAI received the Department of Homeland Security Secretary Team Award; the Science and Technology Directorate's Under Secretary award for Program Management. FY 08: On the Spot Award for the SAFETY Act workshop. FY 08: the SAFETY Act program has been nominated for Service to America Medal for Homeland Security.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Office of SAFETY Act program set measures that it is evaluated against and are tracked by DHS leadership. Weekly, monthly, quarterly and annual status reports are reviewed and analyzed, and are an integral to effective program execution. Over the FY 2007 to FY 2012 timeframe OSAI plans to: ?? Increase the outreach events by 5% each fiscal year. In FY 2006, OSAI established the baseline of 30 outreach efforts. In FY 2007, the target was increased to 32 (or 5% increase from the baseline) and the actual was 44 outreach efforts (or 46% increase over the baseline). OSAI also plans to: ?? Increase the number of SAFETY Act Designations and Certifications awarded by 5% each year. In FY 2006, OSAI established the baseline of 66 awards. In FY 2007, the target was increased to 69 (or 5% increase from the baseline) and the actual was 84 awards (or 28% increase over the baseline). OSAI also has an annual goal to maintain the average processing time under 120 days without sacrificing evaluation quality: ?? Average processing time for SAFETY Act application. In FY 2006, OSAI established the baseline of 163 days. In FY 2007, the target was decreased to 120 (or 26.5% decrease from the baseline) and the actual was 113 awards (or 31% decrease from the baseline). Finally, OSAI has a goal to keep the average cost to process an application at a level at least as low as that achieved during FY 2007 (during which OSAI achieved a substantial decrease - 41% -- in average cost over FY 2006).

Evidence: In FY 07, the OSAI received the Department of Homeland Security Secretary Team Award; the Science and Technology Directorate's Under Secretary award for Program Management; in FY 08: On the Spot Award for the SAFETY Act workshop. SAFETY Act program reports

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The number of applicants for SAFETY Act protections continues to increase, pressuring personnel and systems resources. OSAI has been proactive in seeking effective ways to deal with this anticipated workload increase and has achieved early success. Comparing FY 2007 to FY 2006, OSAI reduced application processing time by 31% (from 163 to 113 days), average application processing cost by 41%, even with a 63% increase in applications. Through the first five months in FY 2008, average application processing cost is steady with FY 2007 levels, and the average application processing time is below 110 days. But additional efficiencies over these initial gains will be difficult without larger changes. As such , OSAI is in the process of planning a complete overhaul of the IT system. The new system will provide greater tracking and reporting capability as well as ease of use, both to the applicant as well as OSAI, thus increasing overall process and cost efficiency. The IT improvements also include a transition into the DHS IT staging environment which will provide accredited and certified systems and lower operating cost. The maintenance and operations of the IT systems will fall under the CIO shop where consolidation of efforts should standardize and lower cost.

Evidence: Booz Allen Hamilton Audit Report, March 2008, Weekly S&T application status reports, weekly SA stats report, Web trends for SAFETY Act, monthly financial reports and status of funds.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The SAFETY Act program has demonstrated improved efficiencies and cost effectiveness in achieving program goals each year. The cost (measured by total direct labor dollar expenditures) per application on an annual basis has decreased, applications have increased, time to process has decreased. Although the SAFETY Act program is the only federal program to evaluate and approve effective anti-terrorism technologies for liability protections, similar programs, such as the NIST Advanced Technology Program (ATP), which provides grants for high-risk, high-reward technologies, operate under a timeline of 6 months or more, and are not as actively aggressive in providing outreach to potential applicants. The ATP program promotes the development of new technologies but gives no liability protection incentive to the developer. In addition, the ATP grant program operates on a timeline of business days, unlike the SAFETY Act program which is required to process applications in 120 calendar days.

Evidence: In FY 07, the OSAI received the Department of Homeland Security Secretary Team Award; the Science and Technology Directorate's Under Secretary award for Program Management; FY 08: On the Spot Award for the SAFETY Act workshop

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The SAFETY Act Implementation program has been independently reviewed and evaluated by the Homeland Security Institute, Marsh Insurance group, and Booz Allen Hamilton. Recommendations have been reviewed, analyzed and implemented to the maximum extent possible. No major flaws in the design and execution of the program have been recognized. HSI found that OSAI has implemented a proactive organization capable of effectively managing the application evaluation process and is dedicated to facilitating the applicant's success. Booz Allen Hamilton concluded that OSAI has made significant improvements to its processes, procedures and productions including the implementation of many of HSI's recommendations.

Evidence: HSI's SAFETY Act Program Assessment review, MARSH's Process for Enhancing SAFTY Act Insurance Analysis, BAH's Review and Audit