Detailed Information on the
Science and Technology: Emerging Homeland Security Threat Detection Assessment

Conduct assessment of program and project milestones.

Conduct assessment of program and project milestones.

Conduct assessment of program and project milestones.

Conduct assessment of program and project milestones.

Conduct assessment of program and project milestones.

Collect requirements from customers and make adjustments in current and outyear plans.

Conduct assessment of program and project milestones.

Conduct assessment of program and project milestones.

Develop additional annual and long-term performance metrics and targets that focus on outcomes that accurately measure the program.

Conduct independent assessments of sufficient scope and quality.

Complete a program execution plan for FY 2007-2008.

Realign funding and program based on the realignment of the Science and Technology Directorate.

Develop 5 Year Research and Development Plan.

Conduct assessment of program and project milestones.

Collect requirements from customers and made adjustments in current and out year plans.

Measure: Number of threat assessments completed

Explanation:

Measure: Customer satisfaction with risk assessments to identify potential future threats.

Explanation:The Emerging Threats Portfolio is a research and development program conducted by the Science and Technology Directorate. Program focus is on identifying potential future threats that have not yet manifested themselves but whose potential future appearance is suggested by economic and technology trends, trends in observed terrorist behavior, intelligence and other disparate information.

Is the program purpose clear?

Explanation: The mission of the Emerging Threats Portfolio is to anticipate and define potential terroristic threats arising from new scientific and technological advances, from terrorist use of existing devices in new or unexpected ways, and to effect countermeasures development against these new weapons.

Evidence: Pub. L. 107-296 Homeland Security Act of 2002: ?? SEC. 302: "The [DHS] Secretary, acting through the Under Secretary for Science and Technology, shall have the responsibility for??(2) developing, in consultation with other appropriate executive agencies, a national policy and strategic plan for, identifying priories, goals, objectives and policies for, and coordinating the Federal Government's civilian efforts to identify and develop countermeasures to chemical, biological, radiological, nuclear, and other emerging terrorist threats, including the development of comprehensive, research-based definable goals for such efforts and development of annual measurable objectives and specific targets to accomplish and evaluate the goals for such efforts"; ?? FY06 DHS Congressional Budget Justification

Does the program address a specific and existing problem, interest, or need?

Explanation: The Emerging Threats portfolio addresses the dynamic nature of terrorism threats, as science and technology advancements enable new agents of harm and new ways to employ them.

Evidence: The existence of the problem of emerging threats and the interest and needs to address emerging threats expressed by multiple U.S. Government agencies is illustrated by the following selected examples: o US/Australia bilateral conferences on Emerging Threat Technologies o Technology Surprise Symposium, September 2005 o Science and Technology Surprise Working Group o Scientific and Technical Intelligence Committee o Documentation from these conferences/working groups

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Emerging Threats program is designed so that it is not redundant or duplicative of any other Federal, State, local or private effort. This program is uniquely focused on identifying emerging threats relevant to homeland security whereas other programs develop countermeasures targeted to overseas combatting terrorism tools. Other Federal Agencies, particularly the Department of Defense and Intelligence Community, have similar programs that address emerging and disruptive technologies, and the potential threats they pose. They do not serve the homeland security officer or agent population. The Emerging Threats portfolio manager is an active participant in interagency efforts that specifically address this subject area. This active participation provides access to the information resulting from other agencies' efforts, thereby minimizing the potential for redundant or duplicative efforts. [This has been revised by OMB; please proof]

Evidence: Documentation of DOD/intelligence missions/activities to indicate coordination and lack of duplication.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Emerging Threats program design is free of major flaws that would limit the program's effectiveness or efficiency in meeting it objectives and performance goals in counterterrorism. The program has clear lines of communication with its customers to ensure their needs are met and is able to quickly create countermeasures research or end-users to effect the creation of the countermeasures without obstacles.

Evidence: The program design is intended to use the intellectual capital and capabilities of both the Department of Energy's National Laboratories and the private sector to develop effective capabilities to characterize, assess and counter new and emerging threats, and to exploit technology developments as they arise. The National Labs are the recognized leader in creating countermeasures.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The Emerging Threats Portfolio is designed to conduct efforts that characterize, assess and counter new and emerging threats, and to exploit technology developments as they arise. These activities directly address the program's purpose and would not have occurred in the absence of the program's subsidies.

Evidence:

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The Emerging Threats Portfolio does have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program, they are listed below. The Emerging Threats Portfolio is a research and development program conducted by the Science and Technology Directorate. Program focus is on identifying potential future threats that have not yet manifested themselves but whose potential future appearance is suggested by economic and technology trends, trends in observed terrorist behavior, intelligence and other disparate information. For those projects executed through this Portfolio, a customer satisfaction survey will be conducted starting in FY 2005. This could not have been started earlier because most projects started in FY 2004 and FY 2005 have not yet been completed due to the infancy of this program.

Evidence: The Emerging Threats Portfolio was created in FY 2004 and thus is less than two years old. As a R&D new program it is expected that revisions to the existing performance measures will be made to ensure the effectiveness of this program to meet the desired outcomes. Furthermore, additional measures will be considered and applied as the program matures so that quality "leading indicators" of outcomes are identified. Long-term measures established for the Emerging Threats program are: ??Emerging Threat assessments to identify potential future threats relevant to homeland security ??Development of capabilities to counter emergent threats for which capabilities do not yet exist. ??Conduct high-risk, high-pay off basic technology research

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The Emerging Threats Portfolio does have ambitious targets and time frames for its long-term measures, they are listed below. The Emerging Threats program targets are used to measure program project performance. The program has been in existence less than two years, it is expected that the long-term performance measures will be reviewed and revised based on internal and external program reviews to ensure desired outcomes are most effectively met.

Evidence: Assessment of customer satisfaction and value will be ambitious and ensure that this portfolio is effective in meeting the respective needs of the operational end-users. The targets for capability development to meet emergent threats are stated in percentages because the number of emergent threats identified per annum is not predictable in advance; thus the measure is responsive to the dynamic nature of terrorist threats and current world events. The targets for high-risk, high-pay off basic technology research are ambitious for a portfolio of the current resources, yet this is an essential component to ensure capabilities are being developed to counter emergent threats before they present themselves as a current threat. Long-term measures established for the Emerging Threats program are: ?? Emerging Threat assessments to identify potential future threats relevant to homeland security ?? Development of capabilities to counter emergent threats for which capabilities do not yet exist. ?? Conduct high-risk, high-pay off basic technology research

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The Emerging Threats Portfolio has a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goal, they are listed below. Annual performance measures track the specific products that improve understanding of emerging threats thereby enabling new technologies and actionable information to counter emerging terrorist activities. The annual measures tie into the overall performance goal of develop effective capabilities to characterize, assess and counter new and emerging threats, and to exploit technology development opportunities as they arise. The Emerging Threats Portfolio was created in FY 2004 and thus is less than two years old. As a R&D new program it is expected that revisions to the existing performance measures will be made to ensure the effectiveness of this program to meet the desired outcomes. Furthermore, additional measures will be considered and applied as the program matures so that quality leading indicators of outcomes are identified. The Emerging Threats Portfolio was created in FY 2004 and thus is less than two years old. As a R&D new program it is expected that revisions to the existing performance measures will be made to ensure the effectiveness of this program to meet the desired outcomes. Furthermore, additional measures will be considered and applied as the program matures so that quality leading indicators of outcomes are identified.

Evidence: The customer survey of effectiveness will provide direct assessment of whether desired outcomes are being met. Projects in capability development to meet emergent threats and high-risk, high-pay off basic technology research are monitored and reviewed individually by Federal program managers and the portfolio manager to ensure that each project is performing per contractual agreement and whose results support the Emerging Threats long-term goals. Internal reviews of project performance are conducted at least annually and external review of program performance will be conducted in FY 2006. As an efficiency measure, each project, on average will cost no more than a few million dollars and will be less than two years in duration. Long-term measures established for the Emerging Threats program are: ?? Emerging Threat assessments to identify potential future threats relevant to homeland security ?? Development of capabilities to counter emergent threats for which capabilities do not yet exist. ?? Conduct high-risk, high-pay off basic technology research

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The Emerging Threats Portfolio has baselines and ambitious targets for its annual measures, they are listed below. The Emerging Threats Portfolio was created in FY 2004 and thus is less than two years old. The baseline for this program was established in FY 2004 as conducting assessments to identify potential future threats, initiating projects to counter emergent threats and identification of high-risk, high-pay off basic technology research. As a R&D new program it is expected that revisions to the existing performance measures will be made to ensure the effectiveness of this program to meet the desired outcomes. Furthermore, additional measures will be considered and applied as the program matures so that quality leading indicators of outcomes are identified.

Evidence: The customer survey of effectiveness will provide direct assessment of whether desired outcomes are being met. The targets for capability development to meet emergent threats are stated in percentages because the number of emergent threats identified per annum is not predictable in advance; thus the measure is responsive to the dynamic nature of terrorist threats and current world events. The targets for high-risk, high-pay off basic technology research are ambitious for a portfolio of the current resources, yet this is an essential component to ensure capabilities are being developed to counter emergent threats before they present themselves as a current threat. Long-term measures established for the Emerging Threats program are: ?? Emerging Threat assessments to identify potential future threats relevant to homeland security ?? Development of capabilities to counter emergent threats for which capabilities do not yet exist. ?? Conduct high-risk, high-pay off basic technology research

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All Emerging Threat partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program. In the planning process, partners develop performance measures as it relates to accomplishing Emerging Threat goals. These goals are captured in the individual project plans, and are measured on a periodic basis per the project plan and the total efforts are reviewed both at the execution organization and the Portfolio Level. Each project in the Emerging Threats Portfolio explicitly includes project goals and expected outcomes that are described in the project Statement of Work. Although individual project goals may not explicitly state how they support the annual and/or long-term goals of the program, each project, its goals, and deliverables can be directly linked with these annual or long-term goals at the portfolio/program level.

Evidence: Emerging Threats projects are assigned to professional Federal Program Managers, who have responsibility for project technical and financial oversight. Emerging Threats projects are assigned to professional Federal Program Managers, who have responsibility for project technical and financial oversight. In addition, most of the projects have iterative development or intermediate deliverables. This gives the Program Manager additional insight into the progress of the project and its applicability to user need and support to annual and long-term program goals. If needed, the Program Manager is authorized to terminate projects not meeting the goals of the program. Partners and contractors are held accountable through accepted government contracting requirements. Included in these requirements is the requirement that authorization to pay invoices must be made by the technical contract managers for each project.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Independent evaluations of sufficient scope and quality are conducted as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need. The Emerging Threats portfolio is less than two years old and many projects are just beginning to demonstrate results that can be evaluated. Consequently, independent evaluations (independent of a DHS entity) of sufficient scope and quality have not yet been started because they would not yet be able to measure outputs or outcomes of the programs. Regular independent reviews are scheduled to begin in FY 2006 and will be focused on supporting program improvements as well as evaluating effectiveness and relevance to counter emerging terrorism threats. Additionally, this will ensure the efforts are not duplicative with other research within the Department or with other federal agencies.

Evidence: The Emerging Threats program is less than two years old and many of the program efforts in initiated in FY 2004 continued into FY 2005. Program evaluations are currently conducted on individual programs by the executing organizations and the portfolio manager. In addition, some programs have established independent reviews. Three principal options exist for conducting independent evaluations beginning in FY 2006: 1.) The Homeland Security Institute, an FFRDC, could be requested and funded to conduct such an evaluation - a determination of whether the Homeland Security Institute is sufficiently independent of DHS is required.; 2.) A request to the Undersecretary of S&T to task the Homeland Security Science and Technology Advisory Committee to conduct an independent evaluation can be made.; 3.) An independent organization could be contracted to perform an independent evaluation. The specific choice will be made in the first quarter of FY 2006. The S&T Directorate's RDT&E process requires that each portfolio conduct and document a risk assessment to help guide the planning. These risk assessments are independently reviewed by the S&T Program Analysis and Evaluation (PA&E) staff and the Office of Comparative Studies (OCS). In addition, the Comparative Studies office is responsible for conducting independent risk assessments. Although these responsibilities are through contacting mechanisms to others outside of DHS, within the S&T Directorate they provided an independent evaluation of all programs which is reviewed by the S&T Senior Management, and who subsequently provide management guidance to each program. Independent input on the threats, risks, vulnerabilities etc is also provided by the Information Analysis component of DHS.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Emerging Threat's budget request is explicitly tied to accomplishment of the annual and long-term performance goals. The resource needs for each aspect of the program are presented in a complete and transparent manner in the program's budget.

Evidence: ?? The Emerging Threats program develops its program budgeting using annual performance targets and budget resources as presented in the DHS Five Year Homeland Security Program (FYHSP). The FYHSP captures and expresses the impact of funding, policy, or legislative decisions on the Emerging Threats Program. ?? The Emerging Threats program reports all direct and indirect costs which are captured in the DHS Federal Financial Management System (FFMS). This system reports costs that are used to assess overall Emerging Threats performance results. ?? The Emerging Threats budget for FY05 and FY06 supports the goal of developing effective capabilities to characterize, assess and counter new and emerging threats, and to exploit technology development opportunities as they arise.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: No strategic planning deficiencies have been identified either within the S&T Directorate.

Evidence: The Emerging Threats program provides direct input into the Science and Technology Directorate's research, development, test and evaluation (RDT&E) process; in which the strategic planning for the S&T Directorate is incorporated. The Emerging Threats program is working to adopt a limited number of specific, ambitious long-term performance goals that demonstrate progress toward achieving the longer term goals, thereby ensuring that its strategic planning is of value to the total RDT&E process.

If applicable, does the program assess and compare the potential benefits of efforts within the program and (if relevant) to other efforts in other programs that have similar goals?

Explanation: The Emerging Threats program is designed to focus on new and emerging threats that are specific to the DHS mission. There are no other programs within DHS or other government agencies that address the scope or focus of the Emerging Threats program. Within the Emerging Threats program, individual projects are specifically selected and executed to meet the mission needs of DHS. The Emerging Threats program is less than two years old and independent evaluations of its efforts will be initiated in FY 2006. Evaluations prior to this point would have been of marginal value since there are not yet significant project outputs or outcomes that could be reviewed. In both FY 2004 and FY 2005, the S&T Directorate has been responsible for DHS's self-assessment of compliance with the R&D Investment Criteria. The self-assessment is then independently reviewed by the Office of Management and Budget (OMB); the Emerging Threats program is a contributor to the overall self-assessment.

Evidence: President's Management Agenda: R&D Investment Criteria ?? Participation in inter-agency forums on this subject permits alternative approaches to be identified and evaluated for their relevance to DHS. ?? Selected Examples of interagency collaborations: o US/Australia bilateral conferences on Emerging Threat Technologies o Technology Surprise Symposium, September 2005 o Science and Technology Surprise Working Group o Scientific and Technical Intelligence Committee Documentation from these conferences and working groups can not be included in this document due to the classification level of the material.

Does the program use a prioritization process to guide budget requests and funding decisions?

Explanation: The Emerging Threats program prioritizes its budget requests and funding decisions based on input from beneficiaries, existing threat assessments and need to provide input into the S&T Directorate's RDT&E process. [Need to identify, describe, and show evidence of a documented process to identify priorities and how the resulting priorities are used in decision-making.]

Evidence: The Emerging Threats portfolio is currently conducting studies and analyses of emerging threats in order to prioritize and jump-start countermeasures capability. Additionally, in support of the prioritization process, this program interacts with independent science and technology advisory boards and inter-agency working groups such as The Scientific and Technical Intelligence Committee and the Science and Technology Surprise Working Group to obtain independent, national-level assessments used in prioritization of the Emerging Threats program's efforts. [Documentation?]

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The program regularly collects cost, schedule, and performance data from the key program partners. This information is used to inform current and future year program planning and prioritization of sub-projects and associated resources. The performance of the program partners is evaluated in accordance with the execution organizations' requirements to ensure that the program purpose and scope are being met in a timely manner.

Evidence: Emerging Threats Project Plans ?? DHS S&T Directorate FYHSP ?? ORD Emerging Threats Review Documents For the Homeland Security Advanced Research Projects Agency (HSARPA) programs, including the Emerging Threats program, weekly reports are provided to the HSARPA Deputy Director detailing: 1) The date of program origination; 2) the date of funding obligation; 3) the amount of the funding that left DHS; 4) the amount of funding obligated (if available); and 5) the amount of funding expended (if available). Furthermore, each program under Emerging Threats is assigned an HSARPA program manager to closely monitor the programs' progress.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Each project awarded has an agreed-upon set of cost, schedule and performance results which are clearly conveyed in the contract or reimbursable agreements (for other federal agencies). The Federal program managers and program execution partners are held to these criteria when evaluating their performance.

Evidence: The Emerging Threats portfolio manager and program partners are held accountable for cost, schedule, and performance results. The program has identified program managers at the headquarters and within the executing organizations who are responsible for achieving key program results and performance measures. These performance measures are documented in each project award document. Example project award documents are provided. ?? Homeland Security Bioinformatics and Assay Development Program (BIAD) Broad Agency Announcement 04-03 (BAA 04-03), Homeland Security Advanced Research Projects Agency (HSARPA), April 16, 2004 Section 4.1.2 TAA-2: Assays for new, emerging, or engineered ?? Office of Research and Development: Statement of Work

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Emerging Threats program funds are obligated per S&T Directorate allocation (quarterly in FY 2005) and execution organization procurement process once an emerging threat is identified and a commensurate RDT&E program or project is established. Individual projects are required to report expenditures on an agreed upon schedule (normally monthly) and invoices are reviewed by both program managers and resource managers to ensure payments are proper for work completed prior to authorization of payment.

Evidence: The Emerging Threats program funds are obligated in accordance with overall program plan and obligations and expenditures are tracked as part of the DHS Federal Financial Management System (FFMS). FY 2004 funding was more than 79% obligated at the beginning of FY 2005 and FY 2005 funding is projected to be 60% obligated by the end of FY 2005.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation:  

Evidence: As an efficiency measure, each project, on average will cost no more than a few million dollars and will be less than two years in duration (in order to be responsive to the dynamic nature of terrorist threats and current world events). An additional efficiency measure for the Emerging Threats portfolio is: percentage of responding recipients indicating the annual emerging threat assessment reports are valuable, thereby guiding RDT&E investments into those areas most relevant to homeland security. The Emerging Threats program is less than two years old and efficiency measures are expected to be revised as the program matures to ensure cost effectiveness. The Emerging Threats program develops its program budgeting using annual performance targets and budget resources as presented in the DHS Five Year Homeland Security Program (FYHSP). The FYHSP captures and expresses the impact of funding, policy, or legislative decisions on the Emerging Threats Program. Emerging Threat projects are awarded on both competitive source selection and a merit review process with proposals required to identify type and magnitude of costs to competitively source projects, drive efficiencies and achieve cost effectiveness. The program uses existing contracting officers and thus reduces the cost of program execution relative to what would be required if dedicated contract officers were used. Existing Program managers are also used in order to improve efficiency and reduce costs. Efficiency is being gained by cooperative efforts with other Federal agencies that reduce the costs of studies required to be funded solely by the Emerging Threats program. Additionally, cost savings in some individual programs/projects are being used to accelerate the efforts to fully address the scope of the Emerging Threats program, i.e. efforts will be initiated/conducted in FY 2005 that would have been scheduled for FY 2006.

Does the program collaborate and coordinate effectively with related programs?

Explanation: Relevant R&D is underway at other agencies and organizations; thus partnerships in this area hold great potential for synergistic focus on Homeland Security. Some selected collaborations and coordinating activities are identified in the response to question 1.2. In addition, to date, the Emerging Threats program has collaborated and co-funded (minority contribution) two projects with another government agency thereby leveraging this program's resources with those of others.

Evidence: This program collaborates regularly with related federal programs including the IC, DOE, DOJ, and DoD Director of Defense Research and Engineering to develop multi-year planning and leverage the investments of others. Documentation of this collaboration is primarily in meeting records and can not be included in this document due to the classification level of the material.

Does the program use strong financial management practices?

Explanation: Emerging Threat projects are monitored by program managers who routinely receive project expenditure information; monthly financial reports are provided at the portfolio level for information and review. Each project within the Emerging Threats program must routinely report its progress and expenditures. Prior to authorization to pay project invoices, these expenditures are reviewed by both the DHS technical program manager and a resource manager for compliance with project scope and progress. Financial systems are in place providing regular financial status reports - these meet the statutory requirements and are currently assessed as one of the President's Management Agenda areas.

Evidence: The DHS IG is initiating an audit of DHS's FY05 consolidated financial statements, to be performed by an independent accounting firm. See attached memorandum dated Feb 9, 2005 for Adm. Loy from R.L. Skinner. Also see the answer to question 3.1 above.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: This program has taken significant and meaningful steps to correct program management deficiencies. The Emerging Threats program is a new program less than two years old. When it began, there were no formal management systems in place. To date, management systems have been implemented in the S&T Directorate that track program performance, cost, schedule, and milestones and hold regular reviews of execution status. All Emerging Threats projects are now using these program management systems so that any program management deficiencies are identified and corrected in a timely manor.

Evidence: Each Emerging Threats Program/Project is assigned a government program manager who will maintain ownership of, and carefully monitor, program/project progress. Performance Reviews are conducted at scheduled performance checkpoints and/or annually; efforts not meeting expectations are identified and necessary corrections made. Project Statements of Work (see example in response to question 2.5) identify required deliverables and these are tracked using existing management systems.

For R&D programs other than competitive grants programs, does the program allocate funds and use management processes that maintain program quality?

Explanation: The Emerging Threat program allocates funds through a mix of competitive and non-competitive (e.g., to national and Federal laboratories, FFRDC's and other governmental agencies) processes. In FY 2005, $4 million will be awarded by competitive solicitation and the balance through merit review to the uniquely capable national laboratories, the Homeland Security Institute (a FFRDC) and cooperative agreements with other Federal agencies.

Evidence: Evidence of both competitive, merit-review based and cooperative agreements with other Federal agencies is demonstrated in project award procurements. ?? Homeland Security Bioinformatics and Assay Development Program (BIAD) Broad Agency Announcement 04-03 (BAA 04-03), Homeland Security Advanced Research Projects Agency (HSARPA), April 16, 2004 Section 4.1.2 TAA-2: Assays for new, emerging, or engineered ?? Office of Research and Development approved Program Execution Plan (PEP) for Emerging Threats Merit-review awards are based on the unique capabilities resident in the national and Federal Laboratories, FFRDC's, and other government agencies. The process is largely conducted under the guidance of the Office of Research and Development within the S&T Directorate. Program planning sessions with the potential National Laboratory performers are held by the appropriate program manager(s) to ensure the most capable performer to complete the projects being funded.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Emerging Threats Portfolio has demonstrated some progress in achieving its long-term performance goals. The Emerging Threats program has a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals. Annual performance measures track the specific products that improve understanding of emerging threats thereby enabling new technologies and actionable information to counter emerging terrorist activities. The annual measures tie into the overall performance goal of development of effective capabilities to characterize, assess and counter new and emerging threats, and to exploit technology development opportunities as they arise. The Emerging Threats Portfolio was created in FY 2004 and thus is less than two years old. As a R&D new program it is expected that revisions to the existing performance measures will be made to ensure the effectiveness of this program to meet the desired outcomes. Furthermore, additional measures will be considered and applied as the program matures so that quality "leading indicators" of outcomes are identified.

Evidence: In FY 2004 and FY 2005, emerging threat assessments have been conducted or are in progress for biological and explosive threats, compounded infrastructure threat (evaluate ways in which infrastructure could be used as a weapon), and transit systems. Capability development projects to address specific identified emergent threats for which no capabilities existed to counter those emergent are in progress and can not be discussed in detail due to the classification level of the material. One high-risk, high-pay off basic technology modeling capability for sensor development for emergent biological and chemical threats has been identified and will be initiated. The customer survey of effectiveness will provide direct assessment of whether desired outcomes are being met. Projects in capability development to meet emergent threats and high-risk, high-pay off basic technology research are monitored and reviewed individually by Federal program managers and the portfolio manager to ensure that each project is performing per contractual agreement and whose results support the Emerging Threats long-term goals. The program has been in existence less than two years, it is expected that the long-term performance measures will be reviewed and revised based on internal and external program reviews to ensure desired outcomes are most effectively met. As an efficiency measure, each project, on average will cost no more than a few million dollars and will be less than two years in duration. Long-term measures established for the Emerging Threats program are: ?? Emerging Threat assessments to identify potential future threats relevant to homeland security ?? Development of capabilities to counter emergent threats for which capabilities do not yet exist. ?? Conduct high-risk, high-pay off basic technology research

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Emerging Threats Portfolio (including program partners) achieves its annual performance goals. All Emerging Threat partners (including program partners) commit to and work toward the annual goals of the program. In the planning process, partners develop performance measures as it relates to accomplishing Emerging Threat goals. These goals are captured in the individual project plans, and are measured on a periodic basis per the project plan and the total efforts are reviewed both at the execution organization and the Portfolio Level. Each project in the Emerging Threats Portfolio explicitly includes project goals and expected outcomes that are described in the project Statement of Work. Although individual project goals may not explicitly state how they support the annual and/or long-term goals of the program, each project, its goals, and deliverables can be directly linked with these annual and/or long-term goals at the portfolio/program level.

Evidence: Program and project goals have been met and are on track. Emerging Threats projects are assigned to professional Federal Program Managers, who have responsibility for project technical and financial oversight. In addition, most of the projects have iterative development or intermediate deliverables. This gives the Program Manager additional insight into the progress of the project and its applicability to user need and support to annual and long-term program goals. If needed, the Program Manager is authorized to terminate projects not meeting the goals of the program. Partners and contractors are held accountable through accepted government contracting requirements. Included in these requirements is the requirement that authorization to pay invoices must be made by the technical contract managers for each project. Long-term measures established for the Emerging Threats program are: ?? Emerging Threat assessments to identify potential future threats relevant to homeland security ?? Development of capabilities to counter emergent threats for which capabilities do not yet exist. ?? Conduct high-risk, high-pay off basic technology research

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Emerging Threats portfolio demonstrates improved efficiencies or cost effectiveness in achieving program goals each year. The Emerging Threats program is less than two years old. However, cost effectiveness is being achieved by collaborative efforts with other Federal agencies that reduces the resource requirements from the Emerging Threats program to achieve the desired programmatic objective as well as putting consistent program management processes in place.

Evidence: Specifically, cost effectiveness is being achieved by collaboration with specific members of the Intelligence Community (IC); net assessment information sponsored by the Department of Defense (DoD) is being used, as appropriate, to address emerging threats relative to homeland security. [Specific details can not be disclosed in this document due to the classification level of the material.] The program uses existing contracting officers and thus reduces the cost of program execution relative to what would be required if dedicated contract officers were used. Existing Program managers are also used in order to improve efficiency and reduce costs. Efficiency is being gained by cooperative efforts with other Federal agencies that reduce the costs of studies required to be funded solely by the Emerging Threats program. Additionally, cost savings in some individual programs/projects are being used to accelerate the efforts to fully address the scope of the Emerging Threats program, i.e. efforts will be initiated/conducted in FY 2005 that would have been scheduled for FY 2006. In addition, program performance deficiencies are identified and corrected by monitoring individual project performance and customer feedback. Long-term measures established for the Emerging Threats program are: ?? Emerging Threat assessments to identify potential future threats relevant to homeland security ?? Development of capabilities to counter emergent threats for which capabilities do not yet exist. ?? Conduct high-risk, high-pay off basic technology research

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The performance of the Emerging Threats portfolio is unique to Homeland Security and can not be favorably compared to other programs. The appropriate answer for this Appeal is Not Applicable.

Evidence: The Emerging Threats program is a new program less than two years old. It is specifically structured to address new and emerging threats relative to the mission and strategic objectives of DHS. Programs in other agencies such as the Department of Defense (DoD) and the Intelligence Community (IC) that address emerging and disruptive technologies are not focused on the needs of homeland security. The Emerging Threats program is therefore unique to homeland security. Programs in the IC that address emerging and disruptive technologies have resources far exceeding those of the Emerging Threats program. However, individual projects have similar cost and project duration objectives as those in the Emerging Threats portfolio. [Specific details can not be disclosed in this document due to the classification level of the material.]

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The Emerging Threats portfolio is less than two years old and many projects are just beginning to demonstrate results that can be evaluated. Consequently, independent evaluations (independent of a DHS entity) of sufficient scope and quality have not yet been started because they would not yet be able to measure outputs or outcomes of the programs. Regular independent reviews are scheduled to begin in FY 2006 and will be focused on supporting program improvements as well as evaluating effectiveness and relevance to counter emerging terrorism threats. Additionally, this will ensure the efforts are not duplicative with other research within the Department or with other federal agencies.

Evidence: The Emerging Threats program is less than two years old and many of the program efforts in initiated in FY 2004 continued into FY 2005. Program evaluations are currently conducted on individual programs by the executing organizations and the portfolio manager. In addition, some programs have established independent reviews. Three principal options exist for conducting independent evaluations beginning in FY 2006: 1.) The Homeland Security Institute, an FFRDC, could be requested and funded to conduct such an evaluation - a determination of whether the Homeland Security Institute is sufficiently independent of DHS is required.; 2.) A request to the Undersecretary of S&T to task the Homeland Security Science and Technology Advisory Committee to conduct an independent evaluation can be made.; 3.) An independent organization could be contracted to perform an independent evaluation. The specific choice will be made in the first quarter of FY 2006. The S&T Directorate's RDT&E process requires that each portfolio conduct and document a risk assessment to help guide the planning. These risk assessments are independently reviewed by the S&T Program Analysis and Evaluation (PA&E) staff and the Office of Comparative Studies (OCS). In addition, the Comparative Studies office is responsible for conducting independent risk assessments. Although these responsibilities are through contacting mechanisms to others outside of DHS, within the S&T Directorate they provided an independent evaluation of all programs which is reviewed by the S&T Senior Management, and who subsequently provide management guidance to each program. Independent input on the threats, risks, vulnerabilities etc is also provided by the Information Analysis component of DHS.