Detailed Information on the
Federal Emergency Management Agency: Grants and Training Office National Exercise Program Assessment

Enhance regional exercise structure through multijurisdictional exercises involving FEMA Regions.

Refine national-level exercise after action review process and improvement plan process to include the implementation of the Top Officials (TOPOFF) 4 improvement plan.

Define scope of National Exercise Program needed to support other SLGCP performance measures.

SLGCP budget links NEP program funding to goals.

Submit the National Exercise Plan for interagency review

Increase state and local compliance with HSEEP doctrine and reporting requirements.

Implement capabilities-based quarterly analysis of submitted final exercise after-action reports (AARs) to determine HSEEP compliance.

Measure: Number of DHS funded exercise per year

Explanation:This measure reflects the volume of exercises conducted using direct-support or preparedness grant funds.

Measure: Percent of DHS funded exercises demonstrating use of Homeland Security Exercise and Evaluation Program (HSEEP)

Explanation:The number of after-action reports with analysis of HSEEP critical tasks is divided by the total number of after-action reports from DHS funded exercises, including both those funded through grants and direct support.

Measure: Percent of jurisdictions demonstrating acceptable performance on applicable critical tasks in exercises using SLGCP approved scenarios

Explanation:This measure evaluates jurisdictions' performance on Homeland Security Exercise and Evaluation Program (HSEEP) critical tasks in homeland security exercises. Measuring improvements in jurisdictions' performance on critical tasks over time reflects the impact of SLGCP preparedness activities on jurisdictions' overall preparedness levels. To measure preparedness levels, critical task analyses included in exercise after-action reports (AARs) are evaluated using HSEEP Exercise Evaluation Guides (EEGs) to determine whether the jurisdiction's performance met expectations or required improvement. Jurisdictions' performance on each critical task is analyzed by comparing the results documented in the AAR to the expected outcome described in the EEG.

Measure: Percent of National Preparedness Priority capabilities that were analyzed in DHS funded exercises

Explanation:This measure determines the percent of DHS priority capabilities, as defined in the National Preparedness Goal, were analyzed during exercises for the period of performance.

Measure: Average satisfaction rating by exercise participants

Explanation:Exercise participants are given a survey after each direct support exercise, rating different elements of the exercise on 1-5 scale (5 being the best). The average feedback score for each exercise is calculated and then averaged for all exercises. A rating of 4 or 5 is considered "satisfied."

Measure: Percentage of action items identified in After-Action Reports that were implemented

Explanation:This measure is designed to assess the number of improvement plan action items that jurisdictions implement/execute following SGCLP-funded or supported exercise. Determining the percent of action items that are implemented reflects the impact of the National Exercise Program on jurisdictions' ability to identify and resolve issues and/or preparedness gaps. Data is collected from exercise after-action reports (AARs) that include improvement plans and from participating jurisdictions' responses to an online survey on action item implementation.

Measure: Average unit cost of regional support exercises

Explanation:This measure evaluates the cost of two categories of direct support exercises, discussion-based and operations-based exercises. The purpose is to ensure the the cost to perform exercises does not increase over time and that the HSEEP continues to support self-sustaining programs. This measure was developed to provide data on the NEP Direct Support Exercise Program, which provided exercise development, implementation, and evaluation support to State and local jurisdictions nationwide. This program has now become the NEP Regional Exercise Support Program (RESP), providing support to homeland security exercises at the regional level, across multiple States, urban areas, and local jurisdictions. As such, the language of the measure must be changed, and the targets have been updated to reflect the new expected cost of the program. Nevertheless, the new targets are within a few thousand dollars of the old measure targets. These targets are at the lowest end of the expected cost spectrum for the program, and are therefore ambitious given the program restrictions.

Is the program purpose clear?

Explanation: The purpose of the National Exercise Program (NEP) is to prepare the Nation's first responders to prevent, respond to, and recover from acts of terrorism by providing homeland security professionals with the tools to plan, conduct, and evaluate exercises. The NEP provides opportunities for officials and emergency responders to practice and test capabilities, policies, plans, and procedures. Through the after-action report and improvement plan processes, the NEP highlights potential preparedness shortfalls. Analysis of improvement plans and their corrective actions inform future preparedness priorities by highlighting common problems, untested plans and equipment, and exercise design requirements. These priorities then become the basis for future funding, training, equipment, and technical assistance, which become the basis for future exercises.

Evidence: "■ HSPD-8, National Preparedness Goal, Section 18, Page 5 ■ HSPD-8 Implementation Concept Memo-April 16, 2004 ■ HSEEP Volume I, Preface, Page iii ■ Homeland Security Act of 2002, Section 430 (c) (1)"

Does the program address a specific and existing problem, interest, or need?

Explanation: "The National Exercise Program addresses the need for performance assessment of national preparedness; validation of preparedness improvements; and building relationships among Federal, State, and local homeland security officials and emergency responders In support of these needs, the NEP also develops standardized methodologies for the administration of homeland security exercises. After-action reports from the Oklahoma City bombing and from the September 11, 2001 attacks discuss the need for homeland security officials and emergency responders to establish relationships and response protocols prior to an actual incident. The NEP serves this need by providing opportunities for government leaders and emergency responders to interact and operate together in scenarios that simulate real terrorist attacks.

Evidence: "???? Homeland Security Act of 2002, Section 430 (c) (1) ???? National Strategy for Homeland Security, Mission Statement ???? National Strategy for Homeland Security, Strategic Goal 2.4 ???? National Strategy for Homeland Security, Objective 3.7 ???? National Preparedness Goal ???? U.S. Department of Homeland Security Strategic Plan ???? HSEEP Volume I, Introduction

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: Yes, the National Exercise Program (NEP) is the lead Federal program for providing doctrine and support for State and local terrorism preparedness exercises. It has been designated by the Secretary of DHS as the department's exercise coordinator. In 2003, the Homeland Security Council designated the NEP as the principal program for exercising national homeland security roles and responsibilities. The following attributes of the NEP are unique among other Federal, State, local, and private efforts: The Exercise Division developed and implemented the Homeland Security Exercise and Evaluation Program (HSEEP), the first common homeland security exercise doctrine and policy. To avert duplication,the Exercise Division administers the National Exercise Schedule (NEXS), which compiles, publishes, and disseminates exercise schedules from all Federal agencies and all 56 States and Territories. SLGCP's Exercise Division focuses on the full range of State and local emergency responders (fire, law enforcement, emergency medical services, etc.) and terrorism scenarios. Other exercise programs focus on specific disciplines (e.g. CDC exercises focus on the public health community) or on specific emergency scenarios (e.g. FEMA sponsors exercises focused on chemical stockpile and nuclear plant emergencies).

Evidence: Homeland Security Act of 2002, Section 430 (c) (1) Homeland Security Council, Deputies Committee Meeting Minutes, November 13, 2003 Homeland Security Grant Program-State Homeland Security Program Homeland Security Grant Program-Urban Area Security Initiative (UASI) HSEEP Volumes I-IV National Exercise Schedule"

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The National Exercise Program is based on the DOJ Nunn-Lugar-Domenici (NLD) Domestic Preparedness Program. The measurable effectiveness of NLD exercises was limited by several factors: limited in scope to chemical and biological exercises in the 120 largest cities; lack of consistent, objective, and measurable outcome standards; and highly inconsistent after-action report data. To correct the identified deficiencies in the NLD program, the NEP developed and implemented the Homeland Security Exercise and Evaluation Program (HSEEP to establish clear outcome standards and formalize reporting processes for after-action reports and improvement plans. As FY2004 was the first year of HSEEP availability, implementation by Federal, state, and local agencies remains inconsistent.

Evidence: National Exercise Schedule Lessons Learned Information Sharing (llis.gov) ODP Secure Portal (odp.esportals.com)"

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: "The National Exercise Program (NEP) is effectively targeted so that resources will reach intended beneficiaries and directly contribute to the program's purpose: to prepare the Nation's first responders to prevent, respond to, and recover from acts of terrorism. To ensure that NEP resources distributed to states address national priorities and reach the largest number of beneficiaries possible, the NEP contributes to SLGCP's State Homeland Security Strategy process. This process requires each State and Territory that receives Homeland Security Grant Program funding from DHS/SLGCP to develop and submit a State Homeland Security Strategy, based on a statewide assessment of planning, training, equipment, and exercise needs, as well as evaluations of risk and threat. This strategy then becomes the basis for determining the resources needed to support the number and type of exercises each State requests. As part of this process, NEP requires States to conduct annual Exercise Plan Workshops (EPWs) to review the State Strategy and develop a Multi-year Exercise Plan and Schedule. NEP Exercise Managers review these Exercise Plans and Schedules to ensure that states have appropriately targeted exercise resources towards high-risk, high-threat jurisdictions, and exercise scenarios are reflective of identified vulnerabilities. For example, when reviewing the State of New York's Exercise Plan and Schedule, Exercise Managers would expect to see a large portion of the state's exercise resources focused on increasing preparedness in New York City, and the notional scenarios focused on vulnerabilities such as critical infrastructure, tourism, etc. In addition to those exercises that States fund directly out of their grant funding (i.e., Homeland Security Grant Program or Urban Area Security Initiative), States and territories can also request direct exercise support (i.e., technical assistance) from ODP for exercise design, development, conduct and evaluation. In order to receive this direct exercise support, the State Administrative Agency (SAA) must submit an application requesting the support. These requests must also be aligned with the State's Strategy and Multi-year Exercise Plan. Each application is then reviewed by a panel of SLGCP Exercise Managers using criteria weighted toward the following targeted areas: 1. Exercise Plan Workshops to review and produce a Multi-year Exercise Plan and Schedule 2. Exercises tied back to the State's Strategy and Multi-year Exercise Plan, which identify targeted areas based on risk/vulnerability 3. Exercises to meet HSPD-8 Implementation requirements. In FY '05 every UASI is required to conducted a full-scale exercise using the IED scenario contained within the Homeland Security Council's suite of 15 scenarios. 4. Exercises in preparation for a National Special Security Event or Special Event Level 1-4 (designated by DHS I-Staff) "

Evidence: HSEEP Volume II, After Action Report/Improvement Plan template Direct Support Application and Users Handbook Homeland Security Grant Program Exercise Plan Workshop Packet Target Capabilities List, Version 1.1

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The NEP tracks two primary long-term outcome measures over time to monitor its progress toward meeting its goals: 1) Percentage of jurisdictions demonstrating acceptable performance on applicable critical tasks in exercises using SLGCP-approved scenarios 2) Percentage of action items identified in after-action reports that were implemented. Such action items reflect concrete, actionable improvement measures proposed by jurisdictions in post-exercise improvement plans. Both performance measures capture data directly relevant to the NEP's goals. Performance measure 1 captures Federal, State, and local success in performing critical terrorism response tasks, as defined by HSEEP. This measure is shared by other SLGCP programs. Performance measure 2 tracks the percentage of action items identified in Program-sponsored after-action reports that were subsequently implemented by jurisdictions. Tracking action item implementation provides a quantitative measure of the program's success in identifying and remedying specific gaps in the Nation's emergency response capabilities. As with other outcome measures, NEP is not the sole determinant of peformance. Performance measure 1 is affected by a broad range of Federal, State, and local initiatives affecting the capability of a jurisdiction's emergency responders. Similarly, performance measure 2 is affected by programs and actors outside of the NEP's authority. While the NEP can help a jurisdiction identify action items and require documentation of improvement plans, the actual implementation of an action item rests solely with the jurisdiction.

Evidence: FYHSP Congressional Justification FY '06

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Ambitious long-term performance goals have been set based on the expectation that broader implementation of the HSEEP will improve performance on critical tasks and action item implementation. The long-term performance goals incorporate ambitious, compounding rates of improvement. 1) Percent of jurisdictions demonstrating acceptable performance on applicable critical tasks in exercises using SLGCP approved scenarios. Seven-Year Goal: 90% of jurisdictions demonstrating acceptable performance on applicable critical tasks. 2) Percent of action items identified in after-action reports (AARs) that were implemented. Seven-year goal: 87% of action items implemented on schedule.

Evidence: Congressional Justification FY '06 FYHSP U.S. Department of Homeland Security Strategic Plan FY04 AARs in Outcome Measure 1 Data Set" "Outcome Measure 2: Methodology and Data Set"" Source: Analysis of AARs in the Lessons Learned Information Sharing (LLIS.gov) system. Source: Survey on action item implementation distributed to jurisdictions participating in NEP Exercises

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The National Exercise Program (NEP) uses seven annual measures to track its progress in achieving its long-term goals. These measures are linked to the long-term goals, and demonstrate progress toward achieving preparedness on the Federal, State and local levels. The NEP's annual measures are: 1) Percentage of jurisdictions demonstrating acceptable performance on applicable critical tasks in exercises using SLGCP approved scenarios 2) Percentage of action items identified in after-action reports that were implemented 3) Average satisfaction rating by exercise participants 4) Percentage of SLGCP-supported exercises demonstrating use of HSEEP to design, develop, and evaluate their exercises. 5) Number of SLGCP grant-funded exercise projects per year. 6) Percentage of critical tasks exercised that were conclusively assesed by the exercise.

Evidence: FYHSP U.S. Department of Homeland Security Strategic Plan Congressional Justification FY '06

Does the program have baselines and ambitious targets for its annual measures?

Explanation: By using clearly established baseline information, the National Exercise Program (NEP) has developed ambitious, yet achievable, targets for the program's seven annual measures. The ambitious targets are generally set at levels that ensure continued progress toward achieving preparedness at the Federal, State, and local levels. The measure for "Percentage of Critical Tasks Conclusively Analyzed" should be updated to focus on whether exercises are achieving their stated aims, not just the quality of after action reports. HSEEP compliance is expected to continue to grow and jurisdictions will be realigning their preparedness priorities based on the ongoing implementation of capabilities-based planning. Introduction of the Improvement Plan Tracking System and HSEEP Toolkit in 2006 will better ensure compliance with HSEEP standards and provide more consistent and accurate data to measure program effectiveness in future years.

Evidence: 1) Outcome Measure 1: Percentage of jurisdictions demonstrating acceptable performance on applicable critical tasks in exercises using SLGCP approved scenarios FY 2004 Baseline: 20% FY 2005 Target: 23% FY 2006 Target: 60% 2) Outcome Measure 2: Percentage of action items identified in After-Action Reports that were implemented on schedule* FY 2004 Baseline: 13% FY 2005 Target: 41% FY 2006 Target: 65% *Note: This data reflects only the items for which completion was required at the time of the report. Many of the action items have future deadlines that this report does not capture. The sample for this baseline will increase once jurisdictions have more time to implement these action items. 3) Average Satisfaction Rating by SLGCP Exercise Participants FY 2004 Baseline: 4.1 FY 2005 Target: 4.1 FY 2006 Target: 4.2 4) Percent of SLGCP-Supported Exercises Demonstrating Use of HSEEP to Design, Develop, and Evaluate Their Exercises FY 2004 Baseline: 30% FY 2005 Target: 70% FY 2006 Target: 90% 5) Number of SLGCP grant-funded exercise projects per year (Source: State Assessment data): FY 2004 Baseline: 525 FY 2005 Target: 540 FY 2006 Target: 565 6) Average unit cost of direct support exercises FY 2004 Baseline: Table Top Exercise Average Cost: $88,011 Full-Scale Exercise Average Cost: $153,490 FY 2005 Target: Table Top Exercise Average Cost: $80,000 Full-Scale Exercise Average Cost: $120,000 FY 2006 Target: Table Top Exercise Average Cost: $83,000 Full-Scale Exercise Average Cost: $124,800 7) Percentage of Critical Tasks Conclusively Analyzed FY 2004 Baseline: 85%* FY 2005 Target: 87%* FY 2006 Target: 88%* (data needs to be updated to reflect exercise results, not just AAR completeness)

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: "The National Exercise Program partners include the following: Grantees (States and Territories that receive State Homeland Security Grant funds) Sub-Grantees (Jurisdictions receiving sub-grants from States and Territories) Contractors (Vendors the NEP uses to provide direct support to exercises, as well as the vendors States and/or jurisdictions employ with grant funds) Government Partners (The other divisions and elements of SLGCP that provide funding, training, equipment, and technical assistance or federal agencies utilizing and or supporting implementation of HSEEP) Private Sector Partners (Private sector entities that have a role in prevention, protection, response and/or recovery efforts and who work regularly with Federal, State, and local governments to coordinate preparedness efforts) The difficult work of coordinating these stakeholders so that each group is working towards the NEP's annual and long-term goals is still in progress. The NEP has taken the following measures to ensure that program partners move towards contributing to the achievement of NEP goals: NEP Exercise Managers work with SLGCP Preparedness Officers to evaluate submitted State Strategies and determine if the exercise portion of the strategies supports homeland security priorities. Non-compliant State Strategies may be returned for revision, but in practice most are approved with relatively few changes. The SLGCP Training and Exercise Divisions are currently developing an AAR/IP System that will automate this coordination and tie in to existing systems within other SLGCP divisions (e.g., Virtual Technical Assistance Center, Learning Management System, Responder Knowledge Base). These collaborative efforts within SLGCP assure the advancement of the NEP??fs goal of effectively addressing the preparedness shortcomings identified during exercises. ???? The NEP??fs annual and long-term goals are based on standards defined in HSEEP, so HSEEP implementation will ensure that exercises are designed to achieve NEP goals. HSEEP compliance among contractors, States, and local jurisdictions is projected to increase dramatically in the next several years, as discussed in section 4.1. Consequently, in the near future, the majority of NEP funds will go to partners who directly support the NEP??fs goals of improving exercise performance and helping jurisdictions to identify and remedy improvement plan action items. ???? Federal agencies are encouraged to participate in regular local exercises as appropriate, and are involved in the biennial Top Officials (TOPOFF) and the Senior Officials Exercise (SOE). The participation of these Federal partners takes place in accordance with HSEEP doctrine and allows State and local entities to exercise collaboration and integration with Federal agencies (e.g. military). Furthermore, several Federal agencies have provided concrete support for the implementation of HSEEP."

Evidence: Homeland Security Grant Program HSEEP Volume I State Strategy Slides Initial Strategy Implementation Plan (ISIP)-Project Summary Report Homeland Security Act of 2002, Section 430 (c) (1)

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Each of the National Exercise Program's (NEP) TOPOFF exercises has been the subject of evaluations by outside agencies and independent contract teams drawn on subject-matter experts from across the Nation in first responder disciplines and academia. A 2003 study by ThougtLink evaluated SLGCP's exercise models and implementation in order to refine requirements and objectives. In addition, SLGCP exercises include hotwashes and debriefings in which participants provide feedback on exercise effectiveness and suggested improvements.

Evidence: ThoughtLink Report, "ODP Exercise Program Review: Opportunities for Models, Simulations, and Games" GAO Report, "COMBATING TERRORISM-Federal Response Teams Provide Varied Capabilities; Opportunities Remain to Improve Coordination," TOPOFF 2 Participant Evaluation Forms TOPOFF 2 Data Collector Logs TOPOFF 2 Methodology TOPOFF 2 Executive Summary Statement of Work for Independent Evaluations of all SLGCP Program Timeline for conducting Independent Evaluations of all SLGCP Programs Acquisition Plan for Independent Evaluations of all SLGCP Programs

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The long-term and annual goals delineated in Sections 2.1 and 2.3 were introduced relatively recently, and FY05 will be the second year in which they are used as performance measures. Consequently, no past budget request has been explicitly tied to their accomplishment. Beginning in 2005, conclusions on program performance, based on the NEP's current efforts to collect and analyze data on accomplishment of the annual and long-term goals, will inform subsequent budget requests. Specifically, requests for funding will be tied to the goals of improving critical task performance and implementing post-exercise action-items, in accordance with sections 2.1 and 2.3 above.

Evidence: FYHSP Congressional Justification FY '06 One Strategic Direction DHS PA&E Quarterly Milestone Spreadsheet, March 2005

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: DHS/SLGCP took the a number of steps to address the deficiencies identified in an assesment of its predecessor program by the Henry L. Stimson Center. The related to strategic planning included: Identifying four objectives for the foundation of the program. Developing and promulgating the Homeland Security Exercise and Evaluation Program, a standardized policy and methodology for the design, development, conduct, and evaluation of exercises. Howeve, the program has not completed the National Exercise Plan required under HSPD-8.

Evidence: "The Henry L. Stimson Center Report: The Chemical Terrorism Threat and the US Response, ThoughtLink Report, ODP Exercise Program Review: Opportunities for Models, Simulations, and Games National Exercise Program Implementation Plan HSEEP Volume IV

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Exercise Division attempts to collect a wide variety of performance information to manage the National Exercise Program (NEP) and improve performance. Data collection protocols have improved over time, but state and local compliance remains uneven. The original methodology was an exercise after-action report Analysis and Reporting Tool which captured analysis from the inherited Nunn-Lugar-Domenici Program. This data was not systematically cataloged under recently, but now allows the Exercise Division to analyze the 167 Nunn-Lugar-Domenici AARs dating back to 1997, capturing over 4,000 discrete observations from exercises. With development of the more robust HSEEP, the Exercise Division now requests all State and local exercises to complete and submit a standardized after-action report and improvement plan. Compliance with this requirement remains uneven. Of the 525 SLGCP-funded exercises in FY04, formal AAR's for just 46% were submitted as of March 2005. Besides identifying lessons learned from exercise results, the Exercise Division looks for lessons learned pertaining to exercise conduct. These exercise conduct lessons learned are used to improve the performance and efficiency of the Exercise Division's programs. Feedback forms are distributed at exercises to all NEP partners, and the data generated is used to improve the functioning of Exercise Division programs. Such improvements include determining proper staffing levels for control and evaluation and providing standardized methodology and processes. The Program recently commenced work on a Corrective Action/improvement plan Program, which will allow jurisdictions to use Web-based tools to create AARs/IPs and update the NEP on the progress of IP action item implementation. This new tool will allow the Exercise Division to identify national-level trends in IP implementation, improving the ability of SLGCP to direct funding, training, equipment, and technical assistance to remedy preparedness gaps and shortfalls. Further, the Exercise Division is currently developing a methodology to address USG Interagency-level action items identified during exercises.

Evidence: HSEEP Volume II, Introduction, "Scope and Purpose" Lessons Learned Information Sharing (LLIS) Quarterly Report

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: Contractors supporting SLGCP's Exercise Division are held accountable for their performance and the delivery of tasks enumerated in a Statement of Work. During the Top Officials Exercise 2 (TOPOFF 2) and the 2001 exercise support to the Utah Olympic Public Safety Command, the Exercise Division terminated selected contracts due to inadequate performance. Contractors are required to submit monthly reports, monthly costing, and progress timelines and milestones; which are stored in an electronic database that tracks vendor financial information and tracks invoices against incoming workplans. Vendors with invoices over budget are warned and notified that they will not be reimbursed unless the overages are justifiable. The Exercise Division's State partners are given the opportunity to provide feedback on contractor support. If negative feedback is received, the vendor is given 15 days to respond with appropriate solutions. At the end of each month, the respective SLGCP Contracting Officer's Representatives (COR) meet with each vendor to discuss overall performance, which includes a financial and quality assurance review. In addition to COR management, each SLGCP Exercise Manager fills out a 'Contractor Survey' every quarter that rates each vendor on Quality of Product or Service, Timeliness and Business Relations. Grantees and sub-grantees are also responsible for performance results. State expenditure of grant money is monitored carefully by SLGCP through required submissions of Initial Strategy Implementation Plans (ISIPs) and Biannual Strategy Implementation Reports (BSIRs). These documents provide SLGCP with detailed information as to how States obligate and spend grant funding. ISIPs and BSIRs are reviewed for consistency with program objectives by SLGCP personnel. Grant funding for exercise programs is made available to the States as part of SLGCP's Homeland Security Grant Program (HSGP). HSGP funds are allocated among States on the basis of a block grant formula, and States may choose to spend a portion of their grant funding on an exercise program. States that are going to allocate a portion of HSGP funding for exercise support are required to submit a Budget Detail Worksheet to SLGCP's Preparedness Programs Division outlining expected expenditures for exercises conducted in the State. The respective SLGCP Preparedness Officer reviews these worksheets with the SLGCP Exercise Manager to ensure it is consistent with the State Homeland Security Strategy, Multi-year Exercise Plan and is in line with the allowable exercise costs set forth in the HSEEP. Furthermore, HSEEP doctrine requires States and localities that receive Homeland Security Grant Program funding to submit an AAR/IP, which assesses performance, 60 days after each exercise is complete. "

Evidence: Memo dated January 10, 2003 to Tracy Henke Memo dated March 20, 2003 to Vice President of SAIC Memo dated December 10, 2002 to Tracy Henke Letter dated December 12, 2001 to CRA "Direct Support Exercise Schedule" Spreadsheet "Work Plan Tracking Matrix" Spreadsheet Sample Contracts Requirements, Statement of Work Synopsis of the National Exercise Program Acquisition Plan Direct Support Standard Operating Procedures "Direct Support Provider Evaluation Form"

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Two sources of funding support the Program's state and regional exercises: grant funding and direct support funding. Direct Support Funding The SLGCP budget includes a line item the Exercise Division, this funding is used to retain contractors with expertise in exercise design, conduct, and evaluation; and to provide support to Federal, State, and local entities engaged in exercise programs. These funds are only available for obligation during one year, and the Program has successfully met this requirement since 2003. Every support contract awarded for the NEP sets guidelines and minimum standards for the contractor. Oversight of the supporting contractors is carried out by the SLGCP Contracting Office, and includes monitoring spending rates, reviewing contractors' work plans, invoices, tasks submitted, and other deliverables.. All contracts are administered under the Federal Acquisition Regulations (FAR). When circumstances necessitate the close-out of a contract, DHS/SLGCP requires every vendor to notify the appropriate COR when they have reached 80% of their total contract value, per the FAR. [Could use data

Evidence: Initial Strategy Implementation Plan (ISIP)-Project Summary Report "Funding Distribution Report by Solution Area-United States-FY2004" Contract with Research Planning Inc. Inter-agency Agreement with SPAWAR Inter-agency Agreement with SBCCOM Direct Support Standard Operating Procedures Sample Work Plans Sample Invoices NLD-DPP Closeout Report "Work Plan Tracking Matrix" Spreadsheet

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: he Exercise Division ues a vendor and budget tracking system (i.e., Vendor Tracking SOP) to develop and monitor standardized budgets for each type of exercise. For example, a tabletop (discussion-based) exercise cannot cost more than $80,000, and a full-scale (operations-based) exercise can cost no more than $120,000. These budget thresholds have reduced the costs of table-tops and full-scale exercises by 14% and 26%, respectively (adjusted for inflation). This tracking system also serves to inform the National Exercise Program's (NEP) efficiency measure of "Average unit cost of direct support exercises" and assists NEP in achieving the measure's targets. In addition, the Exercise Division is developing a new, web-based system that tracks improvement actions generated from exercises. This system will also track performance measures, and will streamline the creation and implementation of post-exercise improvement plans. The program will outline processes, interactions, and decision points for all phases of improvement plan action, from initial identification and analysis to long-term tracking and reporting. In addition, the Exercise Division will conduct long-term analyses of trends in tasks, improvement plan items, remediation times, funding requirements, and other, related data.

Evidence: Average Unit Cost of Direct Support Exercises "FY04 Percentage of Critical Tasks Conclusively Analyzed" Direct Support Standard Operating Procedures"

Does the program collaborate and coordinate effectively with related programs?

Explanation: The National Exercise Program (NEP) facilitates coordination and collaboration with related programs, and is greatly dependent on the effectiveness of that collaboration in the design, planning, conduct, and evaluation of exercises. One of the focal points of the NEP's collaborative efforts has been the development and promulgation of the Homeland Security Exercise and Evaluation Program (HSEEP). To develop the program, the Exercise Division worked closely with many other agencies with exercise experience, such as the Federal Emergency Management Agency (FEMA) and the Department of Defense (DOD). Going forward, the Exercise Division is working closely with a variety of Federal agencies to update HSEEP. For example, FEMA, the Centers for Disease Control and Prevention (CDC), the Health Resources and Services Administration (HRSA), and SLGCP are co-sponsoring a National Training and Exercise Conference this year where HSEEP will be discussed with an eye toward modification. The Exercise Division is also working with many partners toward the broadest implementation of the HSEEP doctrine. For example, SLGCP is currently working with FEMA to incorporate HSEEP into exercise training courses, contributing to growing national-level exercise standardization. The Transportation Security Administration adopted HSEEP methodology for its National Intermodal Transportation Exercise and Evaluation Program. While the Exercise Division developed and administers the NEP and the HSEEP doctrine, it encourages broad participation and input. Another collaborative element of the NEP is the exercise design and planning process. Both the Top Officials Exercise Series (TOPOFF) and the Senior Officials Exercises (SOEs) are key elements of the NEP, and require extensive coordination and collaboration throughout the Federal Interagency. TOPOFF 3 in April 2005 involved over 10,000 participants and featuring representatives from across the Federal government. The NEP is also currently working closely with New York state to develop a pilot Prevention and Deterrence Exercise. The Exercise Division coordinates frequently with other programs in the implementation of its collaborative system. National Exercise Schedule (NEXS) reflects input from DOD, FEMA, HHS, and other agencies that conduct homeland security exercises. This schedule allows for better coordination, and reduces conflicts and redundancy in exercise programs.

Evidence: HSEEP Volume I, Appendix A National Exercise Schedule (NEXS) Lessons Learned Information Sharing (llis.gov) National Intermodal Transportation Exercise and Evaluation Program (NITSEEP)

Does the program use strong financial management practices?

Explanation: The Exercise Division utilizes a budget tracking system that tracks all contracts and ensures sound financial management. After a contract is awarded, a Contracting Officer Representative (COR) tracks progress on a monthly basis, overseeing expenditures, ensuring the quality and punctuality of submitted deliverables, and recording the contract's status in a central repository for all Exercise Division contracts. Vendors that wish to exceed a contract's maximum cost estimate are required to obtain approval from the COR. Furthermore, all direct support vendors are required to submit a budget tracking matrix on the fifth day of every month. This matrix captures the approved work plan budget, the amount billed to date, and any expected cost overrun and allows it to be easily referenced to the approved work plan. This regular, systematic scrutiny provides scrutiny to ensure contractor accountability and consistency. The Division has designated a single point of contact to coordinate all contract management and program budget activity. In addition to providing internal coordination, this position provides quarterly reports to the Budget Director of SLGCP to ensure precise budget tracking capabilities.

Evidence: Direct Support Standard Operating Procedures Work Plan Tracking Matrix" Spreadsheet Budget Tracking Standard Operating Procedures

Has the program taken meaningful steps to address its management deficiencies?

Explanation: Though the Exercise Division has generally sound management practices, it also seeks opportunities for improvment. Recent steps include identifying the need for a consolidation of support contracts. In the past SLGCP has managed vendor relationships through using numerous support contracts. While flexible, this practice led to a substantial amount of pass-through fees through multiple layers of contracts and sub-contracts. The National Exercise Program (NEP) is pursuing a new contract vehicle that will serve all of its program management needs, streamlining oversight and eliminate approximately 8% in pass-through expenditures. It will also provide for better cost control by the program.

Evidence: Synopsis of the National Exercise Program Acquisition Plan Participant Feedback Forms-Direct Support Exercises

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Exercise Program is making credible progress on its long-term goals. The FY05 goals were established earlier this year, and are in line with expected performance. Outcome Measure 1: Percentage of jurisdictions demonstrating acceptable performance on applicable critical tasks in exercises using SLGCP-approved scenarios: Performance Target (FY 05): 23% Actual Performance (FY 05): 23% (projected) Performance Target (FY 06): 60% Outcome Measure 2: Percentage of action items identified in after-action reports that were implemented: Performance Target (FY 05): 41% Actual Performance (FY 05): 41% (projected) Performance Target (FY 06): 65% However, achieving the FY06 goals will require a dramatic increase in both exercise performance and after-action follow-up. Full implentation and acceptance of the HSEEP will be critical to achieving this target level of performance.

Evidence: (Source: National Exercise Program - exercise data; Lessons Learned Information Sharing - after-action reports filed)"

Does the program (including program partners) achieve its annual performance goals?

Explanation: The National Exercise Program (NEP) implemented a new set of annual performance goals in FY04. As discussed in Question 4.1, the use of newly established annual performance measures was necessitated by the establishment HSEEP to provide more accurate comparisons. To determine reasonable and ambitious performance goals for FY05 and beyond, the program compared FY04 performance to available data for Quarters 1 and 2 of FY05. As the NEP's HSEEP doctrine continues to evolve, performance on the program's annual goals will continue to improve. In FY05, the NEP expects to provide direct support to 77 preparedness exercises in 62 jurisdictions. From FY05 on, the percentage of jurisdictions demonstrating acceptable performance on applicable critical tasks is expected to grow by at least 20% a year. The percentage of action items implemented will continue to increase annually, as the Corrective Action Program is implemented to track progress on post-exercise improvement plans. The number of exercises using of HSEEP for design, development, and evaluation is also expected to grow in FY05, as the HSEEP doctrine is promulgated by the Exercise Division. Other annual measures that are already at high levels (e.g. average participant satisfaction ratings and percentage of critical tasks conclusively analyzed) have maintained their high levels or shown growth in FY05 thus far. "

Evidence: "1) Outcome Measure 1 Data Set: Percentage of jurisdictions demonstrating acceptable performance on applicable critical tasks in exercises using SLGCP approved scenarios: Performance Target (FY 05): 23% Actual Performance (FY 05): 23% (projected) 2) Outcome Measure 2: Percentage of action items identified in After-Action Reports that were implemented: Performance Target (FY 05): 41% Actual Performance (FY 05): 41% (projected) 3) Average Satisfaction Rating by SLGCP Exercise Participants (Source: survey feedback from jurisdictions that participated in past exercises) in exercises (on a scale of 1 to 5, with 4 being "satisfied" and 5 "highly satisfied"): Performance Target (FY 05): 4.1 Actual Performance (FY 05): 4.1 (projected) 4) Percent of SLGCP-Supported Exercises Demonstrating Use of HSEEP to Design, Develop, and Evaluate Their Exercises Performance Target (FY 05): 70% Actual Performance (FY 05): 70% (projected) 5) Number of ODP grant funded exercise projects per year: State Assessment data Performance Target (FY 05): 540 Actual Performance (FY 05): 540 (projected) 7) FY04 Percentage of Critical Tasks Conclusively Analyzed Performance Target (FY 05): 87% Actual Performance (FY 05): 87% (projected)"

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The National Exercise Program (NEP) has demonstrated improved efficiencies and cost-effectiveness. The NEP has standardized the work plans and cost schedules for its entire inventory of exercises, reducing the per-unit cost of direct support tabletop exercises by 14 percent and full-scale exercises by 26 percent (inflation-adjusted). The NEP is also improving the efficiency processes to share Lessons Learned by publicizing and enhancing Lessons Learned Information Sharing, the national, Web-based network of Lessons Learned and Best Practices for emergency response providers and homeland security officials. The average unit cost of direct support exercises for FY05 is expected to be in line with performance targets. The FY05 goal for discussion-based Table Tops was an average cost of $80,000; and the actual average for FY05 Quarter 2 is $79,312. The goals for full-scale operations-based exercises was $120,000; and the projected actual for FY05 Quarter 2 is $118,661.

Evidence: 1) Decrease in the per unit cost of direct support exercises (Source: Exercise Division Budget materials; after-action reports (AARs))

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The National Exercise Program (NEP) compares favorably with other exercise programs developed by the Federal government to improved the preparedness of state and local governments. NEP's utilization of a formal exercise doctrine provides more consistent operational guidance and after-action reporting. A more ambitious comparison for the NEP's design and evaluation methodology would be to the Department of Defense's exercise program. The NEP's HSEEP doctrine was developed with an awareness of the DOD exercise programs and strives to incorporate valuable features. The emphasis on capabilities-based planning throughout SLGCP, has roots in the military's training doctrine. HSEEP's use of professional observers to control the exercise and enhance training value is also very similar to the military model. The most significant differences between DOD and HSEEP are dictated by the differences in authority, scope, and resources available. Unlike DOD, the DHS and the NEP lack command authority over exercise participants to implement action items. DOD exercises also tend to be longer in duration and involve more participants. In addition, the DOD exercise program is a more mature program, providing a long baseline of exercise performance data, supplemented by the periodic "check" provided by real-world operations. Another DHS exercise program, CSEPP is a national initiative to enhance readiness for an accidental release of chemical weapons agent from any of the continental U.S. stockpile storage installations. The CSEPP evaluation methodology has evolved to emphasize and define successful demonstration of required functional areas. SLGCP's Exercise Division developed HSEEP EEGs from the CSEPP model, applying the evaluation guides to a far wider scope of scenarios, and consequently a far wider set of response issues. Jurisdictional involvement and coordination in the exercise design and development process is another area where CSEPP and HSEEP compare favorably and have benefited from one another. CSEPP support personnel have developed tools to better integrate exercise planners in the planning process. Building on CSEPP's model, SLGCP is creating a comprehensive HSEEP Toolkit that will assist planners with all aspects of exercise planning and development. Community profiles and extent of play agreements are also tools that have assisted CSEPP in gaining buy-in and participation from the CSEPP communities. HSEEP has taken this approach to another level by encouraging Multi-Year Strategies and Exercise Planning Workshops to assist States and localities with developing a comprehensive building-block exercise program. International Programs The NEP also serves as a model for the international community for exercise policy and methodology. For example, the Public Safety and Emergency Preparedness Canada (PSEPC) met with several representatives from the ODP Exercise Division to develop their own respective exercise program and policy based on HSEEP. Australian officials have also visited with NEP representatives to gather information about the program. Conversely, the NEP has been collecting best practices from other countries, particularly in the field of prevention and deterrence.

Evidence: Army Field Manual (FM), 25-100, "Training the Force" Army Field Manual (FM), 25-101, "Battle-Focused Training" Chemical Stockpile Exercise Preparedness Program (CSEPP) Blue Book HSEEP Volume II After-Action Report sample

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: ThoughtLink, Inc. evaluated the simulations used in SLGCP's training and exercise programs. Their March 2003 report was based on observations of 12 exercises and numerous interviews. The study concluded that the exercise program "works well" and was well-regarded by participants. Strengths observed by ThoughtLink included: strong functional and training expertise in the exercise support teams; good facilitation by the teams; and use of exercise templates with customizable elements. The suggested areas of improvement include: Improving participant readiness for exercises; Sharing best practices with local communities; and Conducting pre-exercise audits of city plans and procedures A GAO report from November 2000 discussed the Top Officials 2000 (TOPOFF 2000) exercise. GAO found that TOPOFF 2000 largely met its overall goals, represented progress over previous combating terrorism exercises, but noted that the lack of an interagency process for collecting after-action data delayed ODP's release of the formal after-action reoprt. During the Top Officials 2 (TOPOFF 2) Full-Scale Exercise in 2003, DHS invited representatives of Stanford University's Center for International Security & Cooperation. These subject-matter experts issued an independent report identifying rooms for operational improvement, and suggesting that the exercise served as an effective means of identifying preparedness shortcomings. Evaluations of the recent Top Officials 3 (TOPOFF 3) exercise by the Center for Naval Analyses and Department of Homeland Security's Inspector General are currently underway.

Evidence: ThoughtLink Report,ODP Exercise Program Review: Opportunities for Models, Simulations, and Games GAO Report, COMBATING TERRORISM-Federal Response Teams Provide Varied Capabilities; Opportunities Remain to Improve Coordination,??h pages 23-24 GAO Report, COMBATING TERRORISM-Selected Challenges and Related Recommendations,page 71 Center for International Security and Cooperation, Institute for International Studies Final Report, TOPOFF 2 Full-Scale Exercise- May 11-15, 2003