Detailed Information on the
Transportation Security Administration: Flight Crew Training Assessment

By the end of FY 2009, TSA will develop and deploy a pay-for-performance system to hold federal managers accountable for cost, schedule, and performance results.

Develop a long-term outcome oriented performance measure that measures risk reduced as a result of implementing program objectives. This goal will be included in the FY 2009 Budget.

By August 30th of each year, provide an annual report on FFDO enrollment, attrition, and training needs.

Develop an efficiency measure that better accounts for all activities occurring in the flight crew program.

By FY 2007, develop baselines and ambitious targets for the annual measures.

By FY 2007, develop a plan to systemmatically review current regulations to ensure consistency among all regulations in accomplishing program goals.

By FY 2007, develop an independent evaluation process to ensure programs are comprehensively reviewed on a regular basis.

Measure: Increase the number of positive responses on the following TSA survey question: How confident are you in the ability of the flight crew to keep air travel secure and to defend the aircraft and its passengers from individuals with hostile intentions?

Explanation:This question will be administered in the annual TSA customer survey.

Measure: Percent of eligible flight crew members with Federal Flight Deck Officer (FFDO) certification.

Explanation:This measure reports the cumulative percent of eligible flight crew members that are trained and deputized as Federal Flight Deck Officers (FFDO). The FFDO certification process trains crew members to provide a final, deadly-force deterrent to an attempted hostile breach of the flight deck that could result in the loss of an aircraft, its passengers, and its crew. Increasing the percent of eligible flight crew members deputized as FFDO's increases the likelihood that an FFDO is protecting the flight deck. The increased presence of the FFDOs, improves the confidence of the flying public regarding the flight crew's ability to protect the aircraft.

Measure: Increase trainee satisfaction.

Explanation:Trainee satisfaction is measured by the five-point Likert scale measurement of Level I-Trainee Satisfaction.

Measure: Increase the percentage of training programs within 10 percent of cost, schedule and performance.

Explanation:This is a percentage of projects and programs which have output results of cost, schedule, and performance within 10% of annual plans.

Measure: Improve protection of aircraft cockpits through re-qualification certification FFDOs.

Explanation:Calculated percentage of the number of successful and failed re-qualifications. Measure targets to be developed based upon FY05 (baseline) variance.

Is the program purpose clear?

Explanation: The purpose of the Flight Crew Training program is to provide training to volunteer crewmembers to prevent acts directed against commercial aviation that could result in mass violence and death, destruction of property, and damage to the national economy.

Evidence: Aviation and Transportation Security Act (ATSA), Public Law [P.L.] 107-71; Homeland Security Act of 2002, Title XIV, Arming Pilots Against Terrorism Act (49 U.S.C. § 44921); Vision 100 (49 U.S.C. 40101) - Century of Aviation Authorization Act (P.L. 108-176), Section 603, Crew Training.

Does the program address a specific and existing problem, interest, or need?

Explanation: The program was created by statute in response to the terrorist attacks of September 11, 2001. They provide training to volunteer flight and cabin crewmembers that is essential for them to defend themselves and their aircraft against hostile or threatening acts.

Evidence: ATSA; 49 U.S.C. § 44921; P.L. 108-176; Vision-100.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The program provides an additional layer of security on commercial aircraft. Although the training for the Federal Flight Deck Officer (FFDO) program is mirrored after the Federal Air Marshals (FAMs), TSA has exclusive statutory responsibility for arming pilots to defend the flight deck and providing training to crew members on self defense. FAMs are responsible for and protect air security by detecting, deterring and defeating hostile acts targeting U.S. air carriers, airports, passengers and crews.

Evidence: Federal Flight Deck Officer Program: Interim Status Report (January 3, 2003); Federal Flight Deck Officer Program: Final Report to the Policy Executive Steering Committee (February 12, 2003); Action Memorandum to the Under Secretary: Establishment of the Federal Flight Deck Officer Program (February 25, 2003); Vision 100 (49 U.S.C. 40101) - Century of Aviation Authorization Act (P.L. 108-176), Section 603, Crew Training.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: Oversight inquiries note program design meets the spirit and the letter of both Vision 100 and ATSA. In developing the program, TSA partnered with experts in aviation security, law enforcement, and the airline industry to ensure program efficiency and effectiveness. For example, training for FFDO's is conducted at the Federal Law Enforcement Training Center (FLETC) facility and results in significant cost avoidance; the Crew Member Self Defense Training (CMSDT) program utilizes the national community college system to maximize convenience and accessibility for participants.

Evidence: GAO "Flight and Cabin Crew Security Training (January 13, 2005); GAO "Workforce Performance and Training Follow-up Questions" (February 24, 2005); QFs (August 10, 2004): GAO "Crew Member Self Defense Training Program" (January 15, 2005)

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: The program's resources directly support its training objective. Specifically, for FFDO, 68% of the program's resources supports initial and re-qualification training activities and 24% support the application and assessment process; operational oversight costs account for the remaining 8%. For CMSDT, 95% of resources support the delivery of the training program; the remaining 5% support program management and overhead costs, including printing and distribution of training materials.

Evidence: FFDO & CMSDT Spend Plans; FLETC Reimbursable Agreement; Oversight inquiries.

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Due to data limitations, at this time, the program is unable to measure the risk reduced as a result of implementing program objectives. Therefore, it has developed an interim long-term output goal that helps gather data necessary to develop outcome goals. This goal is increasing the number of positive responses on the following TSA survey question: How confident are you in the ability of the flight crew to keep air travel secure and to defend the aircraft and its passengers from individuals with hostile intentions? In the long-term TSA will develop a measure that focuses on reducing the risk of a potenial hostile or treatening act on an aircraft.

Evidence: FY 2007 Congressional Justificiation.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The baselines and targets for the program's long-term goals are under development.

Evidence: Draft goals measures sheet. FY 2007 Congressional Justification.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Through the PART process, the program developed a limited number of specific annual performance measures that demonstrate progress towards achieving the program's long-term goals. These measures include: training satisfaction and an efficiency measure of the increasing the number of training programs within 10 percent of cost and schedule. However, these measures do not demonstrate process towards achieving the long-term goal of reducing the risk of a potential terrorist attack or the the interim goal of making the public feel safer.

Evidence: FY 2006 and FY 2007 Congressional Justificiation.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program hopes to establish baselines for the annual goals will be developed in FY 2005; targets will be established in 2006.

Evidence: Draft goals measures sheet. FY 2007 Congressional Justificiation.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The program's main partners--FLETC, Lockheed Martin Corporation, and the American Association of Community Colleges (AACC)--operate under the TSA prescribed standards for instructors, facilities, and equipment, and are committed to the annual and/or long-term goals of the program.

Evidence: FLETC Reimbursable Agreement; Lockheed Martin Specialized Security Training Contract (SSTC) Instructors providing firearms re-qualification support; Cooperative Agreement Between the Transportation Security Administration and the American Association of Community Colleges.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: There have not been any evaluation of the Flight Crew Training program. TSA will conduct a comprehensive independent review of the program in the near future. TSA will issue the draft Request for Proposals by August 2005 and select the contractor to perform the work by the fourth quarter of fiscal year 2005. Also, oversight inquiries note both programs meet hte spirit and letter of both Vision 100 and ATSA.

Evidence: Statement of Objectives outlining intent of upcoming audit plans.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The program is uniquely identified in TSA budget requests and performance targets are tied to funding levels.

Evidence: FY 2006 and FY 2007 Congressional Justificiation.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The program will develop long-term outcome goals with baselines and targets, and an independent evaluation plan to ensure assessments are of high quality, sufficient scope, and conducted regularly.

Evidence: FY 2007 Congressional Justificiation; Audit Planning Documents.

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: TSA regularly collects a large amount of timely and credible performance-based data from a number of sources, facilitating effective management decisions TSA-wide. For example, training performance data, including trainee evaluations, course quality assessments, and program performance rates. All collected data is analyzed with customer satisfaction and quality assurance results to drive training performance improvements.

Evidence: TSA's Performance Measurement Information System (PMIS).

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: While TSA managers are held accountable for certain program responsibilities, the program has not established clearly quantifiable performance standards for those managers. TSA is in the process of developing a pay-for-performance system.

Evidence: DHS Management Directive 0782 establishes the policies and procedures for TSA program managers. TSA individual annual performance reports. TSA Management Directive No. 300.8 - Implements TSA policies and procedures for program planning, review, and reporting.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Flight Crew Training resources are obligated and expended for the intended program purpose, including improvements, in a timely manner.

Evidence: Flight Crew Training budget execution data.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has developed efficiency goals: increase the percentage of training programs within 10 percent of cost, schedule, and performance. TSA will develop baselines and targets for this efficiency measure in the FY 2007 Congressional Justification.

Evidence: FY 2007 TSA Congressional Justification.

Does the program collaborate and coordinate effectively with related programs?

Explanation: TSA monitors and assesses flight crew performance holistically, meaning that all crew environmental aspects, including training, are considered in determining program direction, technology, performance standards and procedures, and TSA program management and oversight. Each aspect is actively discussed with program experts and stakeholders to produce unified flight crew performance outcomes.

Evidence: Flight Crew Training program design, standards, and management are directly tied to the advanced self-defense training objectives identified in the governing legislation.

Does the program use strong financial management practices?

Explanation: While TSA received a clean audit opinion, it received a material weakness in internal control, including payroll and personnel security systems, and records retention management.

Evidence: FY 2004 Audit of Financial Statements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: TSA is taking steps to address the financial management deficiencies by implementing an agency-wide internal/management control program to ensure compliance with the Federal Managers' Financial Integrity Act. Furthermore, TSA will establish a pay for performance system that will clearly define quantifiable performance standards for holding managers accountable.

Evidence: DHS Management Directive 0782 establishes the policies and procedures for TSA program managers. TSA individual annual performance reports. Response to FY 2004 Audit of Financial Statements. TSA Management Directive No. 300.8 - Implements TSA policies and procedures for program planning, review, and reporting.

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The long-term measures have not yet been developed; therefore, the program has not yet demonstrated adequate progress in achieving these goals.

Evidence: AMPTAP Level I Data: Curriculum Ratings - Overall Averages by Course.

Does the program (including program partners) achieve its annual performance goals?

Explanation: Baselines for the annual goals will be established in FY 2005. Therefore, progress towards meeting these goals will not be evident until FY 2006 at the earliest.

Evidence: FY 20067 TSA Congressional Justification.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: Baselines for the efficiency goals will be established in FY 2005. Therefore, progress towards meeting these goals will not be evident until FY 2006 at the earliest.

Evidence: FY 20067 TSA Congressional Justification.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The CMSDT program is unique in its advanced self-defense, threat detection, and appropriate use of force techniques. While the FFDO is the only program that deputizes airline crew members as Federal Law Enforcement Officers, it is similar in mission to FAM's. The only independent evaluation of FFDO and FAMs was conducted prior to the establishment of the FFDO program and did not compare FAM and FFDO performance, rather it primarily addressed implementation issues.

Evidence: CRS: Arming Pilots Against Terrorism. January 9, 2004.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: The program has not yet undergone a Federal oversight or external program evaluation that meet the requirements. However, oversight inquiries note both programs' design and performance plans meet the spirit and the letter of both Vision 100 and ATSA.

Evidence: GAO "Flight and Cabin Crew Security Training (January 13, 2005); GAO "Workforce Performance and Training Follow-up Questions" (February 24, 2005); QFs (August 10, 2004): GAO "Crew Member Self Defense Training Program" (January 15, 2005)