Detailed Information on the
Transportation Security Administration: Air Cargo Security Programs Assessment

Develop a long-term outcome oriented performance measure that measures risk reduced as a result of implementing program objectives.

TSA will develop a performance management system that clearly defines and quantifies performance standards and accountability for managers. Institutional guidance will be provided to all staff on the establishment of this new system and development of personnel performance evaluation criteria.

Deploy the Freight Assessment System to improve gap analyses and risk/vulnerability assessments. Obtain Authority to operate by January 31, 2008.

Measure: Reduce the number of days it takes to certify a new applicant as an Indirect Air Carrier.

Explanation:The current method to certify an Indirect Air Carrier is labor intensive. TSA has recently implemented an automated system that provides an online application process for new applicants. The measure will capture the time saved through automation.

Measure: Increase the percentage of Indirect Air Carriers found to be compliant with TSA standard security programs.

Explanation:Standard Security Programs provide detailed guidance to regulated parties such as Indirect Air Carriers (IACs) on how to implement regulatory requirements. Continuing education, outreach efforts, and targeting additional resources on IACs identified as noncompliant, will decrease the violation rate.

Measure: Increase inspection of identified "elevated risk" air cargo on all air carriers.

Explanation:Currently, TSA is not identifying air cargo as "elevated risk." All cargo is being screened randomly. Through a phased-approach, TSA will identify and inspect "elevated risk" air cargo on all aircarriers by 2011. The phases are as follows: FY 2008: domestic; FY 2009: all cargo; FY 2010: international inbound; FY 2011: international outbound.

Measure: Reduce the average time (in minutes) per initial inspection of an Indirect Air Carrier.

Explanation:The program is developing advanced training for the Cargo Aviation Security Inspectors and investing in productivity tools to help them conduct and document inspections more efficiently. This measure will quantify how these improvements will reduce the amount of time spent to inspect and document the results of Indirect Air Carrier inspections.

Measure: Risk reduced by Air Cargo programs and regulations (New measure, added February 2008)

Explanation:The risk index measures the capabilities of air cargo programs in reducing vulnerabilities in the air cargo supply chain to multiple threat case scenarios and evaluating the effectiveness of Air Cargo's countermeasures - programs, policies, and regulations- in reducing the risks posed to the air cargo supply chain. The threat case scenarios are identified based on adversary's capabilities, intent, history, and operational environment and are assigned a risk score. The Risk Reduction Index is calculated based on how air cargo programs and policies assess and counter vulnerabilities identified in the threat case scenarios. The index is a tool to help Air Cargo programs mitigate vulnerabilities that are identified and are not currently being addressed.

Measure: Percent of risk reduction for air cargo on passenger aircraft.

Explanation:The scope of this data is based on all federalized airports. Data only include passengers transported, flight departures performed and cargo volume by passenger aircraft configurations. There are currently a total of 461 airports (Category X, I, II, III, IV). A risk score is determined for each airport and ranked by airport category for all federal airports. These scores are calculated by comparing the passenger and cargo volume for a specific airport to the total passenger and cargo volume for all flights in 2007 BTS data. The risk index includes TSA's assessment of the progress of air cargo programs in implementing various screening mandates including: 1) 100% screening of cargo on a narrow body aircraft; 2) Certified Cargo Screening Program (CCSP) requirements for shipper, Indirect Air Carriers (IAC), Air Carriers (AC), the interim and the final rule for air cargo, and amendments to the Standard Security Programs. Data is collected from two sources: 1) passenger and cargo volume for all flight information provided to Bureau of Transportation Statistics (BTS) by air carriers and 2) Cargo screening data provided directly to TSA by air carriers. The Air Carriers provide TSA with screening data for the amount and type of cargo screened at each airport. The data is entered and stored in a TSA database (Temporary Cargo Database) which is maintained by the TSA Office of Transportation System Network Management (TSNM) Air Cargo Division on TSA's central computer system located at TSA headquarters. TSA gathers the data electronically from BTS and directly from air carriers each month. TSA's TSNM Air Cargo Division then reviews and analyzes the data for security improvements to inform risk reduction activities. TSNM Air Cargo verifies and validates the data and enters the information into the temporary cargo statistics database. TSNM Air Cargo then calculates the index to generate quarterly reports on passenger air cargo screening performance. The amount of risk reduction is determined by comparing the level of air cargo risk from a prior period to the level of risk in the current period. The index uses a scale from 1 to 100 where a value of 100 represents a high risk level (i.e. if the Index value equals 100, then no risk is being reduced).

Is the program purpose clear?

Explanation: The Air Cargo program develops and deploys advanced programs and systems to ensure the safe and secure transport of passengers and property in air transportation.

Evidence: Air Cargo Strategic Plan; Air Cargo Mission Need Statement.

Does the program address a specific and existing problem, interest, or need?

Explanation: The Air Cargo program addresses the risks, vulnerabilities and threats to the national homeland associated with the approximately 39 billions of tons of air cargo transported annually by passenger and all-cargo air carriers.

Evidence: Air Cargo Strategic Plan.

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: The Air Cargo program is designed to fill a unique role as the national oversight authority for domestic air cargo security. The program provides standardized, consistent and coordinated delivery of policy, guidance and compliance inspections to the air cargo industry. Its national scope allows it to provide protective mitigation of air cargo security vulnerabilities.

Evidence: Air Cargo Strategic Plan; Aviation Transportation Security Act (ATSA) P.L. 107-71.

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The Air Cargo Program is relatively new--created in February 2003--and is being enhanced routinely to ensure the programs' design is free of design flaws. For example the program is currently working with DHS's Customs and Border Protection (CBP) program to develop a supplicated, online interactive program known as the Freight Assessment System. This system will help improve interaction with air carriers and improve risk assessment methodologies.

Evidence: Air Cargo Threat Assessment DOT/FAA/AR-02/15; GAO-04-592T; GAO Air Cargo Report to be released July 2005.

Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Explanation: While TSA has made strides to secure air cargo on domestic and all-cargo air carriers, work is underway to improve gap analyses and risk/vulnerability assessments to ensure that the resources are effectively targeted and reaching intended beneficiaries. The development of the Freight Assessment System (FAS) will employ methods to better utilize these assessment of risks and vulnerabilities. The FAS will incorporate various data elements to determine levels of risk for each piece of cargo thus, reducing the current reliance on random inspection of cargo.

Evidence: Freight Assessment System program documentation; GAO report-04-592T; GAO report (pending release 7/05).

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: Due to data limitations, at this time, the program is unable to measure the risk reduced as a result of implementing program objectives. Therefore, it has developed an interim long-term output goal that helps gather data necessary to develop outcome goals. This output goal is: by 2011, increase the percentage of Indirect Air Carriers (IAC) found to be compliant with TSA standard security programs to at least 94 percent. These measures will be included in the FY 2007 Congressional Justification.

Evidence: FY 2007 TSA Congressional Justification.

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: The program has a baseline and targets for the new long-term output goal.

Evidence: FY 2007 TSA Congressional Justification.

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: The program has a limited number of specific annual performance measures, including efficiency measures that support the long-term goal to reduce the risk of transporting air cargo. These goals include: reducing the time to inspect indirect air carriers (IAC) and increasing the percentage of IAC compliance with security directives.

Evidence: FY 2006 and 2007 TSA Congressional Justification.

Does the program have baselines and ambitious targets for its annual measures?

Explanation: The program has established baselines and ambitious targets for its annual measures.

Evidence: Air Cargo Security Program Pamphlet, Air Cargo Road Map, FAS Evolution Gap Analysis team.

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: The air cargo program partners with groups at the federal, local, state, and private levels. Through the Aviation Security Advisory committee and smaller working groups, TSA relies on these groups for recommendations on scope and management of the program. They are working with industry on the development of their long-term and annual goals, but industry is not committed to working towards achieving these goals yet.

Evidence: Aviation Security Advisory Committee Charter, ASAC recommendations to TSA, Aviation Security Advisory Committee Final Recommendations of the Working Groups to TSA, Air Cargo Strategic Plan.

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: There have been several independent comprehensive studies of the program to evaluate program improvements and evaluate program effectiveness; for example, one report was released in March 2004 and addressed areas that need improvement for the entire system. There are also two reports pending by the GAO and Inspector General on the status and effectiveness of TSA implementing the air cargo program; these studies are comprehensive in scope and independent.

Evidence: GAO Reports: "Aviation Security: Improvement Still Needed in Federal Aviation Security Efforts" (3/30/04) and Air Cargo Security Procedures (pending release 7/05); DOT IG Report: "Audit of the Cargo Security Program" (9/19/2002); National Safe Skies Vulnerability Assessment (Draft Report), GAO Report 03-344 "Vulnerabilities and Potential Improvements for the Air Cargo System" (pending release 7/05).

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: Air Cargo is uniquely identified in the TSA Budget requests and performance targets are tied to the funding level.

Evidence: FY 2005, FY 2006, and FY 2007 TSA Congressional Justification.

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: The Air Cargo program has taken steps to adopt a limited number of specific, ambitious long-term goals and annual goals with baselines, targets, and timeframes. These goals will support improvements in program management and increase efficiency and effectiveness.

Evidence: FY 2007 TSA Congressional Justification.

Are all regulations issued by the program/agency necessary to meet the stated goals of the program, and do all regulations clearly indicate how the rules contribute to achievement of the goals?

Explanation: The Air Cargo Notice of Proposed Rulemaking (NPRM) provides clear specified goals to enhance the security of air cargo transportation. The goal is to ensure that DHS has a comprehensive, coordinated policy for securing air cargo within the U.S. This is accomplished by requiring security threat assessments for individuals with unescorted access to cargo, enhancing security requirements for Indirect Air Carriers, and establishing new training requirements for personnel to include protection of stored or enroute cargo.

Evidence: 49 CFR 1540 et al: Air Cargo Security Requirements Proposed Rule (November 10, 2004).

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: The Air Cargo program, a regulatory program, collects data from air carriers to measure industry compliance with federal regulations and standard security programs. The program identifies deficiencies in monthly reports and uses the data to enhance education and outreach efforts to increase compliance.

Evidence: Performance And Results Inspection System, Indirect Air Carrier, Known Shipper Databases slides, CBP GAP Analysis.

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: While TSA managers are held accountable for certain program responsibilities, the program has not established clearly quantifiable performance standards for those managers. TSA is in the process of developing a pay-for-performance system.

Evidence: DHS Management Directive 0782 establishes the policies and procedures for TSA program managers. TSA individual annual performance reports.

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Air Cargo develops annual FY spend plans that are used to obligate funding. Typically, obligations occur during the 3rd and 4th quarters because of prior year buy ahead plans and length of time to execute contracts.

Evidence: FY-05 Spend Plan, Air Cargo Security Program Road Map.

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: The program has developed two efficiency goals: reduce the time to inspect IACs and reduce the time to certify IACs. In addition, the program is leveraging a CBP technology investment to achieve cost efficiencies, and conducting a cost/benefit analysis of all of its major investments.

Evidence: Air Cargo Cost/Benefit Anaylsis Plan; CPB/TSA partnership documents for Freight Assessment System; FY 2007 TSA Congressional Justification.

Does the program collaborate and coordinate effectively with related programs?

Explanation: The Air Cargo Program works collaboratively with CBP on the development of the Freight Assessment System (FAS). Specific roles and responsibilities are assigned and these activities have resulted in reduced resource commitments for both agencies. The program also partners with industry via the Aviation Security Advisory Committee (ASAC), which ensures all air cargo stakeholders' views are incorporated when determining techonology, policy, data management, and modeling techniques.

Evidence: Interagency agreement with CBP on FAS, ASAC Charter, Joint Work Break Down Structure.

Does the program use strong financial management practices?

Explanation: The TSA FY 2004 Audit of Financial Statements revealed material weaknesses in the areas of information technology and internal control monitoring and evaluation.

Evidence: FY 2004 Audit of Financial Statements.

Has the program taken meaningful steps to address its management deficiencies?

Explanation: TSA is taking steps to address the financial management deficiencies by implementing an agency-wide internal/management control program to ensure compliance with the Federal Managers' Financial Integrity Act. Furthermore, TSA will establish a pay for performance system that will clearly define quantifiable performance standards for holding managers accountable.

Evidence: DHS Management Directive 0782 establishes the policies and procedures for TSA program managers. TSA individual annual performance reports. Response to FY 2004 Audit of Financial Statements.

Did the program seek and take into account the views of all affected parties (e.g., consumers; large and small businesses; State, local and tribal governments; beneficiaries; and the general public) when developing significant regulations?

Explanation: Stakeholder views were instrumental in the development of the NPRM.?? Additionally, after the NPRM was published in the Federal Register, a 60-day public comment period was conducted.?? TSA received a total of 676 comments from 134 unique commenters.?? Each of those comments has been considered and the agency responses have been incorporated into the Preamble of the final regulations.??

Evidence: 49 CFR 1540 et al: Air Cargo Security Requirements Proposed Rule (November 10, 2004).

Did the program prepare adequate regulatory impact analyses if required by Executive Order 12866, regulatory flexibility analyses if required by the Regulatory Flexibility Act and SBREFA, and cost-benefit analyses if required under the Unfunded Mandates Reform Act; and did those analyses comply with OMB guidelines?

Explanation: All appropriate analyses were??completed and published in the NPRM on November 10, 2004.?? Update of those analyses based on responses received during the public comment period is complete and the executive branch clearance process is underway.

Evidence: 49 CFR 1540 et al: Air Cargo Security Requirements Proposed Rule (November 10, 2004).

Does the program systematically review its current regulations to ensure consistency among all regulations in accomplishing program goals?

Explanation: Development of the air cargo security regulations is underway. The program is already considering??a second round of revisions and recognizes that additional regulatory changes will be required in order to implement the FAS program.?? Once the initial massive rewrite of the regulation is complete, the program intends to conduct an annual review to identify additional changes that may be required to address emerging threats and vulnerabilities.??

Evidence: 49 CFR 1540 et al: Air Cargo Security Requirements Proposed Rule (November 10, 2004).

Are the regulations designed to achieve program goals, to the extent practicable, by maximizing the net benefits of its regulatory activity?

Explanation: The NPRM is designed to maximize net benefits to increase security of the air transportation system through more effective screening and procedures of air cargo. In addition, the Office of Chief Counsel actively looks for opportunities to leverage ongoing rulemaking activities to implement??all needed changes concurrently.?? For example, a legislative correction to the weight of aircraft over 12,500 lbs??was incorporated into the air cargo NPRM for efficiency.

Evidence: 49 CFR 1540 et al: Air Cargo Security Requirements Proposed Rule (November 10, 2004).

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: The Air Cargo program's long-term output goal is new. Although a baseline has been established, the program has not yet demonstrated progress in achieving the performance goal.

Evidence: Air Cargo Strategic Plan; FY 2005, FY 2006, and FY 2007 TSA Congressional Justifications.

Does the program (including program partners) achieve its annual performance goals?

Explanation: The Air Cargo program's annual goals are new. Although baselines and targets have been established, the program has not yet demonstrated progress in achieving these performance goals.

Evidence: FY 2007 TSA Congressional Justification.

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The Air Cargo program's efficiency goals are new. Although baselines and targets have been established, the program has not yet demonstrated progress in achieving these performance goals. However, the program performed an alternative analysis and determined it would be more cost-effective (estimated savings of about 15 percent) to partner with CBP on the development of the Freight Assessment System.

Evidence: FY 2007 TSA Congressional Justification. FY 2006 Exhibit 300 on Air Cargo.

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: The TSA Air Cargo program is most similar to the CBP international air cargo program. It is difficult to compare the performance of these programs because both inspection processes have been criticized, noting improvements are needed to ensure weapons of mass destruction or other implements do not gain access to the United States. Both programs are taking steps to address these deficiencies and TSA is working jointly with CBP to leverage technology, develop joint situational awareness, screening and targeting capabilities.

Evidence: TSA/CPB interim letter of agreement January 31, 2005.

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Independent reviews of the air cargo program indicated that although there have been some improvements made, work remains to close security loopholes, including improving screening efforts and refining procedures to approve indirect air carriers. The development of a strategic plan and the Freight Assessment System is a step in the right direction to address these issues. The NPRM also addresses these issues.

Evidence: GAO Reports: "Aviation Security: Improvement Still Needed in Federal Aviation Security Efforts" (3/30/04) and Air Cargo Security Procedures (pending release 7/05); DOT IG Report: "Audit of the Cargo Security Program" (9/19/2002); National Safe Skies Vulnerability Assessment (Draft Report), GAO Report 03-344 "Vulnerabilities and Potential Improvements for the Air Cargo System" (pending release).

Were programmatic goals (and benefits) achieved at the least incremental societal cost and did the program maximize net benefits?

Explanation: TSA has not yet released final regulations implementing the air cargo program. Therefore, it is premature at this time to evaluate if program goals achieved the least incremental societal cost. However, TSA is making changes to the program that will have an impact on the security of the air transportation system, which in other words will maximize net benefits.

Evidence: 49 CFR 1540 et al: Air Cargo Security Requirements Proposed Rule (November 10, 2004).