This is the accessible text file for GAO report number GAO-06-266T entitled 'Aviation Safety: FAA's Safety Oversight System Is Effective but Could Benefit from Better Evaluation of Its Programs' Performance' which was released on November 17, 2005. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Testimony: Before the Subcommittee on Aviation, Committee on Commerce, Science, and Transportation, U.S. Senate: United States Government Accountability Office: GAO: Not to Be Released Before10:00 a.m. EST: Thursday, November 17, 2005: Aviation Safety: FAA's Safety Oversight System Is Effective but Could Benefit from Better Evaluation of Its Programs' Performance: Statement for the Record by Gerald L. Dillingham, Ph.D., Director: Physical Infrastructure Issues: GAO-06-266T: GAO Highlights: Highlights of GAO-06-266T, a testimony to the Subcommittee on Aviation, Committee on Commerce, Science, and Transportation, U.S. Senate: Why GAO Did This Study: The U.S. commercial aviation industry has an extraordinary safety record. However, when passenger airlines have accidents or serious incidents, regardless of their rarity, the consequences can be tragic. The Federal Aviation Administration (FAA) works to maintain a high level of safety through an effective safety oversight system. Keys to this system are to: (1) establish programs that focus resources on areas of highest safety risk and on mitigating risks; (2) provide training and communication to ensure that inspectors can consistently carry out the agency’s oversight programs; and (3) have processes and data to continuously monitor, evaluate, and improve the numerous oversight programs that make up the safety oversight system. This statement focuses on these three key areas and is based on recent GAO reports on FAA’s inspection oversight programs, industry partnership programs, enforcement program, and training program. What GAO Found: FAA’s safety oversight system includes programs that focus on risk identification and mitigation through a risk-based system safety approach, leveraging of resources through designee and partnership relationships, and enforcement of safety regulations, but the benefits of these programs are not being fully realized. For example, FAA’s system safety approach includes the addition of a program that emphasizes risk identification to its traditional inspection program for overseeing some airlines, which is not based on risk. However, it is likely that the benefits of this approach could be enhanced if the inspection workload was not as heavily oriented to the traditional inspection program’s non-risk based activities. FAA leverages its resources through its designee programs, in which designated individuals and organizations perform about 90 percent of certification- related activities, and through its industry partnership programs, which are designed to assist the agency in receiving safety information. An outgrowth of FAA’s inspection process is its enforcement program, which is intended to ensure industry compliance with safety regulations. However, GAO has expressed concerns that this program may not be as effective as it could be in deterring violations. FAA has made training an integral part of its safety oversight system, but several actions could improve the results of its training efforts, including ensuring that inspectors are well-trained in FAA’s system safety approach and have sufficient knowledge of increasingly complex aircraft and systems to effectively identify safety risks. FAA has established mandatory training requirements for its workforce and designees. We have reported that FAA has generally followed effective management practices for planning, developing, delivering, and assessing the impact of its technical training for safety inspectors. GAO has found inadequate evaluative processes and limitations with data for FAA’s inspection programs, designee programs, industry partnership programs, and enforcement program. For example, FAA lacked requirements or criteria for evaluating its designee programs. In another example, FAA’s nationwide enforcement database is not as useful as it could be because of missing or incomplete historical information about enforcement cases. FAA’s Safety Oversight System Includes Inspections, Training, and Other Activities: [See PDF for image] [End of figure] What GAO Recommends: In order to help FAA fully realize the benefits of its safety oversight system, GAO has made a number of recommendations to address weaknesses identified in previous reviews, such as improving program evaluation and data. These recommendations have not been fully implemented, although in some cases FAA has taken steps towards addressing them. www.gao.gov/cgi-bin/getrpt?GAO-06-266T. To view the full product, including the scope and methodology, click on the link above. For more information, contact Dr. Gerald Dillingham at (202) 512-2834 or dillinghamg@gao.gov. [End of section] Mr. Chairman and Members of the Subcommittee: The U.S. commercial aviation industry, with less than one fatal accident per 5 million flights from 2002 through 2005, has an extraordinary safety record. This record is a result of the efforts of the Federal Aviation Administration (FAA), airlines, manufacturers, and others in the aviation industry to maintain one of the safest aviation systems in the world. However, when passenger airlines have accidents or serious incidents, regardless of their rarity, the consequences can be tragic, as a single accident can result in hundreds of deaths. In order to maintain a high level of safety, it is critical to have a safety oversight system that is comprehensive, efficient, and effective and can provide an early warning of hazards that can lead to accidents. It is equally important to have a skilled, well-trained workforce to implement and monitor this safety oversight system. FAA's workforce of about 3,200 inspectors stationed at more than 100 field offices throughout the world is responsible for carrying out the agency's processes to certify, inspect, and enforce safety regulations for all aspects of the aviation industry, including the aircraft and its component parts, over 100 commercial airlines, about 5,000 aircraft repair stations, and hundreds of thousands of pilots and mechanics. FAA augments its inspector workforce with nearly 13,600 designated organizations and individuals (designees) that conduct the more routine aspects of industry oversight, such as administering tests to pilots and mechanics as part of their certification requirements, and augments the safety information that it obtains from inspections with industry partnership programs. Keys to making this safety oversight system work are to: (1) establish programs that focus resources on areas of highest safety risk and on mitigating risks; (2) provide training and other communication to ensure that inspectors maintain the skills and knowledge to consistently carry out the agency's oversight programs; and (3) have processes and data to continuously monitor, evaluate, and improve the numerous oversight programs that make up the safety oversight system. This statement focuses on these three key areas of FAA's "early warning system" and is based on our recent reports on FAA's inspection oversight programs, industry partnership programs, enforcement program, and training program. We will also discuss our recommendations that FAA has not fully addressed in these areas. In summary: * FAA's aviation safety oversight system includes programs that focus on risk identification and mitigation through a system safety approach, the leveraging of resources, and enforcement of safety regulations, but the benefits of these programs are not being fully realized. FAA's system safety approach has many strengths, including the addition of a program that emphasizes risk identification to its traditional inspection program for overseeing non-legacy airlines,[Footnote 1] which is not based on risk. However, it is likely that the benefits of this approach could be enhanced if the inspection workload for non- legacy airlines was not still heavily oriented to the traditional inspection program's non-risk based activities. FAA leverages resources through its designee programs, in which designees perform about 90 percent of certification-related activities, thus allowing FAA to better concentrate its limited staff resources on the most safety- critical functions. However, concerns about the consistency and adequacy of designee oversight that FAA field offices provide have been raised by experts and other individuals we interviewed. FAA also leverages its resources through industry partnership programs, which are designed to assist the agency in receiving safety information. For example, FAA encourages voluntary reports of safety violations by responding to them by issuing a warning letter rather than a fine or other legal sanction. FAA's enforcement program, which is an outgrowth of its inspection process, is intended to ensure industry compliance with safety regulations and is another important element of its safety oversight system. FAA's policy for assessing legal sanctions against entities or individuals that do not comply with aviation safety regulations is intended to deter future violations. However, we found that recommendations for sanctions are sometimes reduced on the basis of factors that are not associated with the merits of the case, and the economic literature on deterrence suggests that the goal of preventing future violations is weakened when the penalties for violations are lowered for reasons not related to the merits of the case. * FAA has made training an integral part of its safety oversight system, but several actions could improve the results of its training efforts. FAA's use of a risk-based system safety approach to inspections requires inspectors to apply data analysis and auditing skills to identify and control potential risks. Therefore, it is important that inspectors are well-trained in this approach and have sufficient knowledge of increasingly complex aircraft and systems to effectively identify safety risks. FAA has established mandatory training requirements for its workforce as well as designees. We have reported that FAA has generally followed effective management practices for planning, developing, delivering, and assessing the impact of its technical training for safety inspectors, although some practices have yet to be fully implemented. For example, in developing its training curriculum for inspectors, FAA followed effective management practices, such as developing courses that support changes in inspection procedures resulting from regulatory changes or agency initiatives. On the other hand, FAA develops technical courses on an ad hoc basis rather than as part of an overall curriculum for each type of inspector, such as inspectors of operations or cabin safety, because the agency has not systematically identified the technical skills and competencies each type of inspector needs to effectively perform inspections. FAA has recognized the need for improvements to its training program in this and other areas. * It is important for FAA to have effective evaluative processes and accurate nationwide data for its numerous safety oversight programs so that program managers and other officials have assurance that the safety programs are having their intended effect. Such processes and data are especially important because FAA's workforce is so dispersed throughout the world--with thousands of staff working out of more than 100 offices worldwide--and because FAA's use of a system safety approach represents a cultural shift from its traditional inspection program. Evaluation is important for understanding if the cultural shift has effectively occurred. Our most recent work has shown the lack of evaluative processes and limitations with data for FAA's inspection program for non-legacy airlines, designee programs, industry partnership programs, and enforcement program. For example, we found that FAA lacked requirements or criteria for evaluating its designee programs. In another example, FAA's enforcement policy calls for the assessment of sanctions that would potentially deter future violations. However, FAA lacks an evaluative process, so it is not known whether the agency's enforcement practices, such as at times reducing sanctions, may weaken any deterrent effect that would be expected from such sanctions. Furthermore, FAA's ability to evaluate its programs is hindered by the lack of useful nationwide data. For example, FAA's nationwide enforcement database is not as useful as it could be because of missing or incomplete historical information about enforcement cases. * In order to help FAA fully realize the benefits of its safety oversight system, we have made a number of recommendations to address the weaknesses that we identified in our reviews. These recommendations have not been fully implemented, although in some cases FAA has taken steps towards addressing them. Evaluative processes and relevant data are particularly important as FAA works to change its culture by incorporating a system safety approach into its oversight, and we have recommended that FAA develop continuous evaluative processes for its oversight of non-legacy airlines, its designee programs, and its enforcement program, and systematically assess inspectors' technical training needs. In addition, FAA's nationwide databases are in need of improvements in their comprehensiveness and ease of use. We have recommended that FAA improve the consistency and completeness of its designee and enforcement databases. Continuous improvements in these areas are critical to FAA's ability to have a robust "early warning system" in order to maintain one of the safest aviation systems in the world. Background: FAA's safety oversight system is made up of a number of programs for airlines and other entities. Safety oversight programs for airlines provide for their initial certification, periodic surveillance, and inspection. Since 1985, FAA has used National Work Program Guidelines (NPG), its traditional inspection program for airlines, as a primary means of ensuring that airlines comply with safety regulations. In NPG, an FAA committee of program managers identifies an annual minimum set of required inspections that are to be undertaken to ensure that airlines are in compliance with their operating certificates. In 1998, the agency implemented the Air Transportation Oversight System (ATOS), which currently oversees the nation's largest 15 commercial airlines and cargo carriers, with the goal of eventually including all commercial passenger and cargo airlines in it. ATOS emphasizes a system safety approach that extends beyond periodically checking airlines for compliance with regulations to the use of technical and managerial skills to identify, analyze, and control hazards and risks. For example, under ATOS, inspectors develop surveillance plans for each airline, based on data analysis and assessment of risks, and adjust the plans periodically based on inspection results. However, the agency has been delayed in placing a significant number of other passenger airlines in ATOS, resulting in 99 passenger airlines, which we refer to as non-legacy airlines, continuing to be overseen through NPG, a process that is not risk-based or system safety oriented. In 2002, FAA added the Surveillance and Evaluation Program (SEP) to the NPG inspection program to incorporate principles of ATOS into its oversight of non-legacy passenger airlines. The two programs are used together to establish the number of annual inspections for non-legacy airlines. (Appendix 1 describes each inspection program.) Figure 1 illustrates some typical activities covered during inspections. Figure 1: FAA's Safety Inspections Cover a Wide Range of Activities: [See PDF for image] Note: As a workforce, FAA inspectors conduct a wide variety of inspections, including ensuring that pilots are qualified to operate aircraft, inspecting aircraft for safety, and overseeing FAA-certified mechanics. [End of figure] FAA's safety oversight programs for other aspects of the aviation industry--including manufacturers of aircraft and aircraft parts, repair stations, flight schools, aviation maintenance technician schools, pilots, and mechanics--involve certification, surveillance, and inspection by FAA's safety inspectors, engineers, flight surgeons, and designated representatives. FAA authorizes about 13,400 private individuals and about 180 organizations (called "designees") to act as its representatives to conduct many safety certification activities, such as administering flight tests to pilots, inspecting repair work by maintenance facilities, conducting medical examinations of pilots, and approving designs for aircraft parts. These designees are grouped into 18 different programs and are overseen by three FAA offices--Flight Standards Service, Aerospace Medicine, and Aircraft Certification Service--all of which are under the Office of Aviation Safety (see fig. 2). Figure 2: FAA Offices That Manage the Different Designee Programs and Numbers of Designees (as of May 2004): [See PDF for image] [End of figure] Since 1990, FAA has emphasized gaining compliance from the aviation industry through cooperative means by establishing industry partnership programs with the aviation community that allow participants, such as airlines and pilots, to self-report violations of safety regulations and help identify safety deficiencies, and potentially mitigate or avoid fines or other legal action. For example, the Voluntary Disclosure Program encourages the self-reporting of manufacturing problems and safety incidents by participants that can include air carriers and repair stations. Appendix II describes the industry partnership programs. When violations of statutory and regulatory requirements are identified through inspections, through the partnership programs in certain cases, or through other methods, FAA has a variety of enforcement tools that it may use to respond to them, including administrative actions (such as issuing a warning notice or a letter of correction that includes the corrective actions the violator will take) and legal sanctions (such as levying a fine or suspending or revoking a pilot or other FAA-issued certificate). FAA's Safety Oversight System Focuses on Risk Identification and Mitigation Through System Safety, Leveraging of Resources, and Enforcement of Safety Regulations, but Benefits Are Not Being Fully Realized: In recent reports, we found that FAA's safety oversight system has programs that focus on risk identification and mitigation through a system safety approach, the leveraging of resources, and enforcement of safety regulations, but that the benefits of these programs are not being fully realized. In our recent report on FAA's oversight of non- legacy airlines, we found that the focus on risk identification through the addition of SEP has many strengths and allows for enhancing the efficiency of FAA's oversight activities.[Footnote 2] Rather than relying on NPG's customary method of conducting a set number of inspections of an airline's operations, SEP emphasizes a system safety approach of using risk analysis techniques. SEP allows for the efficient use of inspection staff and resources by prioritizing workload based on areas of highest risk, and it includes a requirement that inspectors verify that corrective actions have occurred. For example, FAA has developed risk assessment worksheets for SEP that are aligned with key airline systems that guide inspectors through identifying and prioritizing risks. The worksheets guide inspectors to organize the results of their previous inspections and surveillance into a number of areas such as flight operations and personnel training in order to identify specific risks in each area and target the office's resources to mitigating those risks. The development of a system safety approach addresses a long-standing concern by us that FAA did not have a methodology for assessing airline safety risks so that it could target limited inspection resources to high-risk conditions.[Footnote 3] Another strength of SEP, consistent with findings in our past reports, is that SEP relies on teams of inspectors, which are generally more effective than individual inspectors in their ability to collectively identify concerns.[Footnote 4] However, the benefits of FAA's system safety approach for the inspection of non-legacy airlines could be enhanced by a more complete implementation of SEP and addressing other challenges. The inspection workload for non-legacy airlines is still heavily oriented to the NPG's non-risk based activities. For example, as shown in table 1, from fiscal years 2002 through 2004, 77 percent of inspection activities required for the top 25 non-legacy airlines in terms of the number of enplanements were identified through NPG, and the remaining percentage of inspection activities were identified based on risk through SEP. Although inspectors can replace NPG-identified activities with SEP- identified activities that they deem constitute a greater safety risk, we found that FAA inspectors interpret agency emphasis on NPG as discouraging this practice. In order to ensure that all inspectors who oversee non-legacy airlines have a complete and timely understanding of the agency's policies relating to the inspection process, we recommended in September 2005 that FAA improve communication with and training of inspectors in this area. Table 1: SEP-and NPG-Initiated Required Inspections for the Top 25 Non- legacy Airlines, Fiscal Years 2002-2004: Type of inspection: SEP-initiated; 2002: 1,261; 2003: 1,567; 2004: 927; Total: 3,755 (23%). Type of inspection: NPG-initiated; 2002: 5,470; 2003: 3,623; 2004: 3,338; Total: 12,431 (77%). Type of inspection: Total; 2002: 6,731; 2003: 5,190; 2004: 4,265; Total: 16,186 (100%). Source: GAO analysis of FAA information. [End of table] Another way that FAA attempts to enhance the efficiency of its oversight activities is through its designee programs. We reported that FAA maximizes its resources by allowing designees to perform about 90 percent of certification-related activities, thus allowing FAA to better concentrate its limited staff resources on the most safety- critical functions.[Footnote 5] For example, while designees conduct routine certification functions, such as approvals of aircraft technologies that the agency and designees have had previous experience with, FAA focuses on new and complex aircraft designs or design changes. In addition, the use of designees expands FAA's access to technical expertise within the aviation community. For the aviation industry, the designee programs enable individuals and organizations to obtain required FAA certifications--such as approvals of the design, production, and airworthiness of aircraft--in a timely manner, thus reducing delays and costs to the industry that might result from scheduling direct reviews by FAA staff. For example, officials from an aircraft manufacturer told us that the use of designees has added significantly to the company's ability to enhance and improve daily operations by decreasing certification delivery time and increasing the flexibility and utilization of company resources. In addition, designees are convenient to the aviation industry due to their wide dispersal throughout the United States. However, concerns about the consistency and adequacy of designee oversight that FAA field offices provide have been raised by experts and other individuals we interviewed. For example, designees and industry officials that we spoke with indicated that FAA's level of oversight and interpretation of rules differ among regions and among offices within a region, which limits FAA's assurance that designees' work is performed uniformly in accordance with FAA's standards and policy. Experts also ranked this issue as a top weakness.[Footnote 6] Table 2 shows the top five weaknesses identified by our experts. Experts also made a number of suggestions to strengthen the designee program, including clearly defining and following agency criteria for selecting designees and increasing penalties for designees found to violate standards or who do not exercise proper judgment. To improve management control of the designee programs, and thus increase assurance that designees meet FAA's performance standards, we recommended that FAA develop mechanisms to improve the compliance of FAA program and field offices with existing policies and incorporate, as appropriate, suggestions from our expert panel. In response to our recommendations, FAA is planning, among other things, to form a team to identify and share best practices for overseeing designee programs. Table 2: Experts' Ranking of Top 5 Oversight Weaknesses: Ranking: 1; Weakness: FAA offices level of oversight and interpretation of rules are inconsistent. Ranking: 2; Weakness: Inactive, unqualified, or poor performing designees are not identified and removed expeditiously. Ranking: 3; Weakness: It is difficult to terminate poor performing designees. Ranking: 4; Weakness: Inadequate surveillance and oversight of designees. Ranking: 5; Weakness: FAA has not made oversight of designees a high enough priority. Source: GAO analysis of expert panel information. Note: Rankings based on responses from 62 experts and the frequency of responses indicating a "great" or "very great" weakness. [End of table] FAA also leverages its resources through its industry partnership programs. These partnership programs are designed to assist the agency in receiving safety information, including reports of safety violations. According to FAA officials, the Aviation Safety Action Program, Aviation Safety Reporting Program, and Voluntary Disclosure Reporting Program augment FAA's enforcement activities and allow FAA to be aware of many more safety incidents than are discovered during inspections and surveillance. In addition, the Flight Operational Quality Assurance Program provides safety information in the form of recorded flight data from participating airlines. FAA has established some management controls over its partnership programs, such as procedures to track actions taken to correct safety incidents reported under the programs, but the agency lacks management controls to measure and evaluate the performance of these programs, an issue that we will discuss later in the testimony. FAA's enforcement process, which is intended to ensure industry compliance with safety regulations, is another important element of its safety oversight system. FAA's policy for assessing legal sanctions against entities or individuals that do not comply with aviation safety regulations is intended to deter future violations. FAA has established some management controls over its enforcement efforts, with procedures that provide guidance on identifying regulated entities and individuals that are subject to inspections or surveillance actions, determining workload priorities on the basis of the timing and type of inspection to be performed, detecting violations of safety regulations, tracking the actions that are taken by the entities and individuals to correct the violations and achieve compliance with regulations, and imposing punitive sanctions or remedial conditions on the violators. These procedures provide FAA inspectors, managers, and attorneys with a process to handle violations of safety regulations that are found during routine inspections. However, we found that the effect of FAA's legal sanctions on deterrence is unclear, and that recommendations for sanctions are sometimes changed on the basis of factors that are not associated with the merits of the case. We found that from fiscal years 1993 through 2003, attorneys in FAA's Office of the Chief Counsel authorized a 52 percent reduction in the civil monetary penalties assessed from a total of $334 million to $162 million. FAA officials told us that the agency sometimes reduces sanctions in order to prioritize attorneys' caseloads by closing the cases more quickly through negotiating a lower fine. Economic literature on deterrence suggests that although negative sanctions (such as fines and certificate suspensions) can deter violations, if the violator expects sanctions to be reduced, he or she may have less incentive to comply with regulations. In effect, the goal of preventing future violations is weakened when the penalties for present violations are lowered for reasons not related to the merits of the case. In addition, FAA lacks management controls to measure and evaluate its enforcement process, which we discuss later in this testimony. FAA Has Made Training an Integral Part of Its Safety Oversight System but Several Actions Could Improve Results: FAA's use of a risk-based system safety approach to inspections requires inspectors to apply data analysis and auditing skills to identify, analyze, assess, and control potential hazards and risks. Therefore, it is important that inspectors are well-trained in this approach and have sufficient knowledge of increasingly complex aircraft, aircraft parts, and systems to effectively identify safety risks. It is also important that FAA's large cadre of designees is well- trained in federal aviation regulations and FAA policies. FAA has made training an integral part of its safety inspection system and has established mandatory training requirements for its workforce as well as designees. FAA provides inspectors with extensive training in federal aviation regulations; inspection and investigative techniques; and technical skills, such as flight training for operations inspectors. The agency provides its designees with an initial indoctrination that covers federal regulations and agency policies, and refresher training every 2 to 3 years. We have reported that FAA has generally followed effective management practices for planning, developing, delivering, and assessing the impact of its technical training[Footnote 7] for safety inspectors, although some practices have yet to be fully implemented.[Footnote 8] In its planning activities for training, FAA has linked technical training efforts to its goal of safer air travel and has identified technical proficiencies needed to improve safety inspectors' performance in meeting this goal. For example, FAA's Offices of Flight Standards and Aircraft Certification have identified gaps in several of the competencies required to conduct system safety inspections, including risk assessment, data analysis, systems thinking, and designee oversight. According to FAA, it is working to correct these gaps. We have also identified gaps in the training provided to inspectors in the Office of Flight Standards who oversee non-legacy airlines, and have recommended that FAA improve inspectors' training in areas such as system safety and risk management to ensure that these inspectors have a complete and timely understanding of FAA's inspection policies. We have identified similar competency gaps related to designee oversight. For example, FAA does not require refresher training concerning designee oversight, which increases the risk that staff do not retain the information, skills, and competencies required to perform their oversight responsibilities. We recommended that FAA provide additional training for staff who directly oversee designees.[Footnote 9] We did not identify any specific gaps in the competencies of designees.[Footnote 10] In prioritizing funding for course development activities, FAA does not explicitly consider which projects are most critical. Figure 3 describes the extent to which FAA follows effective management practices in planning training. Figure 3: Extent that FAA Follows Effective Management Practices in Planning Technical Training: [See PDF for image] [End of figure] In developing its training curriculum for inspectors, FAA also for the most part follows effective management practices, such as developing courses that support changes in inspection procedures resulting from regulatory changes or agency initiatives. On the other hand, FAA develops technical courses on an ad hoc basis rather than as part of an overall curriculum for each inspector specialty--such as air carrier operations, maintenance, and cabin safety--because the agency has not systematically identified the technical skills and competencies each type of inspector needs to effectively perform inspections. Figure 4 describes the extent to which FAA follows effective management practices in developing training. Figure 4: Extent that FAA Follows Effective Management Practices in Developing Technical Training: [See PDF for image] [A] This management practice is not specifically identified in our assessment guide. However, a management approach that assesses training needs holistically rather than on a course-by-course basis can provide for a more systematic assessment of whether and how training will help meet organizational needs. [End of figure] In delivering training, FAA has also generally followed effective management practices. (See fig. 5.) For example, FAA has established clear accountability for ensuring that inspectors have access to technical training, developed a way for inspectors to choose courses that meet job needs and further professional development, and offers a wide array of technical and other courses. However, both FAA and its inspectors recognize the need for more timely selection of inspectors for technical training. In addition, FAA acknowledges the need to increase communication between inspectors and management with respect to the training program, especially to ensure that inspectors have bought into the system safety approach to inspections. Figure 5: Extent that FAA Follows Effective Management Practices in Delivering Technical Training: [See PDF for image] [End of figure] FAA offers numerous technical courses from which inspectors can select to meet job needs. However, from our survey of FAA's inspectors, we estimate that only about half think that they have the technical knowledge needed for their jobs.[Footnote 11] FAA officials told us that inspectors' negative views stem from their wanting to acquire proficiencies that are not as crucial in a system safety environment. We also found a disparity between inspectors and FAA concerning the receipt of requested training. We estimated that 28 percent of inspectors believe that they get the technical training that they request. However, FAA's records show that FAA approves about 90 percent of these requests, and inspectors are making good progress in receiving training. Over half of the inspectors have completed at least 75 percent of technical training that FAA considers essential. FAA officials told us that inspectors' negative views on their technical knowledge and the training they have received stem from their not accepting FAA's move to a system safety approach. That is, the inspectors are concerned about acquiring individual technical proficiency that is not as crucial in a system safety environment. Given that it has not completed assessing whether training for each inspector specialty meets performance requirements, FAA is not in a position to make definitive conclusions concerning the adequacy of inspector technical training. FAA also generally followed effective management practices in evaluating training. The agency requires that each training course receive a systematic evaluation every 3 years to determine if the course is up to date and relevant to inspectors' jobs, although training officials noted that many courses have yet to undergo such an evaluation. However, FAA collects limited information on the effectiveness of training, and its evaluations have not measured the impact of training on FAA's mission goals, such as reducing accidents. Training experts acknowledge that isolating performance improvements resulting from training programs is difficult for any organization. (See fig. 6.) Figure 6: Extent that FAA Follows Effective Management Practices in Evaluating Its Training Program: [See PDF for image] [End of figure] While FAA follows many effective management practices in its training program, the agency also recognizes the need for improvements, including (1) systematically assessing inspectors' needs for technical and other training, (2) better timing of technical training so that inspectors receive it when it is needed to perform their jobs, and (3) better linking the training provided to achieving agency goals of improving aviation safety. FAA has begun to act in these areas, and we believe that if effectively implemented, the actions should improve the delivery of training and ultimately improve aviation safety. Therefore, it is important for FAA to follow through with its efforts. As a result, we recommended in September 2005, among other things, that in order to ensure that inspector technical training needs are identified and met in a timely manner, FAA systematically assess inspectors' technical training needs, better align the timeliness of training to when inspectors need the training to do their jobs, and gain inspectors' acceptance for changes made or planned to their training. It is important that both FAA's inspection workforce and FAA-certified aviation mechanics are knowledgeable about increasingly complex aircraft, aircraft parts, and systems. While we did not attempt to assess the technical proficiency that FAA's workforce requires and will require in the near future, FAA officials said that inspectors do not need a substantial amount of technical training courses because inspectors are hired with a high degree of technical knowledge of aircraft and aircraft systems. They further indicated that inspectors can sufficiently keep abreast of many of the changes in aviation technology through FAA and industry training courses and on-the-job training. However, in its certification program for aviation mechanics, we found that FAA standards for minimum requirements for aviation courses at FAA-approved aviation maintenance technician schools and its requirements for FAA-issued mechanics certificates do not keep abreast with the latest technologies. In 2003, we reported that those standards had not been updated in more than 50 years.[Footnote 12] We recommended that FAA review the curriculum and certification requirements and update both. FAA plans to make changes in the curriculum for FAA approved aviation maintenance technicians that reflect up-to-date aviation technologies and finalize and distribute a revised Advisory Circular in March 2006 that describes the curriculum changes. FAA then plans to allow the aviation industry time to implement the recommended curriculum changes before changing the requirements for FAA-issued mechanics certificates. FAA Has Evaluated Some Safety Programs, but the Lack of Evaluative Systems and Nationwide Data Impedes FAA's Ability to Continuously Monitor Its Safety Programs: It is important for FAA to have effective evaluative processes and accurate nationwide data on its numerous safety oversight programs so that program managers and other officials have assurance that the safety programs are having their intended effect. Such processes and data are especially important because FAA's workforce is so dispersed worldwide--with thousands of staff working out of more than 100 local offices--and because FAA's use of a risk-based system safety approach represents a cultural shift from its traditional inspection program. Evaluation is important to understanding if the cultural shift has effectively occurred. Our most recent work has shown the lack of such processes and limitations with data for FAA's inspection programs for non-legacy airlines, designee programs, industry partnership programs, and enforcement program. In response to recommendations that we have made regarding these programs, some improvements are being made. On the positive side, as we mentioned earlier, our most recent work found that FAA generally follows effective management practices in evaluating individual technical training courses. FAA has not evaluated its inspection oversight programs for non-legacy airlines--which include SEP and NPG--to determine how the programs contribute to the agency's mission and overall safety goals, and its nationwide inspection database lacks important information that could help it perform such evaluations--such as whether risks identified through SEP have been mitigated. In addition, the agency does not have a process to examine the nationwide implications of or trends in the risks that inspectors have identified through their risk assessments-- information it would need to proactively determine risk trends at the national level on a continuous basis. FAA's evaluation office instead conducts analyses of the types of inspections generated under SEP by airline and FAA region, according to FAA. We recommended that FAA develop a continuous evaluative process for activities under SEP and link SEP to the performance-related goals and measures developed by the agency, track performance toward these goals, and determine appropriate program changes. FAA is considering our recommendation, but its plan to place the remaining non-legacy airlines in the ATOS program by the end of fiscal year 2007 might make this recommendation unnecessary, according to the agency. Since FAA's past efforts to move airlines to ATOS have experienced delays, we believe that this recommendation is still valid. We also found that FAA lacked requirements or criteria for periodically evaluating its designee programs. In 2004, we reported that the agency had evaluated 6 of its 18 designee programs over the previous 7 years and had plans to evaluate 2 more, although it had no plans to evaluate the remaining 10 programs because of limited resources.[Footnote 13] FAA conducted these evaluations on an ad hoc basis usually at the request of headquarters directors or regional office managers. In addition, we found that FAA's oversight of designees is hampered, in part, by the limited information on designees' performance contained in the various designee databases.[Footnote 14] These databases contain descriptive information on designees, such as their types of designations and status (i.e., active or terminated). More complete information would allow the agency to gain a comprehensive picture of whether staff are carrying out their responsibilities to oversee designees. To improve management control of the designee programs, and thus increase assurance that designees meet the agency's performance standards, we recommended that FAA establish a process to evaluate all designee programs and strengthen the effectiveness of its designee databases by improving the consistency and completeness of information in them. To address our recommendations, FAA expects to develop a plan to evaluate all designee programs on a recurring basis and intends to establish a team that will examine ways to improve automated information related to designees. In addition, we found that FAA does not evaluate the effects of its industry partnership and enforcement programs to determine if stated program goals, such as deterrence of future violations, are being achieved. For example, little is known about nationwide trends in the types of violations reported under the partnership programs or whether systemic, nationwide causes of those violations are identified and addressed. Furthermore, FAA's enforcement policy calls for inspectors and legal counsel staff to recommend or assess enforcement sanctions that would potentially deter future violations. However, without an evaluative process, it is not known whether the agency's practice of generally closing cases with administrative actions rather than legal sanctions[Footnote 15] and at times reducing the amount of the fines, as mentioned earlier in this testimony, may weaken any deterrent effect that would be expected from sanctions. FAA's ability to evaluate the impact of its enforcement efforts is also hindered by the lack of useful nationwide data. FAA inspection offices maintain independent, site-specific databases because they do not find the nationwide enforcement database--the Enforcement Information System (EIS)--as useful as it could be because of missing or incomplete historical information about enforcement cases. As a result of incomplete data on individual cases, FAA inspectors lack the complete compliance history of violators when assessing sanctions. We recommended that FAA develop evaluative processes for its enforcement activities and partnership programs and use them to create performance goals, track performance towards those goals, and determine appropriate program changes. We also recommended that FAA take steps to improve the usefulness of the EIS database by enhancing the completeness of enforcement information. FAA expects to address some of these issues as it revises its enforcement policy, which is expected to be issued later in fiscal year 2006. In addition, FAA has established a database workgroup that is developing long-and short-term solutions to address the problems with EIS. Recommendations We Have Made to Improve FAA's Safety Oversight System: In order to help FAA fully realize the benefits from its safety oversight system, we have made a number of recommendations to address weaknesses that we identified in our reviews. These recommendations have not been fully implemented, although in some cases FAA has taken steps towards addressing them. Evaluative processes and relevant data are particularly important as FAA works to change its culture by incorporating a system safety approach into its oversight, and we have recommended that FAA develop continuous evaluative processes for its oversight programs for non-legacy airlines, its designee programs, and its industry partnership and enforcement programs, and systematically assess inspectors' technical training needs. In addition, FAA's nationwide databases are in need of improvements in their comprehensiveness and ease of use. Without comprehensive nationwide data, FAA does not have the information needed to evaluate its safety programs and have assurance that they are having the intended results. We have recommended that FAA improve the completeness of its designee and enforcement databases. Continuous improvements in these areas are critical to FAA's ability to have a robust "early warning system" and maintain one of the safest aviation systems in the world. Contacts and Acknowledgments: For further information on this testimony, please contact Dr. Gerald Dillingham at (202) 512-2834 or by email at dillinghamg@gao.gov. Individuals making key contributions to this testimony include Brad Dubbs, Phillis Riley, Teresa Spisak, and Alwynne Wilbur. [End of section] Appendix I: Description of FAA's Inspection Programs: Table 3 describes the Federal Aviation Administration's (FAA) three inspection processes for overseeing airlines: Air Transportation Oversight System (ATOS), National Work Program Guidelines (NPG), and Surveillance and Evaluation Program (SEP). Many of the elements of ATOS, such as the use of data to identify risks and the development of surveillance plans by inspectors, are incorporated in the SEP process. The NPG process, in contrast, is not focused on the use of data and relies on an established set of inspections that are not risk based. Table 3: Various Elements of ATOS, NPG, and SEP: Description of program; ATOS: * Focuses on safety vulnerabilities rather than regulatory compliance; * Analysts and inspectors review airline data to identify areas of safety risk; * Inspectors develop surveillance plans for each airline, based on data analysis and assessment of risks, and adjust the plans periodically based on inspection results; NPG: * Focuses on inspectors completing a prescribed number of inspection activities; * Primarily based on checking airline compliance with regulations; * Relies on inspectors' expertise to identify trends and risks; SEP: * Focuses on inspectors conducting a risk assessment of various areas; * Inspectors review data to identify areas of safety risk and use Flight Standards Safety Analysis Information Center and the Safety Performance Analysis System as analytical tools; * Inspectors develop surveillance plans for each airline, based on data analysis and assessment of risks, and adjust plans periodically based on inspection results; * Inspectors can also verify that planned NPG activities meet the surveillance needs for a particular year. Type of commercial passenger airline inspected; ATOS: Legacy commercial airlines; NPG: Non-legacy commercial airlines; SEP: Non-legacy commercial airlines. Frequency of inspections; ATOS: Continuous safety oversight; NPG: Periodic; regular inspections are established annually by an FAA headquarters committee; SEP: Periodic; inspections are established during meetings held at least twice a year using risk-based criteria. Approximate number of aviation safety inspectors conducting inspections[A]; ATOS: 585; NPG: 1,100[B]; SEP: 1,100[B]. Number of commercial passenger airlines under the program[B]; ATOS: 13[C]; NPG: 99; SEP: 99. Source: GAO and FAA: [A] As of July 2005. [B] There are a total of about 1,100 inspectors for both the NPG and SEP inspections. [C] FedEx and United Parcel Service, two cargo air carriers, are also in the ATOS program. This number reflects the recent merger of US Airways and America West, which were both in ATOS prior to the merger. [End of table] [End of section] Appendix II: Description of FAA's Partnership Programs: Aviation Safety Action Program (ASAP): Year Established: 1997: Participation: Participants include employees of air carriers and repair stations that have entered into a memorandum of understanding with the Federal Aviation Administration (FAA). The memoranda can cover employee groups, such as pilots, maintenance employees, dispatchers, or flight attendants. Each employee group is covered by a separate memorandum of understanding. As of June 2004, FAA had accepted 54 memoranda of understanding and received over 80,000 ASAP reports, which may or may not include safety violations, according to FAA officials. Purpose: ASAP seeks to improve aviation safety through the voluntary self-reporting of safety incidents under the procedures set forth in the memorandum of understanding. Under the program, FAA does not take enforcement action against employees who voluntarily self-reported safety violations for reports that are sole-source (the report is the only way FAA would have learned about the incident) and will pursue administrative action only for reports that are not sole-source. Incidents that involve alcohol, drugs, criminal activity, or an intentional disregard for safety are not eligible for self-reporting under ASAP. Process: Each memorandum of understanding is a voluntary partnership between FAA, the airline, and an employee group. Although employee groups are not always included, FAA encourages their participation. The memorandum of understanding ensures that employees who voluntarily disclose FAA safety violations in accordance with the procedures and guidelines of ASAP will receive administrative action or no action in lieu of legal enforcement action. Once a memorandum of understanding is approved, employees can begin reporting violations that fall under the agreement. When a violation occurs, an employee notifies the Event Review Committee, which includes representatives from FAA and the airline or the repair station and generally includes the appropriate employee association. The committee must be notified in writing within the time limit specified in the memorandum of understanding. The committee then determines whether to accept the report under the ASAP program. If the report is accepted (it meets the acceptance criteria in the memorandum and does not involve criminal activity, substance abuse, controlled substances, or alcohol), then the committee determines the action to take. That action may include remedial training or administrative action, but it will not include a legal sanction. Results: FAA does not know the overall program results because it does not have a national, systematic process in place to evaluate the overall success of ASAP. However, FAA cites examples that describe ASAP's contribution to enhanced aviation safety. These examples include identifying deficiencies in aircraft operations manuals, airport equipment, and runways. In July 2003, FAA's Compliance and Enforcement Review recommended that FAA evaluate the use and effectiveness of this program. Aviation Safety Reporting Program (ASRP): Year Established: 1975: Participation: Participants are all users of the national airspace system, including air traffic controllers and employees of air carriers and repair stations. Purpose: The program is designed to improve aviation safety by offering limited immunity for individuals who voluntarily report safety incidents. ASRP was founded after TWA Flight 514 crashed on approach to landing in December 1974 after the crew misinterpreted information on the approach chart. This accident occurred only 6 weeks after another plane experienced the same error. Process: The National Aeronautics and Space Administration (NASA) administers this program. When a safety incident occurs, a person may submit a form and incident report to NASA. There are four types of forms that can be submitted to NASA: (1) Air Traffic Control, (2) General Reports (includes Pilots), (3) Flight Attendants, and (4) Maintenance Personnel. At least two aviation safety analysts read these forms and the incident reports that accompany them. The analysts at NASA screen the incident reports for urgent safety issues, which will be marked for immediate action to the appropriate FAA office or aviation authority. NASA analysts also edit the report's narrative to eliminate any identifying information. In addition, each report has a tear-off portion, which is separated and returned to the individual who reported the incident as a receipt of the incident report's acceptance into the ASRP. When a safety violation that has been previously reported under ASRP comes to the attention of FAA, the agency issues a legal sanction, which is then waived. Reports that would not be eligible to have a legal sanction waived include deliberate violations, violations involving a criminal offense, or accident; reports filed by participants who have committed a violation of federal aviation regulations or law within the last 5 years and reports filed later than 10 days following an incident. Results: While FAA and NASA do not know the overall program results because they do not have a formal national evaluation program to measure the overall effectiveness of the program, the agencies widely disseminate information generated from the program to aircraft manufacturers and others. ASRP reports are compiled into a database known as the Aviation Safety Reporting System. When a potentially hazardous condition is reported, such as a defect in a navigational aid or a confusing procedure, NASA will send a safety alert to aircraft manufacturers, the FAA, airport representatives, and other aviation groups. The database is used for a monthly safety bulletin that includes excerpts from incident reports with supporting commentary by FAA safety experts. NASA officials estimate that the bulletin is read by over 150,000 people. In addition, individuals and organizations can request a search of the database for information on particular aircraft aviation safety subjects, including human performance errors and safety deficiencies. Further, NASA has used the database to analyze operational safety issues, such as general aviation incidents, pilot and controller communications, and runway incursions. Flight Operational Quality Assurance (FOQA): Year Established: 1995: Participation: Participants include air carriers that equip their airplanes to record flight data. As of March 2004, 13 airlines had FAA- approved FOQA programs, and approximately 1,400 airplanes were equipped for the program. Purpose: FOQA is designed to enhance aviation safety through the analysis of digital flight data generated during routine flights. Process: Air carriers that participate in the program equip their aircraft with special acquisition devices or use the airplanes' flight data recorders to collect data and determine if the aircraft are deviating from standard procedures. These data include engine temperatures, descent rate, and deviations from the flight path. When the aircraft lands, data are transmitted from the aircraft to the airline's FOQA station, where they are analyzed for flight trends and possible safety problems. Once the data are transmitted to the FOQA ground station, the data are extracted and analyzed by software programs. The FOQA data are combined with data from maintenance databases, weather conditions, and other safety reporting systems, such as ASAP, in order to identify trends in flight operations. The analysis typically focuses on events that fall outside normal boundaries specified by the manufacturer's operational limitations and the air carrier's operational standards. FOQA data are collected and analyzed by individual air carriers. The data on safety trends are made available to FAA in an aggregated form with no identification of individual carriers. According to FAA officials, air carriers do not want to release this data to any outside party (including FAA) because of concerns that the data could then be publicly released. Air carriers pay for the special flight data recorders that can record FOQA data, which cost approximately $20,000 each. Although this can be an expensive investment for some air carriers, most newer aircraft models come with the data recorder built into the airplane. The International Civil Aviation Organization (ICAO) has recommended that airlines from member countries implement a FOQA program. FAA has notified ICAO that the program will remain voluntary in the United States. Results: Although FAA has no formal national evaluation program to measure the overall results or effectiveness of FOQA programs, FAA cites examples that describe FOQA's contribution to enhanced aviation safety. For example, one FOQA program highlighted a high rate of descent when airplanes land at a particular airport. On the basis of the information provided from FOQA, air traffic controllers at the airport were able to develop alternative approach procedures to decrease the rate of descent. Voluntary Disclosure Reporting Program (VDRP): Year Established: 1990: Participation: Participants include air carriers, repair stations, and production approval holders.[Footnote 16] Purpose: FAA initiated the program to promote aviation safety by encouraging the voluntary self-reporting of manufacturing, and quality control problems and safety incidents involving FAA requirements for maintenance, flight operations, drug and alcohol prevention programs, and security functions. Process: Upon discovering a safety violation, participants can voluntarily disclose the violation to FAA within 24 hours. The initial notification should include a description of the violation, how and when the violation was discovered, and the corrective steps necessary to prevent repeat violations. Within 10 days of filing the initial notification to FAA, the entity is required to provide a written report that cites the regulations violated, describes how the violation was detected, provides an explanation of how the violation was inadvertent, and provides a description of the proposed comprehensive fix. FAA may pursue legal action if the participant discloses violations during, or in anticipation of, an FAA inspection. The violation must be reported immediately after being detected, must be inadvertent, must not indicate that a certificate holder is unqualified, and must include the immediate steps that were taken to terminate the apparent violation. If these conditions are met, and the FAA inspector has approved the comprehensive fix, then the FAA inspector will prepare a letter of correction and the case is considered closed with the possibility of being reopened if the comprehensive fix is not completed. Results: FAA does not know the overall program results because it does not have a process to measure the overall effectiveness of the program nationwide. A 2003 internal FAA report recommended that the agency evaluate the use and effectiveness of this program. [End of section] Selected GAO Reports on Aviation Safety: Aviation Safety: System Safety Approach Needs Further Integration into FAA's Oversight of Airlines. GAO-05-726. Washington, D.C.: September 28, 2005. Aviation Safety: FAA Management Practices for Technical Training Mostly Effective; Further Actions Could Enhance Results. GAO-05-728. Washington, D.C.: September 7, 2005. Aviation Safety: Oversight of Foreign Code-Share Safety Program Should Be Strengthened. GAO-05-930. Washington, D.C.: August 5, 2005. Aviation Safety: FAA Needs to Strengthen the Management of Its Designee Programs. GAO-05-40. Washington, D.C.: October 8, 2004. Aviation Safety: Better Management Controls are Needed to Improve FAA's Safety Enforcement and Compliance Efforts. GAO-04-646. Washington, D.C.: July 6, 2004. Aviation Safety: Information on FAA's Data on Operational Errors at Air Traffic Control Towers. GAO-03-1175R. Washington, D.C.: September 23, 2003. Aviation Safety: FAA Needs to Update the Curriculum and Certification Requirements for Aviation Mechanics. GAO-03-317. Washington, D.C.: March 6, 2003. Aviation Safety: FAA and DOD Response to Similar Safety Concerns. GAO- 02-77. Washington. D.C.: January 22, 2002. Aviation Safety: Safer Skies Initiative Has Taken Initial Steps to Reduce Accident Rates by 2007. GAO/RCED-00-111. Washington, D.C.: June 30, 2000. Aviation Safety: FAA's New Inspection System Offers Promise, but Problems Need to Be Addressed. GAO/RCED-99-183. Washington, D.C.: June 28, 1999. Aviation Safety: Weaknesses in Inspection and Enforcement Limit FAA in Identifying and Responding to Risks. GAO/RCED-98-6. Washington, D.C.: February 27, 1998. FOOTNOTES [1] We refer to all passenger airlines that are not in FAA's Air Transportation Oversight System (ATOS) as non-legacy airlines. The seven "legacy" airlines and eight other airlines are overseen through ATOS. The air carriers in the ATOS program are Alaska; American; Continental; Delta; Northwest; United; American Eagle; Champion; ExpressJet; SkyWest; Southwest; Trans States; FedEx; United Parcel Service; and US Airways, which recently merged with America West. [2] GAO, Aviation Safety: System Safety Approach Needs Further Integration into FAA's Oversight of Airlines, GAO-05-726 (Washington, D.C.: Sept. 28, 2005). [3] GAO, Aviation Safety: Weaknesses in Inspection and Enforcement Limit FAA in Identifying and Responding to Risks, GAO/RCED-98-6 (Washington, D.C.: Feb. 27, 1998); GAO, Aviation Safety: FAA Needs to More Aggressively Manage Its Inspection Program, GAO/T-RCED-92-25 (Washington, D.C.: Feb. 6, 1992). [4] GAO/RCED-98-6; GAO/T-RCED-92-25. [5] GAO, Aviation Safety: FAA Needs to Strengthen the Management of Its Designee Programs, GAO-05-40 (Washington, D.C.: Oct. 8, 2004). [6] We identified 62 aviation experts with knowledge and expertise in FAA's designee programs, who participated on a Web-based panel that provided the group's views on the strengths and weaknesses of the designee programs and ways to improve the programs. The experts included designees, FAA inspectors and engineers, independent experts and university academics, and private sector and aviation industry associations. We obtained the experts' views by employing an iterative and controlled feedback process for obtaining individual views and then allowing each participant to respond to the entire panel's comments. [7] We define technical training as training in aviation technologies. FAA includes in its definition of technical training topics such as system safety and risk analysis, inspector job skills, data analysis, and training in software packages. [8] GAO, Aviation Safety: FAA Management Practices for Technical Training Mostly Effective; Further Actions Could Enhance Results, GAO- 05-728 (Washington, D.C.: Sept. 7, 2005). We compared FAA's management of its inspector technical training efforts with effective management practices outlined in GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: Mar. 1, 2004). [9] GAO-05-40. [10] However, many experts on our panel indicated it was of high or highest importance to ensure standard training of designees within specific specialties to improve the consistency of their work, and to increase the number of subject-matter workshops for designees. [11] Because of the statistical survey techniques we employed in surveying FAA's inspectors, we are 95 percent confident that the results we present are within 4.6 percentage points of the results that we would have obtained if we had surveyed all 3,000 inspectors. That is, we are 95 percent confident that had we surveyed all inspectors, between 48 and 57 percent of them would have told us that, to a great or very great extent, they have the technical knowledge to do their jobs. All percentage estimates from the survey have a margin of error of plus or minus 4.6 percentage points or less, unless otherwise noted. [12] GAO, Aviation Safety: FAA Needs to Update the Curriculum and Certification Requirements for Aviation Mechanics, GAO-03-317 (Washington, D.C.: Mar. 6, 2003). [13] GAO-05-40. [14] These databases are Flight Standards Service's Program Tracking and Reporting Subsystem and National Vital Information Subsystem, Aircraft Certification Service's Designee Information Network, and Office of Aerospace Medicine's Airmen Medical Certification Information Subsystem. [15] We found that during fiscal years 1993 through 2003, FAA closed about 53 percent of the nearly 200,000 enforcement actions with administrative actions (such as warning notices). About 28 percent of the actions were closed with legal sanctions (such as fines) and about 18 percent were closed with no enforcement action. [16] A production approval holder is an entity that holds a certificate, approval, or authorization from FAA to manufacture aircraft, aircraft engines, propellers, and related parts and articles.