Summary

The National Aviation Operations Monitoring Service (NAOMS), begun by the National Aeronautics and Space Administration (NASA) in 1997, aimed to develop a methodology that could be used to survey a wide range of aviation personnel to monitor aviation safety. NASA expected NAOMS surveys to be permanently implemented and to complement existing federal and industry air safety databases by generating ongoing data to track event rates into the future. The project never met these goals and was curtailed in January 2007. GAO was asked to answer these questions: (1) What were the nature and history of NASA's NAOMS project? (2) Was the survey planned, designed, and implemented in accordance with generally accepted survey principles? (3) What steps would make a new survey similar to NAOMS better and more useful? To complete this work, GAO reviewed and analyzed material related to the NAOMS project and interviewed officials from NASA, the Federal Aviation Administration, and the National Transportation Safety Board. GAO also compared the development of the NAOMS survey with guidelines issued from the Office of Management and Budget, and asked external experts to review and assess the survey's design and implementation.

NAOMS was intended to demonstrate the feasibility of using surveys to identify accident precursors and potential safety issues. The project was conceived and designed to provide broad, long-term measures on trends and to measure the effects of new technologies and aviation safety policies. Researchers planned to interview a range of aviation personnel to collect data in order to generate statistically reliable estimates of risks and trends. After planning and development, a field trial, and eventual implementation of the air carrier pilot survey and the development of a smaller survey of general aviation pilots, the project effectively ended when NASA transmitted a Web-based version of the air carrier pilot survey to the Air Line Pilots Association. NAOMS's air carrier pilot survey was planned and designed in accordance with generally accepted survey principles, including its research and development, consultation with stakeholders, memory experiments to enhance the questionnaire, and a large-scale field trial. The survey's sample design and selection also met generally accepted research principles, but there were some limitations, and the survey data may not adequately represent the target population. Sample frame and design decisions to maintain program independence and pilot privacy complicate analysis of NAOMS data. Certain implementation decisions, including extended methodological experiments and data entry issues, also complicate analytical strategies. Also, working groups of aviation stakeholders were convened as part of NAOMS to assess the validity and utility of the data, but these groups never had access to the raw data and were disbanded before achieving consensus. To date, NAOMS data have not been fully analyzed or benchmarked against other data sources. While NAOMS's limitations are not insurmountable, a new survey would require more coherent planning and sampling methods, a cost-benefit analysis, closer collaboration with potential customers, a detailed analysis plan, a reexamination of the sampling strategy, and a detailed project management plan to accommodate concerns inherent in any survey endeavor. As a research and development project, NAOMS was a successful proof of concept with many strong methodological features, but the air carrier pilot survey could not be reinstated without revisions to address some of its methodological limitations. The designers of a new survey would want to supplement NAOMS where it was self-limiting. Alternatively, a newly constituted research team might lead operational, survey, and statistical experts in extensively analyzing existing data to illuminate future projects. In reviewing a draft of this report, NASA reiterated that NAOMS was a research and development project and provided technical comments, which GAO incorporated as appropriate. NASA also expressed concern about protecting NAOMS respondents' confidentiality, a concern GAO shares. However, GAO noted that other agencies have developed mechanisms for releasing sensitive data to appropriate researchers. The Department of Transportation had no comments.