This is the accessible text file for GAO report number GAO-08-258 entitled 'Aviation Weather: FAA Is Reevaluating Services at Key Centers; Both FAA and the National Weather Service Need to Better Ensure Product Quality' which was released on February 26, 2008. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Report to Congressional Requesters: United States Government Accountability Office: GAO: January 2008: Aviation Weather: FAA Is Reevaluating Services at Key Centers; Both FAA and the National Weather Service Need to Better Ensure Product Quality: GAO-08-258: GAO Highlights: Highlights of GAO-08-258, a report to congressional requesters. Why GAO Did This Study: The National Weather Service’s (NWS) weather products are a vital component of the Federal Aviation Administration’s (FAA) air traffic control system. In addition to providing aviation weather products developed at its own facilities, NWS also provides staff on-site at each of FAA’s en route centers (see fig.) This group of NWS meteorologists—called a center weather service unit—provides air traffic managers with forecasts and briefings on regional conditions including turbulence, icing, and freezing precipitation. GAO agreed to (1) determine the status of NWS’s plans for restructuring the offices that provide aviation weather services at FAA’s en route centers, (2) identify FAA’s requirements and its alternative sources for these services, and (3) evaluate both agencies’ current abilities to ensure the consistency and quality of these services. To do so, GAO evaluated agency plans for restructuring offices, defining requirements, and ensuring quality products, and interviewed agency officials. What GAO Found: NWS developed a proposal for restructuring the offices that provide aviation weather services at FAA’s en route centers, but these plans are currently on hold. In 2005, FAA requested that NWS restructure its center weather service units by consolidating offices, providing remote services, and reducing personnel costs. In response, NWS conducted a prototype that demonstrated that the services the center weather service units currently provide could be provided remotely by the closest weather forecast office. It subsequently proposed to implement this prototype, but FAA declined this proposal. NWS may reconsider its proposal or other alternative organizational structures as it works to meet FAA’s needs in the future. FAA considers its existing requirements governing the center weather service units to be too broad to ensure the efficiency and cost- effectiveness of the services, so the agency worked for several months to redefine its requirements. By September 2007, FAA had developed draft requirements that specified the products and services to be performed by meteorologists at the en route center, including conducting weather briefings and developing local icing and turbulence forecasts. FAA finalized a more expansive set of requirements at the end of December 2007, and expects NWS to respond within 120 days on its ability to fulfill the requirements. FAA has stated that, if NWS is unable to meet the requirements, it will consider using alternative sources such as private industry or government laboratories to meet the requirements. Although interagency agreements between NWS and FAA state that both agencies have responsibilities for assuring and controlling the quality of aviation weather observations, neither agency consistently does so for weather products and services produced at the en route centers. Specifically, neither agency has developed performance measures and metrics, regularly evaluated weather service unit performance, or provided feedback to improve these aviation weather products and services. Because of this lack of performance tracking and oversight, NWS cannot demonstrate the quality or value of its services, and FAA cannot ensure the quality of the services it funds. Until both agencies are able to measure and ensure the quality of the aviation weather products at the en route centers, FAA may not be getting the information it needs to effectively manage air traffic. Figure: FAA Facilities Involved in Air Traffic Control: [See PDF for image] This figure is an illustration of FAA Facilities Involved in Air Traffic Control. The following data is depicted: Air Traffic Control System Command Center: Preflight: Air traffic control tower; Takeoff: Air traffic control tower; Departure: Terminal radar approach control; En route: Air route traffic control center; Descent: Air route traffic control center; Terminal radar approach control; Approach: Terminal radar approach control; Landing: Air traffic control tower. Source: GAO analysis of FAA data. [End of figure] What GAO Recommends: GAO is recommending that Commerce and Transportation define performance measures for aviation weather services and evaluate the quality of these services. Commerce agreed with the recommendations. Transportation did not agree or disagree with the recommendations, but stated that its just-released requirements include performance measures and evaluation procedures. To view the full product, including the scope and methodology, click on [hyperlink, http://www.GAO-08-258]. For more information, contact David Powner at (202) 512-9286 or pownerd@gao.gov. [End of section] Contents: Letter: Results in Brief: Background: NWS's Plans for Restructuring Its Center Weather Service Units Are on Hold: FAA Finds Its Existing Requirements Are Not Sufficiently Precise and Is Developing New Ones; Agency Has Not Ruled Out Private Industry Sources: Neither NWS Nor FAA Ensure the Quality of Aviation Weather Services at En Route Centers: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Department of Commerce: Appendix III" GAO Contact and Staff Acknowledgments: Tables: Table 1: Weather Products Developed by the Aviation Weather Center: Table 2: Systems Used in the Center Weather Service Units: Table 3: Key Products and Services Provided by Center Weather Service Units: Figures: Figure 1: FAA Facilities Involved in Air Traffic Control: Figure 2: Center Weather Service Unit Locations and Service Areas: Abbreviations: FAA: Federal Aviation Administration: NOAA: National Oceanic and Atmospheric Administration: NWS: National Weather Service: [End of section] United States Government Accountability Office: Washington, DC 20548: January 11, 2008: Congressional Requesters: The National Weather Service (NWS) plays a significant role in providing weather services to the aviation community. NWS's weather products and data are vital components of the Federal Aviation Administration's (FAA) air traffic control system, providing weather information to local, regional, and national air traffic management, navigation, and surveillance systems. NWS aviation weather products include forecasts and warnings of meteorological conditions that could affect air traffic, including thunderstorms, air turbulence, and icing. In addition to providing aviation weather products that are developed at its own facilities, NWS also provides staff on-site at each of FAA's en route centers--the facilities that control high-altitude flight outside the airport tower and terminal areas. This group of NWS meteorologists--called a center weather service unit--provides air traffic managers with forecasts, advisories, and periodic weather briefings on regional conditions. Over the last few years, FAA has been exploring its options for enhancing the efficiency of the aviation weather services provided at its en route centers. Because of your interest in possible changes to NWS's aviation weather services, we agreed to (1) determine the status of NWS's plans for restructuring the offices that provide aviation weather services at FAA's en route centers, (2) identify FAA's requirements and its alternative sources for these services, and (3) evaluate both agencies' current abilities to ensure the consistency and quality of these services. To address our objectives, we reviewed NWS's plans for restructuring its center weather service units, interagency agreements governing the aviation weather program and its requirements, and efforts by both FAA and NWS to ensure the quality of aviation weather service. We compared the agencies' efforts with best practices for quality assurance. We also interviewed relevant agency officials, as well as FAA and NWS employees at en route centers. We performed our work at FAA and NWS headquarters offices, an FAA air traffic control tower, and FAA's Air Traffic Control System Command Center, in the Washington, D.C., metropolitan area. In addition, we conducted work at four en route centers across the country and at NWS's Aviation Weather Center in Kansas City, Missouri. We conducted this performance audit from May 2007 to December 2007, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Additional details on our objectives, scope, and methodology are provided in appendix I. Results in Brief: NWS developed a proposal for restructuring the offices that provide aviation weather services at FAA's en route centers, but these plans are currently on hold. In 2005, FAA requested that NWS restructure its center weather service units by consolidating offices, providing remote services, and reducing personnel costs. In response, NWS conducted a prototype that demonstrated that the services currently provided by the center weather service units could be provided remotely by the closest weather forecast office--effectively removing the center weather service unit staff from the en route center. It subsequently presented a proposal for implementing this prototype, but FAA declined this proposal. Instead, FAA decided to more clearly define its requirements for the weather services provided at en route centers. NWS officials stated that they may revise the proposal or consider other alternative organizational structures to meet FAA's needs in the future. FAA considers its existing requirements governing center weather service units to be too broad to ensure the efficiency and cost- effectiveness of the services, so the agency worked for several months to redefine its requirements. By September 2007, FAA had developed draft requirements that specified the products and services to be performed by the meteorologists at the en route centers, including conducting weather briefings and developing local icing and turbulence forecasts. FAA finalized a more expansive set of requirements at the end of December 2007 and expects NWS to respond within 120 days on its ability to fulfill the requirements. FAA stated that if NWS is unable to meet the requirements, it will consider using alternative sources such as private industry or government laboratories to meet the requirements. Although interagency agreements between NWS and FAA state that both agencies have responsibilities for assuring and controlling the quality of aviation weather observations, neither agency consistently does so for weather products and services produced at the en route centers. Specifically, neither has developed performance measures and metrics, regularly evaluated weather service unit performance, or provided feedback to improve these aviation weather products and services. Because of this lack of performance tracking and oversight, NWS cannot demonstrate the quality or value of its services, and FAA cannot ensure the quality of the services it funds. Until both agencies are able to measure and ensure the quality of the aviation weather products and services at the en route centers, FAA may not be getting the information it needs to effectively manage air traffic. We are making recommendations to the Secretaries of Commerce and Transportation to ensure that NWS and FAA develop performance measures for aviation weather services provided at en route centers, evaluate the services against those measures, and provide feedback to the NWS staff on how to improve services. In written comments on a draft of this report, the Secretary of Commerce agreed with our recommendations and stated that after FAA provides its revised requirements, the National Oceanic and Atmospheric Administration (NOAA) would work with FAA to develop methods for performance monitoring and evaluation. The Department of Transportation's Director of Audit Relations also provided comments via e-mail on a draft of this report, but the department did not agree or disagree with our recommendations. In its comments, the department stated that FAA's revised requirements document establishes performance measures and evaluation procedures, and that FAA would negotiate with NWS to implement them. Both departments also provided technical comments that we incorporated as appropriate. Background: FAA is responsible for ensuring safe, orderly, and efficient air travel in the national airspace system. NWS supports FAA by providing aviation-related forecasts and warnings at air traffic facilities across the country. Among other support and services, NWS provides four meteorologists at each of FAA's 21 en route centers to provide on-site aviation weather services. This arrangement is defined and funded under an interagency agreement. FAA's Mission and Organizational Structure: FAA's primary mission is to ensure safe, orderly, and efficient air travel in the national airspace system. FAA reported that, in 2006, air traffic in the national airspace system exceeded 46 million flights and 750 million passengers. In addition, at any one time, as many as 7,000 aircraft--both civilian and military--could be aloft over the United States. In 2004, FAA's Air Traffic Organization was formed to, among other responsibilities, improve the provision of air traffic services. More than 36,000 employees within FAA's Air Traffic Organization support the operations that help move aircraft through the national airspace system. The agency's ability to fulfill its mission depends on the adequacy and reliability of its air traffic control systems, as well as weather forecasts made available by NWS and automated systems. These resources reside at, or are associated with, several types of facilities: air traffic control towers, terminal radar approach control facilities, air route traffic control centers (en route centers), and the Air Traffic Control System Command Center. The number and functions of these facilities are as follows: * 517 air traffic control towers manage and control the airspace within about 5 miles of an airport. They control departures and landings, as well as ground operations on airport taxiways and runways. * 170 terminal radar approach control facilities provide air traffic control services for airspace within approximately 40 miles of an airport and generally up to 10,000 feet above the airport, where en route centers' control begins. Terminal controllers establish and maintain the sequence and separation of aircraft. * 21 en route centers control planes over the United States--in transit and during approaches to some airports. Each center handles a different region of airspace. En route centers operate the computer suite that processes radar surveillance and flight planning data, reformats it for presentation purposes, and sends it to display equipment that is used by controllers to track aircraft. The centers control the switching of voice communications between aircraft and the center, as well as between the center and other air traffic control facilities. Two en route centers also control air traffic over the oceans. * The Air Traffic Control System Command Center manages the flow of air traffic within the United States. This facility regulates air traffic when weather, equipment, runway closures, or other conditions place stress on the national airspace system. In these instances, traffic management specialists at the command center take action to modify traffic demands in order to keep traffic within system capacity. See figure 1 for a visual summary of the facilities that control and manage air traffic over the United States. Figure 1: FAA Facilities Involved in Air Traffic Control: [See PDF for image] This figure is an illustration of FAA Facilities Involved in Air Traffic Control. The following data is depicted: Air Traffic Control System Command Center: Preflight: Air traffic control tower; Takeoff: Air traffic control tower; Departure: Terminal radar approach control; En route: Air route traffic control center; Descent: Air route traffic control center; Terminal radar approach control; Approach: Terminal radar approach control; Landing: Air traffic control tower. Source: GAO analysis of FAA data. [End of figure] NWS's Mission and Organizational Structure: The mission of NWS--an agency within the Department of Commerce's NOAA- -is to provide weather, water, and climate forecasts and warnings for the United States, its territories, and its adjacent waters and oceans to protect life and property and to enhance the national economy. In addition, NWS is the official source of aviation-and marine-related weather forecasts and warnings, as well as warnings about life- threatening weather situations. The coordinated activities of weather facilities throughout the United States allow NWS to deliver a broad spectrum of climate, weather, water, and space weather services in support of its mission. These facilities include 122 weather forecast offices located across the country that provide a wide variety of weather, water, and climate services for their local county warning areas, including advisories, warnings, and forecasts; 9 national prediction centers [Footnote 1] that provide nationwide computer modeling to all NWS field offices; and 21 center weather service units that are located at FAA en route centers across the nation and provide meteorological support to air traffic controllers. NWS Provides Aviation Weather Services to FAA: As an official source of aviation weather forecasts and warnings, several NWS facilities provide aviation weather products and services to the FAA and aviation sector. These facilities include the Aviation Weather Center, weather forecast offices located across the country, and center weather service units located at FAA en route centers. Aviation Weather Center: The Aviation Weather Center located in Kansas City, Missouri, issues warnings, forecasts, and analyses of hazardous weather for aviation. Staffed by 65 personnel, the center develops warnings of hazardous weather for aircraft in flight and forecasts of weather conditions for the next 2 days that could affect both domestic and international aviation. The center also leads a collaborative effort to develop a forecast of expected convective events for the entire country every 2 hours. This is used by FAA to manage aviation traffic flow across the country. The Aviation Weather Center's key products are described in table 1. Table 1: Weather Products Developed by the Aviation Weather Center: Weather product: Significant Meteorological Information; Description: A brief description of the development and occurrence or expected occurrence of certain nonthunderstorm weather conditions that may affect the safety of aircraft in the en route environment. These conditions include severe icing not associated with thunderstorms, severe or clear air turbulence not associated with thunderstorms, dust or sand storms that lower visibility to below 3 miles, volcanic ash, and tropical cyclones. Weather product: Convective Significant Meteorological Information; Description: A text product describing the occurrence or expected occurrence of thunderstorms and related weather conditions over the contiguous United States within 2 hours of issuance time. Weather product: Airman's Meteorological Information; Description: A brief description of the development and occurrence or expected occurrence of certain nonthunderstorm weather conditions that may affect the safety of aircraft in the en route environment, but that do not meet the criteria to develop a Significant Meteorological Information product. Weather product: Collaborative Convection Forecast Product; Description: A graphical convection forecast developed for strategic planning and management of en route air traffic. It is produced every 2 hours through collaboration--by way of an online chat room--among the Aviation Weather Center, the Meteorological Services of Canada, airline meteorology departments, FAA's Air Traffic Control System Command Center, and the center weather service units. These collaborative forecasts are produced between March 1 and October 31 every year. Source: GAO analysis of NWS data. [End of table] Weather Forecast Offices: NWS's 122 weather forecast offices issue terminal area forecasts for approximately 625 locations every 6 hours or when conditions change. These forecasts consist of the expected weather conditions significant to a given airport or terminal area and are primarily used by commercial and general aviation pilots. Center Weather Service Units: NWS's center weather service units are located at each of FAA's 21 en route centers and operate 16 hours a day, 7 days a week (see fig. 2). Each weather service unit usually consists of three meteorologists and a meteorologist-in-charge who provide strategic advice and aviation weather forecasts to FAA traffic management personnel. Governed by an interagency agreement, FAA currently reimburses NWS approximately $12 million annually for this support. Figure 2: Center Weather Service Unit Locations and Service Areas: [See PDF for image] This figure is a map of the United States, illustrating Center Weather Service Unit Locations and Service Areas. The following locations are depicted: Boston; New York; Washington, DC; Cleveland; Indianapolis; Memphis; Atlanta; Jacksonville; Miami; Houston; Fort Worth; Kansas City; Chicago; Minneapolis; Denver; Albuquerque; Salt Lake City; Los Angeles; Oakland; Seattle; Anchorage. Source: NWS (data). Map Resources (map). [End of figure] Center Weather Service Units: An Overview of Systems and Operations: The meteorologists at the center weather service units use a variety of systems to gather and analyze information compiled from NWS and FAA weather sensors. Key systems used to compile weather information include FAA's Weather and Radar Processor, FAA's Integrated Terminal Weather System, and a remote display of NWS's Advanced Weather Interactive Processing System. Meteorologists at the en route centers located along the Northeast air traffic corridor also use FAA's Corridor Integrated Weather System to oversee the interaction of air traffic routes and weather. Table 2 provides a description of selected systems. Table 2: Systems Used in the Center Weather Service Units: System: Weather and Radar Processor; Description: FAA's Weather and Radar Processor is used in en route centers and receives information from automated weather sensors located at airports and from other sources such as weather satellites. It compiles the information and provides current weather and forecasts to air traffic supervisors, traffic flow managers, and the center weather service unit meteorologists. System: Advanced Weather Interactive Processing System--Remote Display; Description: NWS's Advanced Weather Interactive Processing System integrates hydrometeorological data from a variety of sources and produces graphical displays at NWS weather forecast offices, river forecast centers, and national centers. This system aids forecaster analysis and decision making. Meteorologists at the en route centers have access to this system through a remote display system, which provides a dedicated connection to the supporting weather forecast office. The Remote Display is funded by FAA, and maintenance is provided by NWS. System: Integrated Terminal Weather System; Description: FAA's Integrated Terminal Weather System furnishes air traffic controllers and meteorologists with full-color graphic displays of weather information concerning airport terminal airspace within a 60- mile radius. The system also projects movement of severe weather systems up to 1 hour in the future and has been installed at 22 airports. System: Corridor Integrated Weather System; Description: FAA's Corridor Integrated Weather System is a prototype decision support tool that gathers weather information occurring along the Northeast air traffic corridor to help controllers select the most efficient routes for diverting traffic to avoid severe weather conditions. This system provides traffic flow managers with comprehensive convective weather data needed for tactical modifications, occurring within 2 hours, to the operational plan. These tactical modifications to the operational plan may include the weather impacts on air traffic control capacity, a need to modify the mitigation plan, and the execution of a modified mitigation plan. Source: GAO analysis of FAA and NWS data. [End of table] NWS meteorologists at the en route centers provide several products and services to the FAA staff, including meteorological impact statements, center weather advisories, periodic briefings, and on-demand consultations. These products and services are described in table 3. In addition, center weather service unit meteorologists can provide input every 2 hours to the Aviation Weather Center's creation of the Collaborative Convective Forecast Product, train FAA personnel on how to interpret weather information and, if warranted, provide weather briefings to nearby terminal radar approach control facilities. Table 3: Key Products and Services Provided by Center Weather Service Units: Product or service: Meteorological impact statement; Description: An unscheduled forecast of weather conditions that are expected to adversely impact the flow of air traffic in the en route center's area of responsibility within 4 to 12 hours. Product or service: Center weather advisory; Description: A short-term, unscheduled warning of hazardous weather conditions used primarily by air crews to anticipate and avoid adverse weather conditions in the en route and terminal environments. It describes current weather conditions or adverse weather conditions-- such as moderate to severe icing or turbulence, thunderstorms, and low ceilings and visibility--beginning within the next 2 hours. Product or service: Briefings; Description: Short updates provided by en route center meteorologists to FAA supervisors twice a day; they include current weather advisories, a summary of the predicted weather in the en route area, terminal forecasts, and jet stream and freezing information. Product or service: On-demand consultation; Description: Unscheduled verbal presentations provided to traffic management controllers, supervisors, and other FAA facilities within the en route center area. Consultations may be about the expected weather conditions or interpretations of weather information from the satellite images. Source: GAO analysis of FAA and NWS data. [End of table] FAA Is Seeking to Improve Aviation Weather Services Provided at En Route Centers: In recent years, FAA has undertaken multiple initiatives to assess and improve the performance of the center weather service units. [Footnote 2] Studies conducted in 2003 and 2006 highlighted concerns with NWS's weather service units while FAA's more recent initiatives have sought solutions. In November 2003, FAA performed a study of the performance of the weather service units and found that the services provided at different en route locations were inconsistent, the products were not standardized, and there was little communication and collaboration between neighboring service units. Additionally, in January 2006, FAA initiated an analysis of the value of different activities performed by the center weather service units. Similar to the 2003 study, the results of this analysis noted the lack of standardization of products, services, tools, and procedures. In addition, the report found that quality assurance was provided on an informal basis, there was no formal feedback process for products and services, and meteorological training was not standardized. To address these concerns, FAA undertook several initiatives. In September 2005, FAA requested that NWS restructure its aviation weather services to provide improved services more efficiently. FAA also contracted for an analysis of whether weather information could be remotely delivered to air traffic controllers. The subsequent report [Footnote 3] confirmed that it would be possible for weather information, products, and services to be delivered to customers at the en route centers from one or many remote locations with currently available state of the art technology platforms. Following up on this information, in October 2006, FAA administered a market survey to determine whether the private sector could provide remote weather services at a lower cost than currently provided. Ten organizations, including private sector firms and government-funded laboratories, responded that they could provide the services that FAA wanted. NWS's Plans for Restructuring Its Center Weather Service Units Are on Hold: NWS developed a proposal to address FAA's request for more efficient center weather service, but any plans for restructuring the center weather service units are currently on hold. When FAA requested that NWS restructure its aviation weather services, the agency asked NWS to consolidate 20 of the service units (excluding the service unit in Alaska) to a smaller number of sites, reduce related NWS personnel costs by 20 percent, and deliver forecast products and services 24 hours a day, 7 days a week. Subsequently, NWS chartered a prototype team to evaluate approaches for providing services to FAA and to prepare a proposal for modernizing the national aviation weather program. In August 2006, the NWS team conducted a prototype in which center weather service unit products and services were completed and delivered remotely from the closest weather forecast office. This prototype showed that remote operations were possible and effective, but that they would be difficult to implement because of the need for cultural change, technology upgrades, and communication stability. Specifically, forecasters in the prototype were not able to provide dedicated support for the aviation mission because their other duties- -including forecasting severe weather at the weather forecast office-- took precedence. In addition, a collaboration technology used during the prototype was not operationally ready-to-use, servers were unstable, critical radar data were inconsistent with weather forecast office data, and communications lines were unstable throughout the prototype. In spite of these difficulties, in October 2006, NWS presented its proposal for restructuring its aviation weather services to FAA. In April 2007, FAA declined NWS's proposal. FAA officials explained that NWS's proposal was not viable because it did not consolidate the offices to a smaller number of sites and it involved higher training costs. Instead, FAA reported that it would redefine its requirements for the functions provided by the center weather service units. Officials stated that once FAA's requirements are more clearly defined, NWS may revise its proposal or consider other alternative organizational structures to deliver those requirements. FAA Finds Its Existing Requirements Are Not Sufficiently Precise and Is Developing New Ones; Agency Has Not Ruled Out Private Industry Sources: FAA considers its existing requirements governing NWS's center weather service units to be too broad to ensure the efficiency and cost- effectiveness of the services, and has therefore worked for several months to redefine its requirements. By September 2007, the agency had developed draft requirements that specified activities and performance measures. In late December 2007, after we completed our review, FAA finalized a more expansive set of requirements. It expects NWS to respond within 120 days as to whether they are able to meet the requirements. If NWS is unable to fulfill the new requirements, FAA has stated that it will consider using alternative sources from private industry or government laboratories to obtain the weather services necessary to meet its requirements. FAA's Existing Requirements for NWS Services Are Imprecise: FAA's existing requirements for the center weather service units are broadly outlined in an interagency agreement that is updated every few years. The interagency agreement specifies that NWS is to provide meteorological advice and consultation to en route center operations personnel and other designated FAA air traffic facilities within the en route area of responsibility. [Footnote 4] This agreement establishes specific terms that govern the number of NWS staff, their working hours, and cost reimbursement details. It does not specify the contents, quality, or frequency of weather products. An NWS directive, signed in May 2006 and intended for NWS's weather forecast offices and center weather service units, provides more specific information regarding the content of weather products, including daily briefings, meteorological impact statements, and center weather advisories. Specifically, a service unit's briefings are to contain sufficient information for air traffic controllers to make decisions and appropriate operational adjustments based on weather impacts, including a discussion of advisories in effect, weather systems and their movements within the en route center area, flight conditions (including convective weather, turbulence, and icing), weather conditions for large airports (including heavy snow, freezing precipitation, and low visibility), wind direction and speed, and any other locally required items. The service unit's meteorological impact statements are to detail weather conditions expected to adversely impact air traffic flow in the service unit area of responsibility and should include the location, height, extent, and movement of the weather conditions. In addition, the center weather advisories may include forecasts of conditions expected to begin within 2 hours of issuance. Center weather advisories typically include the issuance time, the time the meteorologist expects the condition to begin, other weather advisories that are augmented by the center weather advisory, and the location and a brief description of the weather phenomenon. FAA Is Redefining Its Requirements: In April 2007, FAA's Air Traffic Organization began refining its requirements for aviation weather services at the en route centers. To do this, FAA collected all related NWS and FAA orders and directives and developed a list of over 100 products and services that the different service units provide. FAA then sent this list to traffic managers in each of the en route centers, asking them to specify the products and services that they need, the ones they do not need, and any new products or services that they would like. Traffic managers were also able to specify whether they needed some of the more customized weather products that are currently available at selected en route centers. Using results from this survey, FAA developed a list of approximately 47 products and services. Examples of products and services include conducting scheduled briefings, developing local turbulence and icing forecasts, and issuing products such as the meteorological impact statement and the center weather advisory. In commenting on a draft of this report, FAA noted that it finalized a more expansive set of requirements on December 19, 2007. NWS will have 120 days to respond as to whether they are able to meet the requirements outlined in the final requirements document. Also, FAA plans to request that NWS respond to different assumptions, including having aviation weather services provided at the current en route center locations, having aviation weather services provided at an off- site location, and potentially having a hybrid approach. FAA May Consider Alternative Sources for Weather Service Support: FAA officials within the Air Traffic Organization stated that they are not currently considering private industry sources for weather services at en route centers, but that they may do so in the future. FAA officials stated that until NWS responds as to whether and how it can fulfill the revised FAA requirements, it is premature to consider alternative sources. However, FAA has stated that if NWS cannot meet the refined requirements, it will consider taking steps to procure weather services from alternative sources. While acknowledging that NWS is not directed to be the exclusive provider of weather products and services to FAA, NWS's Senior Counsel stated that the Secretary of Commerce is required to provide meteorological reports, such as those provided by the center weather service units, to persons engaged in civil aeronautics. [Footnote 5] He stated that if NWS cannot meet FAA's requirements, FAA and NWS should enter into negotiations. Neither NWS Nor FAA Ensure the Quality of Aviation Weather Services at En Route Centers: While interagency agreements between NWS and FAA state that both agencies have responsibilities for assuring and controlling the quality of aviation weather observations, neither NWS nor FAA consistently do so for weather products produced at the en route centers. Leading organizations use quality assurance to provide staff and management with objective insights into processes and associated work products. [Footnote 6] Generally, quality assurance includes objectively evaluating performed processes, work products, and services against applicable process descriptions, standards, and procedures; identifying and documenting noncompliance issues; providing feedback to project staff and managers on the results of quality assurance activities; and ensuring that noncompliance issues are addressed. However, neither NWS nor FAA has developed performance measures and metrics, regularly evaluated weather service unit performance, or provided feedback to improve these aviation weather products and services. Because of this lack of performance tracking and oversight, NWS cannot demonstrate the quality or value of its services, and FAA cannot ensure the value of the services it funds. As a result, it is not clear that FAA is getting the information it needs to effectively manage air travel. FAA officials stated that they intend to establish performance measures for their redefined requirements and to improve their oversight against these measures. However, at present, FAA has not worked with NWS to define a comprehensive set of measures for its requirements, and it is unclear how the agency would develop a performance baseline for comparison to actual performance because many of the products and services have not previously been measured. NWS Does Not Measure or Evaluate Aviation Weather Products and Services at En Route Centers: NWS does not measure or evaluate the aviation weather services it provides at en route centers. Under existing interagency agreements, NWS is responsible for controlling the quality of its aviation weather observations. Specifically, NWS is responsible for monitoring and evaluating the quality and effectiveness of its aviation weather services, including the services provided at the weather forecast offices, the Aviation Weather Center, and the en route centers. While NWS has developed and continues to monitor performance measures for aviation weather forecasts provided by its weather forecast offices and the Aviation Weather Center, the agency has not done so for the weather products and services provided at the en route centers. Specifically, NWS has not developed performance measures for aviation weather products and services at en route centers, evaluated the aviation weather products and services developed at the en route centers, or provided feedback for those services. NOAA and NWS officials declined to explain why the agency does not have performance measures for aviation weather products or services at en route centers, but they noted that neither FAA nor NWS have required or funded such an effort. Further, the aviation services branch chief told us that he had planned to begin evaluations for aviation weather services at the en route centers but decided to wait because of the potential for large- scale changes to the services. Until NWS establishes performance measures and evaluates the quality and effectiveness of its products against these measures, the agency will remain unable to ensure that it provides consistent quality products and to effectively demonstrate the value it provides to FAA. FAA Does Not Consistently Evaluate or Provide Feedback on Aviation Weather Services at En Route Centers: FAA has not consistently evaluated NWS services at its en route centers or adequately provided feedback on the results of its few evaluations. Under interagency agreements, FAA is responsible for ensuring that aviation weather services meet its requirements. In addition, it requires the traffic management officer within each traffic management unit to evaluate the aviation weather services at the en route centers annually and to provide feedback to the resident meteorologist-in- charge. FAA has not consistently ensured the quality of aviation weather services at en route centers. Specifically, it currently does not have any quantitative and objective performance measures--such as timeliness, accuracy, or false alarm rate--by which to evaluate these services. Agreements between the agencies broadly specify the types of aviation weather products to be developed at the en route centers but do not provide criteria by which these products can be evaluated. In addition, FAA has not consistently performed its annual evaluations of these products and services. According to the contracting officer's technical representative responsible for the evaluations, the last evaluation was performed in 2006, and its results were largely anecdotal. Specifically, the evaluation called for the traffic management officer to rate the service unit on a scale of 0 to 4 in different categories, including quality and timeliness of products and services, knowledge of air traffic control, and participation in training. The technical representative told us that he could not find any evaluations in 2005, evaluations of only three service units in 2004, and evaluations of a similarly small number of service units in 2003. Further, FAA is not consistently providing feedback to weather staff at the en route centers. According to the technical representative, the evaluations from 2006 were not compiled or analyzed because the evaluations contained no glaring problems or issues that needed additional attention. In addition, the NWS aviation services branch chief told us that FAA had sent him copies of the evaluations from 2006 but did not offer analysis of these evaluations, express concerns about the services, or send the results to the individual center weather service units. This official also stated that he was not aware that FAA had performed any annual evaluations of the center weather service units prior to 2006. Because FAA has not established performance requirements or consistently and thoroughly evaluated the aviation weather services at en route centers, the agency cannot be sure that the products and services provided by the center weather service unit meteorologists are adding value, and they cannot provide feedback to these meteorologists in order to improve the services. To address this shortfall, FAA officials stated that they intend to establish performance measures for aviation weather services at en route centers when they revise their requirements and to improve their oversight of NWS against these measures. However, FAA has not worked with NWS to develop measures for the products and services it will require from NWS, and it is unclear how the agency would develop a performance baseline for comparison to actual performance because many of the products and services have not previously been measured. Conclusions: In seeking to improve the aviation weather services provided at its en route centers, FAA asked NWS to develop a proposal for restructuring its organization to provide weather services more efficiently. NWS subsequently presented a proposal for providing weather services remotely, but FAA declined this proposal, stating that it would revise and reaffirm its requirements before deciding how to proceed. FAA also noted that if NWS is unable to meet FAA's needs, it will consider alternative sources including private industry. Assessing the value and effectiveness of current weather service products provided at en route centers is difficult because neither NWS nor FAA monitors the accuracy and quality of these aviation weather products. Specifically, NWS has not established performance measures for aviation weather products and services provided at the en route centers, evaluated these products and services, or provided feedback on them. FAA has not specified what level of performance it needs, consistently evaluated the aviation weather services at en route centers, or provided NWS feedback on how to improve its services. While FAA plans to include performance measures when it defines its new requirements, it has not worked with NWS to develop a set of measures, and it is not clear how the agency would develop baseline performance for comparison. Until the agencies establish a system of performance tracking and oversight, NWS will not be able to demonstrate the quality or value of its services, and FAA will not be able to ensure the value of the services it funds. Without knowing the quality of the aviation weather services used at en route centers across the country, FAA may not be getting the information it needs to effectively manage air travel. Recommendations for Executive Action: While many steps remain in defining the future of aviation weather services at en route centers--including negotiations between FAA and NWS on the provision of these services and FAA's subsequent decision on whether to obtain selected services from alternative sources--there are steps both agencies can take to ensure that the quality of future aviation weather products and services are measured and evaluated. We are making two recommendations to the Secretary of Commerce and three recommendations to the Secretary of Transportation to improve the quality of aviation weather products and services at en route centers. We recommend that the Secretary of Commerce direct the Assistant Administrator for the National Weather Service to: * assist FAA in developing performance measures and metrics for the products and services to be provided by center weather service units, and: * perform annual evaluations of aviation weather services provided at en route centers and provide feedback to the center weather service units. Further, we recommend that the Secretary of the Department of Transportation direct the FAA Administrator to: * work with NWS to define performance measures and metrics for aviation weather services provided by meteorologists, * evaluate the services it receives against those measures and metrics, and: * ensure that results of these evaluations are provided to staff stationed at center weather service units so that they can improve performance, where applicable. Agency Comments and Our Evaluation: The Department of Commerce provided written comments on a draft of this report (see app. II). In the department's response, the Secretary of Commerce agreed with our recommendations to assist FAA in developing performance measures and metrics, and to perform annual evaluations of aviation weather services and provide feedback to the center weather service units. The department stated that after FAA provides its revised requirements NOAA would work with FAA to develop methods for performance monitoring and evaluation. Subsequently, on December 19, 2007, FAA provided its revised requirements to NWS. The Department of Transportation's Director of Audit Relations provided comments on a draft of this report via e-mail. In those comments, the department did not agree or disagree with our recommendations. The department stated that FAA's revised requirements are consistent with our recommendations in that they establish performance measures and evaluation procedures, and that FAA would begin to negotiate with NWS to implement them. In its December 2007 requirements document, FAA identified several new requirements for aviation weather. Specifically, the document calls for expanding the scope of the center weather service units to monitor the entire national airspace system, rather than the respective en route center regions. This national scope is expected to allow more integrated decision making at the national level while continuing to provide specialized products at the regional and local levels. The revised requirements also define new products and services, such as providing weather forecasts for terminal radar approach control facilities, increasing weather coverage from 16 hours a day to 24 hours a day, and enhancing the standardization of products among center weather service units. FAA also calls for NWS to prepare three operational concepts for fulfilling the requirements--in its existing configuration located at the 21 en route centers, through remote services provided by a reduced number of regional facilities, and through remote services provided by a single centralized facility. In addition to these requirements, FAA identifies performance measures, as well as processes for evaluating performance and providing feedback to the forecasters. However, the department did not involve NWS in developing its performance measures and did not leverage NWS's expertise in measuring the accuracy, timeliness, and quality of its weather products and services. As a result, the measures may not reflect the leading expertise in measuring the performance of weather forecasters. Thus, we reiterate our recommendation that FAA and NWS work together to establish and monitor performance measures. Both departments also provided technical comments that we incorporated as appropriate. As we agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees, the Secretary of Commerce, the Secretary of Transportation, the Director of the Office of Management and Budget, and other interested parties. In addition, this report will be available at no charge on the GAO Web site at [hyperlink, http://www.gao.gov]. If you have any questions about this report, please contact me at (202) 512-9286 or by e-mail at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Signed by: David A. Powner: Director, Information Technology Management Issues: List of Requesters: The Honorable Nick Lampson: Chairman: The Honorable Bob Inglis: Ranking Member: Subcommittee on Energy and Environment: Committee on Science and Technology: House of Representatives: The Honorable Mark Udall: Chairman: The Honorable Tom Feeney: Ranking Member: Subcommittee on Space and Aeronautics: Committee on Science and Technology: House of Representatives: The Honorable Jerry Costello: Chairman: Subcommittee on Aviation: Committee on Transportation and Infrastructure: House of Representatives: [End of section] Appendix I: Objectives, Scope, and Methodology: Our objectives were to (1) determine the status of the National Weather Service’s (NWS) plans for restructuring the offices that provide aviation weather services at en route centers, (2) identify the Federal Aviation Administration’s (FAA) requirements and its alternative sources for these services, and (3) evaluate the agencies’ abilities to ensure the consistency and quality of these services. To determine the status of NWS’s plans for restructuring aviation weather services, we reviewed agency plans for restructuring its aviation weather services, including prototype plans and results, service unit survey results, and plans for addressing FAA’s requirements. We also interviewed NWS officials to obtain clarifications on these plans. To identify FAA’s requirements and its alternative sources for these services, we reviewed historical requirements documents, including the memorandum of understanding, interagency agreement, and NWS orders regarding center weather service unit products and services; documentation of requirements from FAA; responses from the market study performed by FAA; and legislation regarding aviation weather services. We compared FAA’s current requirements development processes with best practices for developing and validating requirements by the Carnegie Mellon University Software Engineering Institute’s Capability Maturity Model® Integration for Development.[Footnote 7] We also interviewed agency officials and employees who were involved in the requirements gathering process. To evaluate the agencies’ abilities to ensure the consistency and quality of these services, we reviewed agency documentation that governs aviation weather, including the memorandum of understanding, interagency agreement, and NWS orders. We also reviewed results from the most recent FAA evaluation of the center weather service units. We compared these documents with best practices for quality assurance from the Capability Maturity Model® Integration for Development. In addition, we interviewed FAA officials responsible for evaluations of aviation weather services; FAA and NWS en route center staff to obtain information on evaluations and feedback; and NOAA’s Global Systems Division representatives responsible for verifying certain aviation weather products. We performed our work at FAA and NWS headquarters offices, FAA’s Potomac Consolidated Terminal Radar Approach Control facility, FAA’s Air Traffic Control System Command Center, and the Dulles air traffic control tower in the Washington, D.C., metropolitan area. In addition, we conducted work at four FAA en route center offices in Leesburg, Virginia; Denver, Colorado; Dallas, Texas; and Cleveland, Ohio, because they were geographically dispersed and were identified by NWS officials as some of the stronger and weaker weather service units. We also performed observations and interviews at NOAA’s Global Systems Division offices in Boulder, Colorado, because of their expertise in verifying aviation weather products, and NWS’s Aviation Weather Center in Kansas City, Missouri, because of its responsibilities for aviation weather forecasts—some of which are supplemented by the center weather service units. We conducted this performance audit from May 2007 to December 2007, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for findings and conclusions based on our audit objectives. [End of section] Appendix II: Comments from the Department of Commerce: The Secretary Of Commerce: Washington, D.C. 20230: December 18, 2007: Mr. David A. Powner: Director, Information Technology Management Issues: U.S. Government Accountability Office: 441 G Street, NW: Washington, D.C. 20548: Dear Mr. Powner: Thank you for the opportunity to review and comment on the Government Accountability Office's draft report entitled Aviation Weather. FAA Is Reevaluating Services at Key Centers; Both FAA and the National Weather Service Need to Better Ensure Product Quality (GAO-08-258). I enclose the Department of Commerce's comments on the draft report. Sincerely, Signed by: Carlos M. Gutierrez: Enclosure: Department of Commerce's Comments on the Draft GAO Report Entitled "Aviation Weather: FAA is Reevaluating Services at Key Centers; Both FAA and the National Weather Service Need to Better Ensure Product Quality" (GAO-08-258/January 2008): General Comments: The Department of Commerce's National Oceanic and Atmospheric Administration (NOAA) appreciates the opportunity to review this report on aviation weather. The report does a fair and thorough job in assessing the status of the National Weather Service's plans for restructuring offices providing aviation weather services at the Federal Aviation Administration's (FAA) en route centers and evaluating current abilities to ensure the consistency and quality of these services. NOAA Response to GAO Recommendations: Recommendation 1: "We recommend that the Secretary of Commerce direct the Assistant Administrator for the National Weather Service to assist FAA in developing performance measures and metrics for the products and services to be provided to center weather service units." NOAA Response: NOAA agrees with this recommendation. As indicated in the report, the FAA is redefining its requirements for products and services to be performed by meteorologists at en route centers. When the FAA provides its finalized requirements, NOAA will work with them to develop performance measures and metrics for the Center Weather Service Unit products and services. Subsequent collaboration between NOAA and the FAA should lead to a shared service level agreement on milestones, performance measures, and goals. Recommendation 2: "We recommend that the Secretary of Commerce direct the Assistant Administrator for the National Weather Service to perform annual evaluations of aviation weather services provided at en route centers and provide feedback to the center weather service units." NOAA Response: NOAA agrees with this recommendation. NOAA will work with the FAA to develop methods for performance monitoring and evaluation based upon the FAA's service requirements (see the response to Recommendation 1). These methods will involve annual evaluations, at a minimum. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: David A. Powner, (202) 512-9286, or pownerd@gao.gov. Staff Acknowledgments: In addition to the contact person named above, Colleen Phillips, Assistant Director; Kate Agatone; Monica Perez Anatalio; Neil Doherty; Nalani Fraser; Amos Tevelow; and Jessica Waselkow made key contributions to this report. [End of section] Footnotes: [1] These centers include the National Centers for Environmental Prediction Central Operations, Aviation Weather Center, Environmental Modeling Center, Hydrometeorological Prediction Center, Ocean Prediction Center, Storm Prediction Center, Tropical Prediction Center/National Hurricane Center, Climate Prediction Center, and Space Environment Center. [2] FAA is also involved in a longer term initiative to increase the efficiency of the national airspace system and to improve its overall safety. This initiative, called the Next Generation Air Transportation System, is a joint effort between Department of Transportation, the National Aeronautics and Space Administration, the White House Office of Science and Technology Policy, and the Departments of Homeland Security, Defense, and Commerce. FAA anticipates that this initiative may lead to major changes in the aviation weather program that would supercede its current efforts. [3] Federal Aviation Administration, Center Weather Service Unit Post- Operational Study (Washington, D.C.: 2006). [4] In December 2007, FAA and NWS signed an interagency agreement that will be effective for 21 months, with an option for one additional year. [5] 49 U.S.C. § 44720. [6] The Carnegie Mellon University’s Software Engineering Institute is recognized for its expertise in software and system processes. See Carnegie Mellon University Software Engineering Institute, Capability Maturity Model® Integration for Development, Version 1.2 (Pittsburgh, PA: August 2006). [7] Carnegie Mellon University Software Engineering Institute, Capability Maturity Model® Integration for Development, Version 1.2 (Pittsburgh, PA: August 2006). Capability Maturity Model® and Capability Maturity Modeling are registered in the U.S. Patent and Trademark Office. CMM is a service mark of Carnegie Mellon University. [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "Subscribe to Updates." Order by Mail or Phone: The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. Government Accountability Office: 441 G Street NW, Room LM: Washington, D.C. 20548: To order by Phone: Voice: (202) 512-6000: TDD: (202) 512-2537: Fax: (202) 512-6061: To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Gloria Jarmon, Managing Director, jarmong@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: