• COIs/Volpe's Work
  • Centers of Innovation
  • Innovation in Acquisition
  • Project Management Innovation
  • Volpe Organizational Chart
  • Volpe Subsites
    • Human Factors
      • What's New
      • Mission
      • Capabilities
      • Staff
      • Products
      • Publications and Papers
      • Projects
  • Volpe Technical Experts
  • SBIR
  • Technology Transfer
  • Work Portfolio

---

---

Preface

This work was performed as part of an ongoing research program at the U.S. Department of Transportation's John A. Volpe National Transportation Systems Center (Volpe Center) investigating fidelity requirements for simulators used in training and evaluation of airline pilots. The effort documented in this report was performed in collaboration with NASA Ames Research Center, whose participation was supported through the NASA/FAA Interagency Agreement DTFA01-96-Z-02035, Aeronautical Safety and Human Factors Task. The research is supported by the Federal Aviation Administration's Office of the Chief Scientific and Technical Advisor for Human Factors where we thank Dr. Eleana Edens the program AAR-100 manager for her effective guidance and assistance. The need for this work was determined by the Advanced Qualification Program AFS-230, where we thank its manager Dr. Thomas Longridge for his insights.

Many people have contributed to this work. Above all, we would like to thank the instructors/evaluators who shared their experiences and opinions with us. Mr. Cary Ryan, Chairman of the Advanced Qualification Program Instructor/Evaluator Focus Group, reviewed our survey from a pilot's perspective. Mr. James Railey and Ms. Young Jin Jo provided assistance with data analysis and literature review. Captain Bill Hamman and colleagues provided demonstrations and discussions on United Airlines' Interactive Real Time Audio System (IRAS). Mr. William K. Hinkley demonstrated Jeppesen's FS-200/Flightpro system. Dr. Ronald Kruk initiated contact with Jean-Sebastien Lavoie to discuss CAE's Ground Air Traffic Environment System. Dr. E. Donald Sussman, Chief of the Operator Performance and Safety Division at the Volpe Center, provided comments throughout the report generation. We extend our thanks to all of them.

Table of Contents

 

1. Introduction

1.1 Project Background and Motivation

1.2 Project Goals

2. Literature Review

2.1 Advanced Qualification Program

2.2 Crew Resource Management

2.2.1 Approach and Landing Accident (ALA) Reduction Task Force Report

2.2.2 Realistic Radio Communications For Task Management Training

2.3 Summary

3. Instructor/Evaluator Survey

3.1 Current Practices of Radio communications Simulation

3.1.1 Radio Communications Simulations to Own Airplane

3.1.2 Simulation of Other Aircraft

3.1.3 Instructor/Evaluator Workload in the Simulator

3.2 Survey of Instructor/Evaluator Opinion

3.2.1 Radio Communications and Workload

3.2.2 Overall Importance of Simulating Radio Communications for Training Effectiveness

3.2.3 Anticipated Effect of Automating Radio Communications

3.3 Survey Summary and Additional Instructor/Evaluator Comments

4. Technology Review

4.1 Interactive Real Time Audio System

4.2 Ground and Air Traffic Environment System

4.3 Personal Computer Commercial Off-The-Shelf Systems

4.3.1 Jeppesen FS-200/Flightpro

4.3.2 FlightSafety International Aviator PRO 98

4.3.3 Proflight

4.3.4 Flight Unlimited

4.4 Intelligent Systems

4.4.1 Component Agent Approach

4.5 Supporting Technologies

4.5.1 Rapidly Reconfigurable Line-Oriented Evaluation Scenario Generator (RRLOE)

4.5.2 Speech Generation and Recognition Technology

5. Conclusions

6. References

Appendix A - Results of the Approach and Landing Accident Reduction Task Force Report and Related Studies

Appendix B - Realistic Radio communications For Task Management Training

Appendix C - Instructor/Evaluator Questionnaire

List of Figures

1. Perceived I/E Workload in the Simulator

List of Tables

1. Types of ATC Communications Simulated in LOFT, LOE, and SPOT

2. Types of Company Communications Simulated in LOFT, LOE, and SPOT

3. Simulation of ATC Communications with Other Aircraft or Vehicles (Partyline)

4. Effect and Importance of ATC Communications

5. Effect and Importance of Company Communications and Partyline

6. Frequency and Importance of Radio Communications for Skills Training

7. Effect of Automated Radio Communications on Workload and Training Effectiveness

List of Acronyms

ACARS - Aircraft Communications Addressing and Reporting System

AFSS - Automated Flight Service Station

ALA - Approach and Landing Accident

AQP - Advanced Qualification Project

ARTCC - Air Route Traffic Control Center

ASRS - Aviation Safety Reporting System

ATC - Air Traffic Control

ATIS - Automatic Terminal Information Service

CFII - Certified Instrument Flight Instructor

CFIT - Controlled Flight Into Terrain

CFR - Code of Federal Regulations

CRM - Crew Resource Management

CTM - Cockpit Task Management

FAA - Federal Aviation Administration

FAR - Federal Aviation Regulation

FMS - Flight Management System

GATES - Ground Air Traffic Environment System

ICAO - International Civil Aviation Organization

I/E - Instructor/Evaluator

IOE - Initial Operating Experience

IRAS - Interactive Real Time Audio System

LOE - Line Operational Evaluation

LOFT - Line Oriented Flight Training

LOS - Line Operational Simulation

NTSB - National Transportation Safety Board

PCATD - Personal Computer Aviation Training Device

PC COTS - Personal Computer Commercial Off-The-Shelf

PRM - Parallel Runway Monitored Approach

RIDES - Rapid Intelligent Tutoring Development Shell (RIDES)

RRLOE - Rapidly Reconfigurable Line Oriented Evaluation

SFAR - Special FAR

SID - Standard Instrument Departure

STAR - Terminal Arrival Route

SPOT - Special Purpose Operational Training

TacAir-SOAR - Tactical Air Engagements - State, Operator and Result

TCAS - Traffic Alert and Collision Avoidance System

Executive Summary

This project is part of an overall effort of the Federal Aviation Administration (FAA) to ensure that airplane simulator fidelity is commensurate with its current use for total training and evaluation of airline pilots in the flight simulator. To assess the efficacy of current radio communications simulation and to define the requirement and feasibility of simulating radio communications, research into radio communications simulation was initiated. This report summarizes the first-year effort, including 1) a multi-disciplinary literature review focusing on Advanced Qualification Program (AQP) and crew resource management/cockpit task management (CRM/CTM); 2) a survey of instructor/evaluators (IEs) to determine current practices and to assess their opinions regarding the impact of realistic radio communications on training effectiveness; and 3) a review of the enabling technologies that might support future automated simulation of realistic radio communications.

Simulators used for total training and evaluation of airline pilots must satisfy stringent criteria to assure their adequacy for training and checking maneuvers. Radio communications, however, are currently not a part of these criteria, despite their integral role in every aspect of the flight environment. Ideally, simulation of air traffic control (ATC) communications to own and other vehicles (the so-called partyline) as well as company voice communications should:

• be appropriate for the airspace and responsive to crew actions;
• reflect the demands of ATC-driven timing that does not take pilots' activities or workload into account;
• contain a meaningful partyline that can be used for enhancing situation awareness; and
• expose pilots to communication problems such as communication frequency congestion, blocked frequencies due to stuck microphones, or non-native English-speaking ATC.

The underlying premise of this research is that providing a realistic radio communications environment would have a significant impact on safety by enhancing both pilot training and evaluation. A realistic radio communications environment should provide pilots with opportunities to:

• practice frequency monitoring as well as task and distraction management skills;
• enhance crew resource management skills;
• practice coordination with ATC during new procedures, such as landing on parallel runways which requires an additional controller;
• improve their ability to increase situation awareness from meaningful partyline containing information on traffic and weather;
• ensure their currency in ATC communication procedures and phraseology;
• reinforce attention allocation and differentiation skills necessary to distinguish clearances for one's own airplane from clearances for other aircraft;
• receive more focused attention from the instructor/evaluator;
• avoid an impoverished training environment potentially leading to the development of tunnel vision; and
• reduce the so-called "simulator mindset," which may interfere with transfer of pilot performance and behavior to and from the airplane.

Literature Review

A review of the relevant AQP and CRM literature underscores the importance of realistic scenarios, and that coordination with company and ATC is an integral part of both the AQP task list and CRM. Additionally, a number of Aviation Safety Reporting System (ASRS) reports, research investigations, and industry, military, and general aviation magazine articles show that radio communications directly or indirectly contribute to incidents and accidents. These reports underscore the need to train pilots to practice how to recognize and question inadequate or demanding ATC instructions such as last-minute runway changes. The need to practice correct phraseology, especially during emergencies or with non-native English-speaking ATC is also evident from these studies. The requirement for developing and evaluating CRM skills is repeatedly emphasized in these reports, skills which cannot be adequately trained or assessed if key elements of the CRM scenarios, such as radio communications, are absent.

Instructor/Evaluator Survey

The Instructor/Evaluator Survey polled 29 instructor/evaluators from 14 different airlines, all participating in AQP. All I/Es queried indicated simulating some ATC communications, especially in the terminal environment. For company communications, all but one of the I/Es reported simulating at least some of them, most frequently dispatch. With regard to the partyline, only 38 percent provided any communications to or from other traffic, mainly on the surface. Only two of these provided terminal and en-route partyline information. Most radio communications are simulated by I/E role-play, with few exceptions. Five I/Es indicated using handouts or ACARS (Aircraft Communications Addressing and Reporting System) for Clearance Delivery, and over half reportedly use either printed or recorded (synthetic or natural voice) ATIS (Automatic Terminal Information Service).

Regarding workload in the simulator, I/Es reported that for full mission simulation training and evaluation events, radio communications simulation and managing the simulator systems each consume about 20 percent of their time and effort. At least half of their time and effort is spent observing. When training, they spend 8 percent of their time/effort instructing, versus only 2 percent when evaluating. Most felt that I/E impersonation of communications to own airplane increases the instructor workload and "divides his attention."

On the other hand, most instructors indicated that the current method of simulating radio communications produces an unrealistically low communications load for pilots compared to the real world, especially with regard to company communications. One mentioned that I/E impersonation of radio communications even reduces the manual workload of pilots, because they normally do not need to redial a new frequency to communicate in the simulator. Also, the I/Es think that the presence of meaningful partyline information would increase training effectiveness, by "teaching pilots to listen" and "enabling [I/Es] to assess CRM elements such as workload and distraction."

I/Es indicated that scenarios rely on radio communications simulation to effectively teach such skills as (new) ATC procedures, CRM, and situation awareness. One I/E alluded to the fact that I/E role-play may lead to uneven training and unfair evaluation, since effective radio communications simulation is "all relative to the creative ability of the instructor."

Although some I/Es were skeptical with respect to the success of automating radio communications and its effect on reducing workload, they nevertheless indicated that it would increase training effectiveness. Perhaps due to I/Es' skepticism regarding other kinds of communication automation, ACARS and ATIS received the highest rankings for the anticipated beneficial effect of automation on training effectiveness.

Technology Review

The technology review included:

• United Airlines' Interactive Real Time Audio System (IRAS), also (and too dismissively) known as "Chatter Program"
• AE's Ground and Air Traffic Environment System (GATES)
• Personal Computer Commercial Off-The-Shelf Systems (PC COTS)
• Intelligent Systems
• Voice Generation and Recognition Technologies

Two systems specifically designed for commercial air carrier simulation use were identified: IRAS and GATES. IRAS is a United Airlines in-house development with very high operational realism. It is based on field recordings of actual radio communications on the appropriate routes and includes a meaningful partyline. By dubbing ATC with the respective I/E voice, the I/E can intervene without the pilot trainee realizing it. It was eventually abandoned due to technical difficulties with trigger algorithms and the high cost of scenario and system integration. GATES is a simulator add-on offering a representation of visual and radio communications terminal traffic. It is fully integrated with aircraft movement and other simulation events. Realism is increased, according to anecdotal feedback from some users. It does not, however, provide any communications to or from own airplane.

Radio communications simulation applications at the lower end of the flight simulator market were also identified. The PC COTS flight simulator market clearly perceives a market value for including realistic radio communications add-ons. Flexibility and controllability vary greatly among products. The relevance of the partyline and the ability to send messages to ATC is limited. Generally, these programs are limited to frequency chatter and very limited ATC directives to the simulated aircraft.

Enabling technologies that would support the advancement of realistic radio communications simulation include intelligent systems, and voice generation and recognition technologies. Intelligent systems have the potential for a high degree of communication realism with no instructor intervention; they use rule-based models of goals and behavior of speaker (e.g., controller). Messages are generated based on speaker task goals and situation awareness, thus eliminating the need for complex "triggering" algorithms or complex scenario scripting.

Finally, any successful radio communications simulation system would have to rely on speech generation and recognition systems. Great progress has been made in both fields. Both rule-based synthetic speech and digitized natural concatenated speech achieve intelligibility ratings rivaling natural speech, but voice quality is still an issue especially in a high-workload environment. The main problem for voice recognition system remains speaker variability; even with a highly regulated phraseology such as that used in aviation, extensive "training" of the system is required for reliable recognition of the utterances of a particular speaker.

Conclusions

The following findings are documented in this report:

• The literature emphasizes the need for a realistic training and evaluation environment.
• Subject matter experts perceive realistic radio communications as important.
• Airlines mostly leave the simulation of radio communications to the already very busy individual instructor/evaluators due to cost considerations.
• The technology to simulate ATC/Company and partyline automatically is still immature, although there are some promising emerging technologies.

The underlying premise of this workthat the provision of realistic radio communications during training and evaluation of airline pilots would enhance safetyhas been confirmed thus far. The airlines, however, are unlikely to commit the funds required for the provision of realistic radio communications in the absence of a documented gain in safety and an FAA requirement. The technology development seems to be largely market driven, but would of course be furthered by such a requirement. Therefore, a proof of concept is required that will empirically demonstrate the need for realistic radio communications.

A first step may be to systematically compare the communication load and its effect on pilot workload and CRM behaviors as a function of radio communications during actual operations in the air versus simulator Line Oriented Flight Training/Line Operational Evaluations (LOFT/LOEs). Another step may be an assessment of pilots who have just completed the initial qualification in their first fleet. The question of interest would be how prepared such pilots are to handle ATC and company communications requirements during their first Initial Operating Experience (IOE) flights. This may be followed up by a carefully conceived simulator study examining the benefits of different levels of radio communications realism for different training and event types. Full skill transfer to and from the airplane is a critical issue, if simulator use for training and evaluation is to be mandated.

1. Introduction

1.1 Project Background and Motivation

This project is part of an overall effort of the Federal Aviation Administration (FAA) to ensure that airplane simulator fidelity is commensurate with its current uses. Earlier simulators permitted training and checking of procedures, system knowledge, and navigation, but did not represent the airplane faithfully enough to permit training and evaluation of flying skills and complex decision making. In fact, pilots often explained that they passed their in-flight checks despite previous exposure to those low-level fidelity simulators, indicating the danger of negative training in an unsuccessful simulation of the airplane (Boothe, 2000).

It was not until the advent of the FAA's Advanced Simulation Plan in 1980, authorizing much more extensive airline pilot training and checking credit in the simulator, that technical simulator requirements were specified (Federal Register, 1980, CFR FAR Part 121, Appendix H). Today, total initial, transition, and upgrade training and checking of airline pilots may occur in the simulator, followed by initial operating experience but no further training and checking in the airplane. This has virtually eliminated training accidents since at least all major airlines train and check dangerous maneuvers safely in the simulator. In the near future, it is expected that the FAA will make simulator use mandatory for all training and qualification.

Current simulator qualification criteria ensure that flight performance, handling qualities, and simulator models are suitable for training and checking flying skills in addition to procedural knowledge [FAA AC120-40A, B, and C (draft)]. With both the growing complexity of flight deck automated systems and the concurrent elimination of the flight engineer position, however, increased emphasis is being placed on training and evaluation of system management and crew resource management (CRM) skills. This is accomplished through the use of LOFT (Line Oriented Flight Training) and LOE (Line Operational Evaluation). LOFT and LOE both require high fidelity flight simulators and carefully constructed realistic scenarios designed to present to crews a series of events that require crew coordination and system management skills. FAA's Advanced Qualification Program (AQP), an alternative operations personnel qualification program, emphasizes full mission simulations containing complex scenarios involving contingencies that may contribute to accidents and incidents (Federal Register, 1990, CFR Part 121 SFAR 58). It is primarily in the context of AQP requirements that the question arises as to what level of fidelity the operational environment >must be simulated to ensure full transfer of pilot performance and workload, from the simulator to the airplane for training and vice versa for evaluation.

For this purpose, realistic simulation of air traffic control (ATC) and company radio communications appears to be essential. However, the simulation of such communications seems to be often left to the instructor/evaluator's (I/E) role-play. This potentially increases I/E workload, thus reducing their ability to observe the crews and operate the simulator. At the same time, it potentially decreases the workload of pilots compared to what they would experience in actual flight operations. This latter effect in turn may reduce the realism and effectiveness of both pilot training and evaluation. Moreover, I/Es may vary in the way they simulate radio communications for a particular scenario, leading to variations in the complexity of the environment that crews are exposed to. This may result in uneven training and unfair evaluation. This lack of standardization was pointed out by an I/E who remarked that the provision of radio communications is "all relative to the creative ability of the instructor."

1.2 Project Goals

This project addresses the requirement for and feasibility of providing a realistic radio communications environment during full mission simulations in an at least partially automated fashion. Realistic radio communications include air traffic control and company voice communications to own airplane, as well as ATC communications to and from other aircraft controlled by the same controller via a shared communications frequency (i.e., partyline). Such communications would have to be appropriate for the airspace and responsive to crew actions. They should reflect normally occurring ATC-driven timing that cannot take pilots' activities or workload into account, as an instructor/evaluator who is aware of crew activities might do. They also should contain a meaningful partyline that pilots can use to enhance their traffic and weather awareness. Frequency monitoring is important for pilots waiting for an opportunity to be heard on a congested frequency, helping to avoid interrupting an ongoing exchange by keying the microphone. Other communication problems common in the air that may merit simulation include pilots that are unaware that their activated microphone is blocking an entire frequency, non-standard FAA phraseology in airspace using International Civil Aviation Organisation (ICAO) phraseology, or dealing with non-native English-speaking controllers. A recurring problem when flying in international airspace is the use of non-FAA units such as kilometers instead of nautical miles, kilograms instead of pounds of fuel, pressure in millibars instead of mmHg, or the transition altitude for barometric corrected altitude instead of standard pressure (Flight Levels). But even when simulating FAA controlled airspace, communication problems such as complex instructions at a high speech rate, non-standard phraseology, failure to correct pilots' incorrect responses to a clearance, callsign confusions, and even dealing with conceptual errors by controllers should be practiced (Bürki-Cohen, 1995; Cardosi, 1993; Rodgers & Nye, 1993).

The underlying premise of this research is that providing a realistic radio communications environment could have a significant impact on safety by enhancing both pilot training and evaluation. A realistic radio communications environment would provide pilots with opportunities to enhance their time-sharing as well as task and distraction management skills. It would also improve pilots CRM and coordination with company and ATC. The latter may be especially important when training or evaluating new procedures such as landing on parallel runways involving an additional controller. A meaningful partyline containing information on traffic and weather should reinforce careful listening to improve situational awareness. Realistic radio communications also help ensure currency in ATC communication procedures and phraseology, as well as attention allocation and differentiation skills necessary (e.g., distinguishing clearances for one's own airplane from clearances for other aircraft). Automated provision of radio communications should free instructors/evaluators to focus more of their attention on instructing and observing the crew without the competing task of providing radio communications. Perhaps most important, the provision of realistic radio communications may help avoid exposure to an impoverished environment potentially leading to the development of tunnel vision, where pilots focus on aviating skills alone while losing situational awareness and multitasking skills. The overall gain in realism may help pilots to rid themselves of the so-called "simulator mindset," which could interfere with transfer of pilot performance and behavior to and from the airplane.

This report documents the first-year efforts in this project, which focused on radio communications without considering advanced datalink technologies. Once such technologies are implemented, certain aspects of this work regarding the means of information exchange with ATC and company as well as the content of the partyline will have to be reconsidered. For the near future, however, voice communications via congested approach and terminal radio frequencies are likely to remain the dominant form of air-ground information exchange. First, a multi-disciplinary literature review was conducted to determine the need for realistic radio communications from a theoretical perspective. The main findings of this review are included in the body of the report; Appendix A provides additional information on the role of ATC communications in accidents and incidents and Appendix B elaborates on both the need and the methods of task management training. Second, a survey of I/Es determined whether and how airlines are currently simulating radio communications. The same survey also collected I/Es opinions on the impact of realistic radio communications simulation. Third, a review of technologies that may support the provision of realistic radio communications was conducted. The results and conclusions from these efforts are described in detail in the following sections.

2. Literature Review

The literature review focused on the relevant aspects of AQP and CRM to realistic radio communications simulation. Although communication skills are critical in achieving technical and procedural objectives as well as CRM objectives in flight, most of the communication skills literature focuses on CRM (Kanki and Smith, in press). Therefore the following section reviews AQP and CRM in general, CRM between the flight crew and ATC, and "task management" as a specific example of how realistic radio communications simulation can enhance CRM training.

AQP and CRM are closely connected. One of the goals of AQP is to integrate training and evaluation of CRM skills at each stage of the curriculum. Qualification and continuing qualification curricula are required to include an LOE, consisting of a full flight scenario systematically designed to target specific technical and CRM skills. Demonstration of proficiency in maneuver-oriented technical skills is a "necessary but insufficient condition for pilot qualification" (Longridge, 1997, emphasis added). CRM training and evaluation addresses human factors such as leadership, communication skills, time management, situational awareness, and attitudes in flight operations (FAA, 1990, AC120-35B, Crew Line Operational Simulation, LOS). It is an integral part of any AQP, and both the AQP and CRM literature underscore the importance of realistic scenarios.

2.1 Advanced Qualification Program

The founding principle of AQP is that training and evaluation of pilots should be based on the activities encountered on the operational job. A first step in AQP is thus an analysis of all tasks that a pilot needs to perform during actual operations, which then guides curriculum and scenario development in the simulator. Each task is given a label describing what type of skill is required to perform it. The AQP task-listing example found on the FAA AQP Management Web site (http://www.faa.gov/avr/afs/aqphome.htm), clearly shows that coordination with company and ATC is an integral part of line operations and that frequency monitoring is important for maintaining traffic and weather situational awareness. Most activities related to radio communications are coded as "attitudes." Those that are part of situation assessment are labeled "cognitive skills," and tuning the radio is considered a "psychomotor skill."

Communications with dispatch and other "appropriate agencies (i.e., weather, AFSS [Automated Flight Service Station], maintenance, company)," in-person or via telephone, start during pre-departure ground mission preparation. After all the radios have been prepared for flight, the final cockpit preparations include communications with company and maintenance and obtaining departure ATIS (Automatic Terminal Information Service) and ATC clearance followed by Pushback/Start Clearance via radio/ACARS (Aircraft Communications Addressing and Reporting System). The pushback procedure includes interphone communication with the ground crew, followed by the taxi clearance procedure via ACARS/radio. Finally, ATC take-off clearance is obtained. This completes routine communications tasks on the ground, unless the "assessment of the take-off environment" leads to an adjustment to the original ATC instructions.

While the requirement for assessing the take-off environment (weather and airfield conditions) is explicitly spelled out as part of the take-off performance, there is no mention of frequency or partyline monitoring as a means of obtaining information for situation awareness in the terminal environment. Situation awareness has been described as the pilots knowledge of the environment at a given point in time, and more specifically, of which subsets of the environment are important for the task at hand (Endsley, Farley, Jones, Midkiff, and Hansman, 1998). At any stage of operations, pilots need to be aware of the surrounding traffic, terrain, weather, and status of aircraft systems. To arrive at the desired destination, they also need to be aware of their position. With the exception of system awareness, ATC clearances to own and other aircraft play a critical role in maintaining situational awareness. In the AQP task list, frequency monitoring is first alluded to in the section "Perform Enroute Situational Assessment" for the acquisition of weather awareness ("Monitor weather using cues fromradio communicationsand ACARS reports.") In subsequent phases, reference is made to cues from ATC both for weather and maintaining traffic separation.

Throughout the next phases, routine radio communications (or declaration of emergencies) with the appropriate controlling ATC agencies continue to be listed, including tuning the radio to the appropriate frequency. En-route communications procedures warrant their own section, including subsections describing cockpit and cabin crew as well as company and ATC communications. Of these, cockpit crew communications take the lead with 16 items listed, followed by five company communications items (including ACARS). The cabin crew and ATC communications subsections contain three items each. During subsequent phases, cockpit crew communications continue to lead, followed by ATC and company (radio/ACARS). In short, coordination with company and ATC as well as maintaining situational awareness form an integral part of the AQP task list (FAA 2000), and messages from ATC are considered an essential ingredient to ensuring the realism of full mission simulations (FAA, 1990, AC120-35B, Crew Line Operational Simulation, LOS).

2.2 Crew Resource Management

The importance of CRM started to be recognized in the late 1970's, when the human error aspects of airplane crashes were identified as failures of interpersonal communications, decision making and other leadership and crew coordination skills (Helmreich and Foushee, 1993; Kayten, 1993). Originally standing for "Cockpit Resource Management," early CRM programs focused on cockpit group dynamics. As some airlines started to extend their CRM efforts to other groups involved in airplane operations, the phrase "Crew Resource Management" was adopted.

The Advisory Circular on Crew Resource Management Training explicitly lists not only onboard cabin personnel, but also ground-based maintenance personnel, aircraft dispatchers, and air traffic controllers as part of the CRM process (FAA, 1998, AC120-51C, Crew Resource Management Training, CRM). In a section on extending training beyond the cockpit, the advisory circular highlights the benefits of using real air traffic controllers, aircraft dispatchers, and maintenance personnel during LOFT sessions. A 1996 Air Line Pilot article on flight deck confusion concluded, from a survey of recent Aviation Safety Reporting System (ASRS) reports explicitly mentioning pilot confusion, that better communications between cockpit crews and ATC controllers could help prevent confusion-related incidents (Rosenthal, Chamberlin, and Matchette, 1996).

Communication deficiencies included lack of compliance with standard phraseology and communication protocols. Thirty of the 100 reports involved amended clearances, and frequently pilots refrained from asking controllers for clarification of a confusing clearance. Rosenthal et al. state that pilot confusion can "best be prevented through continuing emphasis on crew performance, with the understanding that ATC is a key member of the flight team," and recommend that CRM place greater emphasis on crew/ATC interactions. There are many other ASRS reports (see Billings & Cheaney, 1981; Connell, 1994, 1996), research investigations (see Kanki and Palmer, 1993; Predmore, 1991), and industry, military and general aviation magazine articles that discuss the role of communication in incidents and accidents. Numerous industry and government working groups have also focused reviews on operational communications and their ongoing problems in the system.

2.2.1 Approach and Landing Accident (ALA) Reduction Task Force Report

A recent review of approach and landing accidents by the Flight Safety Foundation again pointed to the importance of efficient interactions with ATC and company (Khatwa & Helmreich, 1999). The task force based its conclusions on an analysis of 76 ALAs and serious incidents involving jet and turboprop aircraft and occurring between 1984 and 1997 (inclusive). "Incorrect or inadequate ATC instruction/advice/service" was a causal factor in 33 percent of all occurrences. It ranked eleventh among the most common causal factors after "Poor professional judgment/airmanship" in first, "Lack of positional awareness" in fourth, and long before "Interaction with automation" in seventeenth place. "[D]emanding ATC clearances" are also explicitly mentioned in context with even higher ranking causal factors such as the sixth-placed "Flight-handling difficulties," the eighth-placed "Press-on-itis," or the tenth-placed "Slow and/or low on approach." In many cases of "press-on-itis," "a breakdown in CRM between the flight crew and ATC" was observed. Further description of results of this and other studies supporting its findings, including a discussion of problems involving non-native English speaking ATC or crews, can be found in Appendix A.

2.2.2 Realistic Radio Communications for Task Management Training

Chou et al. have investigated the significance and nature of the task management process in flight operations by first conducting a review of National Transportation Safety Board (NTSB) accident and ASRS incident reports and then testing their findings in a controlled simulator experiment (Chou, Madhavan, and Funk, 1996). They describe cockpit task management (CTM) as consisting of the following seven functions: 1) task initiation when appropriate conditions exist; 2) task monitoring; 3) task prioritization; 4) resource allocation; 5) task interruption when resources need to be allocated to a higher priority task; 6) task resumption when priorities change or resources become available; and 7) task termination when task has been completed, cannot be completed, or has become irrelevant.

From this functional description, it is clear that many of these functions implicitly refer to intra-cockpit communications that serve to achieve crew coordination, resource management and task prioritization. However, it is also easy to see how these functions depend upon direct information transfer between cockpit crew, ATC and company, and indirect information gained from frequency monitoring.

However, there is substantial evidence from accident and incident data as well as research that the capacity to capture and use information is limited. Furthermore, this limitation can lead to human error. For instance, radio communications were found to be one of the primary contributors to monitoring errors (Sumwalt, Morrison, Watson, and Taube, 1997). In this study, 200 ASRS reports (submitted between 1992 and 1996) were reviewed for monitoring errors which were defined as a "failure to adequately watch, observe, keep track of, or cross check" the aircraft's trajectory or systems. Consequences encompassed a wide range of safety problems, including taxi, flight path, or speed deviations, runway incursions, system damage, and loss of control. For the 170 reports that indicated the tasks or functions accomplished shortly before or during the error, 42 percent of the reports involved radio communications (ATC, company, or obtaining ATIS). (See fuller description of this and other studies in Appendix B.)

In order to train and evaluate pilots on task management skills, significant information resources are omitted when radio communications are not simulated in a realistic manner. A similar but distinct issue is the role of radio communications as a task distractor. That is, pilots should be trained to maximize their use of radio communications as an information source (especially under non-normal operational or procedural conditions), but they also should be trained on interruption/distraction management skills within a realistic ATC training environment (see Latorella, 1996). Also see Appendix B for a continued discussion of Task Management Training Methods, including a discussion of whole-task versus part-task training.

2.3 Summary

As an example of one type of CRM skill, the survey of the task management literature shows how important it is for pilots to be exposed to all aspects involved in flying an airplane. One reason why whole-task training in a fully loaded environment including radio communications may be superior to part-task training in an incomplete environment is that it may avoid the false sense of operational simplicity that comes with tunnel vision. Tunnel vision implies that pilots are trained to concentrate on certain aspects of the flying task while learning to ignore others. To avoid tunnel vision, a complete mental model of the task needs to be acquired during training. As Mangold and Eldredge (1993) put it, the mental model serves as a structure for understanding the task and developing expectations as to the events that will occur.

Bransford and Franks (1976) subscribe to a stage-setting metaphor of learning, where past experiences prepare the ground for dealing with novel situations. Learning is an active process, and practice can lead to either activation or inhibition of cognitive pathways. If pilots are consistently exposed to an impoverished environment during training compared to the real world, they may end up unprepared for the complexity of the flying task in the air. This was also confirmed by one of the participants in the I/E survey, who wrote, "without communication simulation, when the pilot trainee finally arrives in the real world,' he must add another component. This new component can really complicate line flying."

3. Instructor/Evaluator Survey

A questionnaire on realistic radio communications practices and opinions was presented to the AQP LOFT Instructor/Evaluator Focus Group in May 1999. A few additional questionnaires were sent to airlines over the succeeding months. A total of 36 responses were received. Six of these reported on training events for non-scheduled aircraft that are not representative of Part 121 operations or AQP. One additional questionnaire was excluded because the respondent indicated that he did not understand the distinction between clearances to own airplane and partyline. The 29 responses included in the final analysis stemmed from seven major, one cargo, four regional, and two foreign (AQP) airlines. Respondents averaged some 10 years of pilot instruction and evaluation.

3.1 Current Practices of Radio communications Simulation

Instructor/Evaluators were queried on their company's current practices of simulating air traffic control communications and company communications to own and, for ATC, other aircraft (the so-called partyline). They were also queried regarding their company's visual and Traffic Alert and Collision Avoidance System (TCAS) simulation of the traffic environment.

3.1.1 Radio Communications Simulations to Own Airplane

The first question asked I/Es to indicate with a checkmark, which types of communications to own airplane their airline simulated during different training and evaluation events (Appendix C, Question 1). The question was subdivided into ATC and company communications. The second question, again subdivided into ATC and company communications, asked I/Es to indicate how these communications were implemented.

Communications Simulated During LOFT

Twenty-seven answers on communications during LOFT were gathered from the 29 respondents (see Table 1). One I/E did not answer this question, ¹ and another indicated that he is only familiar with LOEs and SPOT (Special Purpose Operational Training).

With regard to ATC communications, all of the LOFT respondents indicated simulating communications from the tower ground, tower local, and approach/departure controller. Twenty-four (89 percent) of respondents reported simulating Automatic Terminal Information Service (ATIS). Twenty-five respondents (close to 93 percent) reported simulating tower clearance delivery and air route traffic control center (ARTCC) communications.²

Only one of the 27 LOFT participants indicated that he did not simulate any company communications (see Table 2). All but one of the remaining 26 respondents (93 percent of all LOFT participants) reported simulating communications with dispatchers. Nineteen (70 percent) indicated simulating communications with ramp or gate personnel. Eighteen respondents (67 percent) indicated simulating cabin personnel communications. In the field "Other" 18 I/Es (67 percent) entered "maintenance," one "ground crew," one "passenger service," and another "emergency vehicles" (4 percent each).

 

Table 1. Types of ATC Communications Simulated in LOFT, LOE, and SPOT

 

Table 2. Types of Company Communications Simulated in LOFT, LOE, and SPOT

Communications Simulated During LOEs

Nineteen responses were available regarding communications during LOEs (also shown in Table 1). In addition to the one skipped question, nine respondents apparently do not perform LOEs.³

With respect to ATC, all participants in LOE indicated simulating some ATC communications. All respondents checked simulating tower ground, tower local, approach/departure and ARTCC communications. Eighteen respondents (95 percent of all LOE participants) reported simulating ATIS. Seventeen (89 percent) reported simulating clearance delivery information.

Regarding company communications (Table 2), one of the 19 respondents reported simulating no company communications. Eighteen respondents (95 percent) reported simulating dispatch. Thirteen respondents (68 percent) indicated simulating cabin personnel communications. Twelve (63 percent) I/Es checked the ramp/gate communications box. In the "Other" box, 16 respondents (84 percent) wrote in "maintenance," one "ground crew," and another "emergency vehicles" (5 percent each).

Communications Simulated During SPOT

Eighteen I/E responses were available regarding communications during SPOT (see Table 1). In addition to the one I/E that omitted his answer, 10 respondents indicated that they did not participate in SPOT.⁴

All I/Es participating in SPOT simulate tower local communications. Seventeen respondents (94 percent) simulate tower ground and approach/departure communications. Sixteen (89 percent) simulate center communications. Thirteen (72 percent) simulate ATIS and clearance delivery.

Company communications during SPOT (see Table 2) are simulated by 15 (83 percent) of the respondents. Fourteen (78 percent) respondents checked the company dispatch box. Ten (56 percent) indicated simulating ramp/gate as well as cabin communications. Eleven respondents (61 percent) wrote in "maintenance" and one "ground crew" in the space provided in the "Other" box.

Methods of Simulating Radio Communications to Own Airplane

Alternative methods (shown in Appendix C, Question 2) included:

• "I/E provides" all radio communications
• "Live controllers' " (i.e., "an additional staff member focuses on radio communications")
• "Recorded controller voice,"
• "Synthetic controller voice,"
• "Other"

One I/E skipped the question on communication simulation methods to own airplane, leaving 28 respondents for this question.

All but one I/E indicated that they themselves provided all company communications via role-play. The one exception checked "Other" for dispatch and indicated in the comment section that many communications between company and the crews were provided by ACARS. He was the only I/E representing his major airline.

With respect to ATC communications, the vast majority appears to be provided via I/E role-play as well. Respondents indicated that this was the case for all tower ground, tower local, approach/departure, and ARTCC communications provided.

Tower clearance delivery and ATIS were the only type of communications to own airplane where the I/Es did not always indicate having to role-play communications. Five I/Es (19 percent of all I/Es indicating their method of clearance delivery) indicated provision of clearance delivery in text-based format (printed handouts, ACARS). A majority of 23 (82 percent) however still had to provide this information themselves (presumably via oral communication).

Twenty-seven I/Es responded to the question of ATIS implementation. Twelve (44 percent) indicated that they had to provide this information themselves. Fifty-six percent indicated some supporting system which may depend on the capabilities of a particular simulator. Four pilots checked the "I/E provides" as well as the "Synthetic controller voice" or the "Recorded controller voice" box. Six indicated synthetic and two recorded natural controller voice. Three wrote in "printer paper" or "handouts" in the space provided for "Other."

Three of the seven respondents from one airline indicated that two instructors may participate in a training session, at least when instructing/evaluating a three-person crew⁵. This gives them the opportunity to share communication duties. One I/E specified that the instructor for the pilots (Captain and First Officer) provides the ATC radio communications, whereas the instructor for the flight engineer provides the company communications.

3.1.2 Simulation of Other Aircraft

The next set of questions (Appendix C, Questions 4 and 5) addressed the simulation of other traffic. Dependent on phase of flight, this may include aircraft and ground vehicles. Other traffic may be simulated both in the visual and the auditory modality. Visual simulation may include representations of other traffic in the out-the-window view and on the TCAS. Auditory simulation of other traffic consists of hearing communications from and to other aircraft on the same air traffic control radio frequency, the so-called partyline.

The first question (Appendix C, Question 4) provided boxes to be checked to indicate the type of vehicle simulated (aircraft or ground vehicle) in each modality (out-the-window view, TCAS, communications to and from other aircraft or ground vehicles) and for each environment (airport surface, terminal, or en-route). The next question (Appendix C, Question 5) provided check boxes regarding the implementation of communications with Clearance Delivery, Tower Ground, Tower Local, and Approach/Departure control. The proposed implementations were "Not simulated," "I/E provides," "Live controllers'," "Recorded" and "Synthetic controller voice," and "Other."

Five I/Es left both questions about partyline completely blank. All these (17 percent of all participants) explicitly commented that no visual simulation of other traffic, either out-the-window or via TCAS, was provided. So they did not provide any communications simulation either.

Table 3. Simulation of ATC Communications with Other Aircraft or Vehicles (Partyline)

Table 3 gives results for type of vehicle simulated in each modality and environment (Appendix C, Question 4). Seventeen (59 percent) of the overall I/Es reported some out-the-window view simulation of traffic. On the airport surface, 14 checked simulation of aircraft (A/C), and seven of these also reported simulating emergency vehicles (E/Vehicle). One additional I/E reported out-the-window simulation of only emergency vehicles on the surface. Eight reported simulating out-the-window aircraft in both the terminal and en-route traffic environment. Two of these (from the same airline) did not indicate any visual simulation of airport surface traffic. Two indicated simulating terminal but not en-route out-the-window aircraft, and two en route but not terminal. The latter four had indicated simulating airport surface traffic in the out-the-window view. Only eight of the 17 I/Es reporting out-the-window simulation of traffic also reported simulating communications to or from other aircraft.

Ten I/Es (34 percent) reported that the simulators they use were TCAS equipped, but two of these indicated that it was available only on the airport surface. All but one of these report some out-the-window view simulation.

Eleven (38 percent) of all I/Es reported providing some communications to or from other aircraft or vehicles, at least on the surface. Eight of these provide communications both ways, two only to and one only from other traffic. On the surface, eight (28 percent) reported providing clearances to aircraft. Two of these reported providing instructions also to ground vehicles (as do two additional ones that do not provide any clearances to aircraft). Of the I/Es providing clearances to other aircraft, five reported that they also simulate pilot readbacks. One additional respondent indicated that he provides communications only from aircraft and vehicles on the airport surface. All the I/Es simulating instructions to ground vehicles report simulating responses from the drivers. Only two respondents indicated that in addition to simulating surface communications, they simulate clearances and readbacks in the terminal and en route environment. One I/E explained that they seldom provide terminal and en route communications due to I/E workload.

Eighteen I/Es indicated a method for providing communications to other aircraft (Appendix C, Question 5).⁶ Most indicated that they simulated communications via role-play, with few exceptions. One respondent checked the "Other" category and commented that two instructors alternate communications. Another checked "Recorded controller voice" for tower ground and tower local and Approach/Departure control communications. For tower ground and local, one last respondent checked, in addition to the "I/E provides" box, the "Synthetic controller voice" box. Follow-up discussions with this instructor revealed that he was referring to the availability of GATES (Ground and Air Traffic Environment) on some simulators, a new technology that will be discussed in the Technology Review.

3.1.3 Instructor/Evaluator Workload in the Simulator

Instructor/Evaluators were also asked, for each simulation event type, to indicate the "percentage of time and effort" spent "Running (the) Simulation," "Simulating Radio Communications," "Instructing," "Observing," and "Other" with the indication that the numbers should add up to 100 percent (Appendix C, Question 8a).

All 29 I/Es tried to answer this question. However, several I/E answers were omitted due to inconsistencies in their responses. In addition, I/Es omitted answers for some but not all training venues, LOFT, LOE or SPOT. This resulted in 24 answers for LOFT, 16 for LOE, and 15 for SPOT for final analysis.

Only three I/Es entered any information into the "Other" category. One indicated that for LOFT and LOE, he spent 1 percent of his time/effort providing "cabin, company communications." Two indicated that they spent some time "[t]aking notes." One of these answered only for LOE and SPOT and indicated spending 20 percent for both events, the other, who answered only for LOFT, indicated spending 5 percent of time/effort taking notes. The 20 percent notetaker was the only representative of his airline. So as not to distort the final picture on the basis of these three respondents, the category "Other" was eliminated by distributing the percentage of time/effort found in it proportionally over the four other categories (running simulation, radio communications, instructing, and observing) shown in Figure 1.

Figure 1 shows perceived time/effort allocation averaged across I/Es for each training event. As can be seen, during all three events, I/Es spend most of their time/effort observing, especially during LOE and LOFT (52 and 51 percent, respectively). Not surprisingly, this percentage drops down to 37 percent for SPOT, which involves "special purpose" training and shows the highest percentage for instructing, namely, 25 percent versus 8 percent for LOFT. Of course, this percentage is lowest for LOE with 4 percent. Running the simulator takes the same amount of time/effort in all three events, i.e., 22 percent. Radio communications simulation takes slightly more time and effort during evaluation, 22 percent versus 20 percent in LOFT. During SPOT, radio communications appear to be less important, with the percentage of time/effort spent on them falling to 16 percent.

These percentages are important because they depict how I/Es provide radio communications while performing significant other tasks. One possible exception may be the three I/Es from one airline instructing/evaluating a three-person cockpit and with support from a second I/E for the flight engineer. Although these reported slightly lower percentages for radio communications in LOFT (18 instead of 20 percent),⁷ the numbers are too small and possibly offset by the extra effort required for instructing a three-person crew to allow any conclusions.

3.2 Survey of Instructor/Evaluator Opinion

Instructor/evaluators were polled on their opinions on the effect of having to simulate radio communications on workload and training effectiveness, as well as on the importance of simulating the partyline. They were also asked for their opinions on the overall importance of realistically simulating radio communications in scenarios.

3.2.1 Radio Communications and Workload

Instructors/evaluators were asked how their companies' radio communications practices affected I/E and pilot workload in the simulator compared to instructing/evaluating pilots in the air with real radio communications (to own airplane, see Appendix C, Question 3). For each type of communication, ATC ATIS, Clearance Delivery, and Company Dispatch, Ramp/Gate, Cabin, and Other, I/Es indicated on a scale from "1" to "5" whether the workload was "Lower" (1), "Same" (3), or "Higher than A/C" (5).

ATIS was the only type of information where the majority of I/Es indicated that it was provided by some kind of supporting system, depending on the simulator used. This could consist of either recorded or synthetic controller voice or printed handouts. Since the focus in this investigation is the workload caused by I/Es having to role-play oral radio communications, the workload perceptions of I/Es for ATIS are not included in the analysis below.

Effect of Radio Communications (To Own Airplane) Role-Play on Perceived Instructor/Evaluator Workload

Effect of ATC Communications Role-Play on I/E Workload

Tower Ground, Tower Local, Approach/Departure and ARTCC ATC communications are almost exclusively simulated by the I/Es alone, with the exception of three I/Es from one airline that are supported by an additional (flight engineer) I/E. These three were excluded from the analysis below, leaving 26 responses (if all others provided an answer).

• For Clearance Delivery, only the responses from 21 I/Es that still have to role play this information alone were included for the effect of ATC communications role-play on I/E workload. The average I/E workload rating was 3.71, with 14 I/Es (67 percent) judging it to be higher, six (29 percent) the same and one (5 percent) lower than in the airplane (see Table 4).

Table 4. Effect and Importance of ATC Communications

Communications with ATC
Compared to Actual Aircraft Operations:	Clearance Delivery	Tower/ Ground	Tower/ Local	Approach/ Departure	Center
Instructor Workload	3.71 n = 21	3.65 n = 26	3.64 n = 25	3.81 n = 26	3.84 n = 25
Pilot Workload	2.86 n = 21	2.46 n = 24	2.60 n = 25	2.56 n = 25	2.50 n = 24
Overall Importance in Simulation	3.39 n = 28	3.86 n = 28	3.77 n = 26	3.92 n = 26	3.69 n = 26

 

• For Tower Ground radio communications, four (15 percent) I/Es indicated that the I/E workload was lower than in the airplane. Seventeen (65 percent), however, felt that the I/E instructor workload was higher than in the airplane. Five perceived no difference between airplane and simulator workload. The average response was 3.65.

• For Tower Local communications, the average of 25 responses (one I/E did not circle anything) was 3.64. Four I/Es (16 percent) felt that the workload in the simulator was lower in the simulator, while 17 (68 percent) felt that it was higher. Four did not perceive any difference.

• For Approach/Departure, the average of 26 responses rose to 3.81. Three (12 percent) indicated lower and 18 (69 percent) indicated higher workload than in the airplane. Five circled "3" for "same."

• The perceived I/E communications workload compared to actual operations was rated slightly higher for communications during the en route phase. The average was now 3.84, with only two (8 percent of 25 respondents) indicating lower communications workload than in the airplane. Eighteen (72 percent) indicated higher and five, the same workload.

Effect of Company Communications Role-Play on I/E Workload

With the exception of company dispatch, where one I/E had indicated that it was provided via simulated ACARS, all company communications were reportedly provided via role-play. These included, in addition to Dispatch for most I/Es, also Ramp/Gate, Cabin, and Other. The three I/Es getting support from an additional I/E were again excluded.

• For Dispatch, the average of the 24 ratings was 3.75. Eighteen (75 percent) respondents indicated that I/E communications workload was higher than in the airplane. Three indicated that it was the same, and three that it was lower (12.5 percent each, see Table 5).

Table 5. Effect and Importance of Company Communications and Partyline

Communications with Company					Other Aircraft
Compared to Actual Aircraft Operations:	Dispatch	Ramp/Gate	Cabin	Other(Mx)	Partyline
Instructor Workload	3.75 n = 24	3.38 n = 24	3.61 n = 23	3.82 n = 11	4.18 n = 17
Pilot Workload	2.83 n = 24	2.71 n = 24	2.83 n = 23	2.09 n = 11	2.88 n = 17
Overall Importance in Simulation	3.46 n = 28	3.15 n = 26	3.82 n = 23	3.80 n = 10	3.60 n = 24

 

• For Ramp/Gate, the average of 24 responses was 3.38. Thirteen (54 percent) perceived the workload as higher, seven (29 percent) the same, and four (17 percent) as lower.

• For Cabin communications, the average I/E workload rating of 23 responses was 3.61. Sixteen (70 percent) perceived the I/E workload as higher, three (13 percent) as the same, and four (17 percent) lower.

• Eleven I/Es circled a rating for "Other," ten writing "Maintenance" in the space provided and one leaving it blank. The average rating was 3.82, with eight (73 percent) circling a rating that was higher, two (18 percent) as lower and one I/E (9 percent) that the workload in the simulator was the same as in the airplane.

Effect of Radio Communications (To Own Airplane) Role-Play on Perceived Pilot Workload

While radio communications role-play may increase I/E workload, any lack of realistic radio communications may decrease pilot workload (Appendix C, Question 3).

Effect of ATC Communications Role-Play on Pilot Workload

• For Clearance Delivery, again only the responses from the 21 I/Es that still have to role-play this information alone were included for the effect of ATC communications role-play on pilot workload. The average pilot workload rating was 2.86, with two (10 percent) of the I/Es judging it to be higher than in the airplane, the rest lower (four or 19 percent) or the same (15 or 71 percent, see Table 4).

• In comparison to actual operations, Tower Ground pilot communications workload was perceived as lowest. Ten of 24 I/Es answering this question perceived pilot workload to be lower in the simulator than in the airplane (42 percent). Thirteen (54 percent) perceived it to be the same, and only one perceived it to be higher. The average of the responses was 2.46.

• Tower Local communications workload was rated by 25 I/Es, resulting in an average pilot workload of 2.60. Eleven (44 percent) felt that the pilots' workload is lower in the simulator. Twelve (48 percent) perceived it to be the same. Two respondents discerned it to be higher.

• For Approach/Departure, the average I/E perception of pilot workload was 2.56. The distribution of responses was identical to Tower Local, with the same I/Es perceiving pilot workload as lower, same, or higher.

• With an average of 2.50, the perceived pilot communications workload compared to actual operations was second lowest for the en route phase. The same 11 I/Es that perceived pilot workload as lower for Tower Local and Approach/Departure said so again for ARTCC communications. Twelve perceived it as the same, and one as higher.

Effect of Company Communications Role-Play on Pilot Workload⁸

• For Dispatch communications, the average workload rating of 24 responses of I/Es providing dispatch information via role-play was 2.83. Whereas 14 (58 percent) of I/Es perceived the pilot workload for handling these communications to be the same in the simulator as in the airplane, seven (29 percent) discerned it to be lower. Three (13 percent) perceived it as higher.

• For Ramp/Gate communications, the average of 24 responses was 2.71. Seventeen I/Es (71 percent) circled the number "3" indicating that pilot workload in the simulator was the same as in the airplane. Six (25 percent) felt that pilot workload was lower and one (4 percent) that it was higher than in the airplane.

• For Cabin communications, the average I/E perception of pilot workload of 23 responses was 2.83. Twelve (52 percent) perceived the pilot workload the same as in the airplane. Seven (30 percent) perceived the pilot workload to be lower, while four (17 percent) perceived it as higher.

• For the eleven responses to workload for "Other" company communications (mainly maintenance), the average rating was 2.09. Eight I/Es (73 percent) perceived pilot workload to be lower and three (27 percent) that it was the same.

Effect of Partyline Communications Role-Play on Workload

I/Es were asked to compare the effect of their "company's partyline simulation practices" on I/E and pilot workload to workload "in the air with a real airplane" by circling a number on a scale from "1" to "5". Again, (1) was labeled as "Lower," (3) as "Same," and (5) as "Higher" (see Appendix C, Question 6).

Four I/Es reported that they did not simulate the "partyline," and four additional I/Es did not answer this question. After exclusion of the three I/Es receiving support from other instructors and one I/E who reported provision of a recorded partyline (presumably GATES, see Technology Review), 17 respondents remain, whose responses are summarized below.

• For perceived I/E workload, the average is 4.18, with 76 percent or 13 of the respondents perceiving it as higher in the simulator than in the air. Three (18 percent) perceive it as the same and one (6 percent) as lower (see Table 5).

• Pilot workload was perceived to be slightly lower than in the air, with an average rating of 2.88. Six I/Es (35 percent) each perceive it to be lower or the same. Five (29 percent) perceive it to be higher.

3.2.2 Overall Importance of Simulating Radio Communications for Training Effectiveness

Two more questions tapped I/Es perception of the importance of realistically simulating radio communications. One asked about the importance of radio communications in scenarios aiming at training specific types of skills such as ATC procedures, CRM, distraction management, and situation awareness (Appendix C, Question 7). The other asked about importance of each type of ATC and company communications in the context of overall training effectiveness (Appendix C, Question 3).

Importance of Radio Communications in Scenarios

I/Es were asked to indicate how frequently they "rely on radio communications as part of the scenario context for team performance" for each type of skill indicated below. The scale ranged from "Never" (1) to "Sometimes" (3) to "Often" (5). They were also asked, for each skill, to indicate the importance of radio communications for effectively training this skill, on a scale ranging from (1) "Very Low" to (5) "Very High."

All I/Es offered their opinion on some of these questions. As shown in Table 6, the frequency with which skills listed were viewed as relying on radio communications fell between "sometimes" and "often" (ranging from 3.58 to 3.89). In addition, radio communications were consistently perceived as important for effective training scenarios (ranging from 3.77 to 4.17) with little variation across skills.

Table 6. Frequency and Importance of Radio Communications for Skills Training

• For "New ATC Procedures, e.g., PRM" (Parallel Runway Monitored Approach), the average frequency rating of the 26 I/Es responding was 3.58. Twenty-four (92 percent) of I/Es circled "3" indicating sometimes or higher. Only two indicated that this type of skill rarely relies on radio communications as part of the scenario by circling "2". The average rating for the importance of radio communications for training new ATC procedures effectively was 3.77, with 16 I/Es (62 percent) indicating the importance as high (4) or very high (5). Eight (31 percent) indicated "3" and two (8 percent) circled "2".

• For "Non-Routine ATC (e.g., pilot/ATC coordination)," the average frequency rating of 28 respondents was 3.89. Twenty-one (75 percent) circled "4" or "5", five (18 percent) circled "3" and two circled "1" or "2". The average rating for the importance of radio communications for training non-routine ATC was 3.96, with 24 (86 percent) indicating high or very high. One I/E (4 percent) circled "3" and three (11 percent) thought the importance was low or very low.

• For "CRM: Crew Coordination (e.g., cabin, dispatch, maintenance)," the average frequency rating of 29 respondents was 3.72. Nineteen (66 percent) circled "4" or "5" and nine (31 percent) sometimes. One circled "2". One I/E commented that "[i]f CA [captain] does not delegate duties, my technique is to load the crew up with B.S. [sic] radio transmissions." For importance of radio communications for CRM training, the average of 28 responses was 4.17, the highest average rating for the type of skills queried. Twenty-five (89 percent) indicated that the importance was high or very high, two (7 percent) were neutral and one circled "2."

• The average frequency rating for "Distraction Management" of 28 responses was 3.64. Sixteen (57 percent) responded by circling "4" or "5", seven (25 percent) indicated "sometimes," and five "rarely" or "never." With regard to the importance of radio communications simulation for distraction management training, the average of 27 responses was 3.93. Twenty-one (78 percent) indicated that it was high or very high, three were neutral and three indicated that it was low or very low (11 percent each).

• For "Situation Awareness/Partyline" skills, the average frequency rating of 27 responses was 3.67. Fifteen (56 percent) circled "4" or "5", nine (33 percent) indicated sometimes, and three (11 percent) rarely. With regard to importance of radio communications for effective training of situation awareness via "partyline," the average of 28 responses was 3.89, with 19 I/Es (68 percent) perceiving it as high or very high, seven (25 percent) as medium, and two as low.

Overall Importance of Radio Communications

The question of overall importance for training effectiveness of "realistically simulating each type of radio communications" was asked in conjunction with the question on the effect of radio communications simulation on I/E and pilot workload (Appendix C, Question 3). The scale provided for circling was "1" to "5" with (1) labeled as "Very Low" and (5) as "Very High."

When considering the results to this question, one caveat should be kept in mind. When I/Es were asked for their opinions on workload, they were explicitly asked to consider "your company's radio communications simulation practices." In this subquestion, which was part of the same question, they were asked about the importance of "realistically simulating" the different types of radio communications on overall importance of radio communications regardless of their opinion on the effectiveness on their company's practices. Because this distinction was not explicitly stated, the answers of some I/Es considering their company's practices as ineffective may have inflated the number of low ratings for overall importance. For example, one I/E who had consistently rated the importance of ATC communications as low and company communications as very low, added a comment that these communications are "not very effective during simulation, since the instructor must cover all bases himself." The ratings of this I/E were not included in the ATC and company communications importance ratings below, because his opinions were clearly better reflected in his answers on radio communications in scenarios (see above). There, he gave high ratings of four to the importance of radio communications for training new ATC procedures and non-routine ATC, adding the comment that "[r]adiocommunications is the most effective way to simulate different kinds of approaches (e.g., ILS approach but glideslope out, localizer only approach)." All other responses from this I/E were included.⁹

ATC Communications to Own Airplane

• The average of 28 ratings of the importance of realistically simulating ATIS for training and evaluation was 3.54. Fourteen (half) of I/Es responded high or very high, 11 (39 percent) rated it "3", and three (11 percent) rated it "2" (see Table 4).

• For Clearance Delivery, an average of 28 ratings for importance was 3.39. Twelve (43 percent) each circled "4" or "5" and 12 circled "3", while four (14 percent) circled "2".

• The importance of simulating Tower Ground communications was perceived as high or very high by 19 or 68 percent of 28 raters. Seven (25 percent) indicated medium importance by circling "3", and two (7 percent) circled "2". The average rating was 3.86.

• On average, the importance of Tower Local was perceived as 3.77 by 26 raters. Seventeen (65 percent) considered it as high or very high, seven (27 percent) as medium (3), and two (8 percent) considered it as low (2 percent).

• The average importance rating of 26 respondents for realistic Approach/Departure communications simulation was 3.92. Nineteen (73 percent) felt its importance was high or very high. Six (23 percent) gave it the medium rating of "3", and one (4 percent) circled "2".

• For importance of Center communications, of 26 responses, the average was 3.69. Sixteen (61 percent) responded that Center communications importance was high or very high. Eight (31 percent) gave it a neutral rating. Two (8 percent) indicated that its importance was low.

Company Communications to Own Airplane

• For importance of realistically simulating Dispatch communications for training and evaluation, the average of 28 responses was 3.46. Twelve (43 percent) each perceived its importance as high or very high, (circling "4" or "5") or medium, (circling "3"). Four (14 percent) circled "2" (see Table 5).

• Ramp/Gate communications were given an average importance rating of 3.15 by 26 respondents. Seven (27 percent) circled high or very high. Fifteen (58 percent) circled "3" for medium. Four (15 percent) circled low or very low.

• For communications with cabin personnel, the average importance rating of 23 responses was 3.83. Thirteen (57 percent) responded high or very high. Ten (43 percent) responded that its importance was medium.

• Ten I/Es rated the importance of "Other" company communications, most filling in "maintenance" and two leaving it blank. Six circled "4" or "5" for high or very high, and four circled "3". The average rating was 3.80.

Partyline Communications

• Twenty-five I/Es rated the importance of the partyline for training and evaluation, giving it an average rating of 3.60. Sixteen (64 percent) believe its importance to be high or very high. Five (20 percent) gave it a medium rating (3 percent). Four (16 percent) believe its importance to be low or very low.

3.2.3 Anticipated Effect of Automating Radio Communications

A last question asked I/Es "what types of radio communications you would like to see automated and what do you think the effect would be on instructor workload and training effectiveness." For each of the four types of radio communications offered, the five-point scale ranged from (1) labeled "Lower" to (3) "Same" to (5) "Higher" than what it is now (see Appendix C, Question 8b).

• Twenty-five instructors responded to "Frequency change/handoffs." The average workload rating was 3.04. Five (20 percent) responded that the I/E workload would be the same, and an equal number of 10 (40 percent each) gave ratings above and below "3". For "Training Effectiveness," the average of 24 responses was 3.58, with 13 (54 percent) rating it as higher than now, eight (33 percent) as the same, and only three (12.5 percent) as lower (Table 7).

Table 7. Effect of Automated Radio Communications on Workload and Training Effectiveness

If the following radio communications simulations were automated:	Instructor/Evaluator Workload would be:	Training Effectiveness would be:
	1=Lower 3=Same 5=Higher	1=Lower 3=Same 5=Higher
Frequency Change/ Handoffs	3.04 n = 25	3.58 n = 24
Clearances to Own Airplane	3.00 n = 25	4.00 n = 25
Clearances to Other Aircraft	2.88 n = 26	3.85 n = 26
ACARS	3.05 n = 21	4.30 n = 20

 

• With regard to "Clearances to own airplane," the average rating of 25 for I/E workload was 3.00. Eleven (44 percent) of I/Es anticipated I/E workload to be lower, four (16 percent) the same, and 10 (40 percent) higher. Again, I/Es felt that training effectiveness would improve, giving it an average rating of 4.00. Nineteen (76 percent) anticipated that it would be higher, five (20 percent) same, and only one (4 percent) that it would lower than now.

• If "Clearances to other aircraft" were automated, 26 I/Es anticipated an average workload of 2.88. Ten (38 percent) gave ratings lower than "same," four (15 percent) as same, and 12 as higher (46 percent). With an average rating of 3.85, most I/Es felt that training effectiveness would be improved. Nineteen (73 percent) anticipated it to be higher, four (15 percent) the same, and three (12 percent) lower.

• For ACARS, the average anticipation of I/E workload of 21 responses was 3.05. Seven or a third anticipated it to be lower, five (24 percent) the same, and nine (43 percent) higher. Training effectiveness got the highest average rating of all options offered, 4.30 for 20 responses. Sixteen (80 percent) anticipated it to be higher and four (20 percent) the same.

Several wrote in suggestions in the Comment space provided. These included "ATIS" (eight votes), and "Cabin," "ATC chatter," and "Company via ACARS" with two votes each. "Dispatch" was written in once.

Survey Summary and Additional Instructor/Evaluator Comments

Twenty-nine I/Es participated from 14 different airlines. All but one of these indicated participating in LOFT, 20 in LOEs, and 19 in SPOT. Both ATC and company communications reportedly were most faithfully simulated during LOFT, followed by LOE. Not surprisingly, SPOT ranked last, since it relies least on full mission simulation.

All I/Es queried indicated simulating some ATC communications, especially in the terminal environment. For company communications, all but one of the I/Es reported simulating at least some of them, most frequently dispatch.

Regarding simulation of other traffic, almost 30 percent of participants did not report simulating other traffic, whether out-the-window, on TCAS, or via partyline communications. Almost 60 percent reported simulating some traffic out-the-window on the airport surface, but less than one-third in the terminal and en route environments. A good third reported simulating TCAS.

With regard to the partyline, less than 40 percent reported providing communications to or from other traffic, mainly on the surface. Only two of these provide terminal and en-route partyline information. "[N]one of our formal training documentation requires it," wrote one of the I/Es, and "very seldom used due to I/E workload," another. One I/E also wrote that it is "rarely used" unless it serves a specific purpose: "Sometimes the I/E will clear other aircraft into position to effect a rejected landing or a TCAS event." This was confirmed by another, who wrote that he simulates the partyline "very little, we may clear the aircraft at the hold line for take-off as a cue that we are next to take off."

Most radio communications are simulated by I/E role-play, with few exceptions. I/Es from one carrier instructing a three-person cockpit report that two I/Es, one for the Captain/First Officer and one for the flight engineer, share communication duties. Five I/Es indicated using handouts or ACARS for Clearance Delivery, and over half reportedly use either printed or recorded (synthetic or natural voice) ATIS.

Regarding their workload in the simulator, I/Es reported that for LOEs and LOFT, they spend at least half of their time and effort observing. In LOFT, they spend 8 percent of their time/effort instructing versus only 2 percent in LOEs. This makes sense because the latter is an evaluation event. In SPOT, they spend a quarter of their time/effort instructing, at the cost of observing (only 36 percent). Radio communications simulation takes between 20 to 22 percent for LOFT/LOE and 16 percent for SPOT. Managing the simulator systems requires a consistent 21 to 22 percent of I/E time and effort.

One I/E volunteered the information that he spends 20 percent of his time and effort filling out forms, while another indicated it was 5 percent of his time/effort. Although this activity was not included in the questionnaire, it is probably safe to assume that all I/Es spend some effort filling out forms. This may further reduce their ability to provide an adequate level of radio communications.

Instructor/evaluators were polled on their opinion on the effect of having to simulate radio communications on workload and training effectiveness, as well as on the importance of simulating the partyline. They were also asked for their opinions on the overall effect of realistically simulating radio communications in scenarios.

Most felt that I/E impersonation of communications to own airplane increases the instructor workload and "divides his attention." One I/E added that this is especially difficult for relatively new instructors. On the other hand, it reduces pilot workload by decreasing the communications load compared to the real world, especially company communications. They feel that "company communications are not normally used [in simulation], too time-consuming." Another wrote that "I/E can become task saturated when crew works two VHF radio and/or communicates with cabin simultaneously." One mentioned that I/E impersonation of radio communications even reduces the manual workload of pilots, because they normally do not need to redial a new frequency to communicate in the simulator.

I/Es generally feel that the partyline is greatly reduced when the task of simulating it remains with the I/E, thus reducing pilot workload further. They think that the presence of meaningful partyline information would increase training effectiveness, by "teaching pilots to listen" and "enabling [I/Es] to assess CRM elements such as workload and distraction." "There needs to be extemporaneous background communication between ATC and other aircraft," one I/E stated, and another added that the partyline is his "biggest concern, so pilots are listening." One more I/E referred to the importance of the partyline for helping to simulate a "realistic crowded frequency."

I/Es indicated that scenarios rely on radio communications simulation to teach such skills as (new) ATC procedures, CRM, and situation awareness effectively. The overall importance of radio communications, for ATC, is perceived highest for the terminal environment. For company, it may be cabin and maintenance, although the latter may be simulated only "if a particular procedure might require maintenance help."

I/E ratings indicated skepticism with respect to the success of automating radio communications and its effect on reducing workload (one I/E commenting "No waythe simulator periods are too dynamic!"). Yet, they feel that it would increase training effectiveness, especially for clearances to other aircraft. Perhaps due to I/Es skepticism regarding other kinds of communication automation, ACARS and ATIS received the highest rankings for the anticipated effect of automation on training effectiveness.

In addition to the one I/E referring to the effect of instructor experience on radio communications role-play, one more I/E alluded to the differences in scenario complexity and workload induced by instructor, which is "all relative to the creative ability of the instructor."

Several I/Es made general comments on how radio communications are a "critical need for LOS realism." "There is a correlation between realism in communications and training effectiveness; better training results from more realism." One I/E concluded his survey with the remarks: "This is a subject near and dear to my heart. [] I believe the simulator mind-set syndrome' must be fought with realism. How can we expect crews to treat sim[ulator] like the aircraft' when the audio environment belies the condition so often?"

4. Technology Review

Regulation, market forces, and airlines' own desire for improvements are driving forces for simulator technology advancement. The need for realistic radio communications including simulation of the partyline is mentioned both in the AQP and CRM Advisory Circulars (FAA, AC120-54A and AC120-35B, see Literature Review section). The Instructor/Evaluator Task Listing Training Level Performance Statements (FAA, 2000) says that I/Es "advise crew members that LOS presents real world' problems and the Instructor will role play ATC, F/A, Dispatcher, etc." and that "crew members should prepare and act as if on the line."

One example of airline development of improved radio communications simulation is the United Airlines IRAS or "Interactive Real Time Audio System," also (and somewhat too dismissively) known as "Chatter Program." United invested in a multi-year in-house program to attempt to modify many of its current simulators to provide a more realistic radio communications environment.

CAE Electronics Ltd. described market forces as driving the development of their GATES or "Ground and Air Traffic Environment System" product. A customer specification of a more realistic airport and terminal environment was the driving force for GATES development. Another example of the market clearly perceiving the value of realistic radio communications is the recent proliferation of Personal Computer Commercial Off-The-Shelf (PC COTS) systems. Personal Computer Aviation Training Devices (PCATD) and entertainment flight simulator products provide interesting approaches to simulating the traffic environment.

Intelligent systems, which have been mainly developed for military applications, have the potential for a high degree of communication realism. TacAir SOAR, e.g., simulates pilot behavior and decision making in complex airborne tactical warfare distributed simulations. Intelligent systems may be the most promising for the future, although a specific ATC implementation remains to be developed (see Section 4.4).

The information in this section was gathered at site visits, through interviews of users and manufacturers, and, for some of the PC COTS systems, by purchasing and testing the system.

4.1 Interactive Real Time Audio System

IRAS is a United Airlines in-house development with very high operational realism. It is based on field recordings of actual ATC communications on UAL routes. By dubbing ATC with the respective I/E voice, the I/E can intervene without the pilot trainee realizing it. The engineers at United attempted to include many subtle nuances of audio communications into the environment including ATC driven, demanding timing, frequency congestion, foreign or regional accents, stuck microphones blocking an entire frequency, and meaningful partyline to train pilots to listen.

The system, however, encountered many technical difficulties. Algorithms coded to trigger ATC or partyline recordings at appropriate times did not always function adequately, degrading realism and forcing an embarrassed instructor to intervene, increasing instructor stress and workload. Operations in dense terminal areas also taxed the scenario algorithms, with normal variations in crew response causing additional timing problems and workload for the instructor.

The IRAS interface was often not well integrated into the instructor station resulting in poor ergonomics and human factors. Different instructor interfaces across simulator models often made it even harder for instructors to operate the system. The cost and difficulty of systemwide implementation was further hampered by different simulator designs with different visual, audio, and navigation models.

All of the above in combination with expensive scenario development including field recording, transcribing, dubbing, database maintenance, as well as costly route, sector map, and simulator- interface code development contributed to the program gradually losing support. Currently, the program has been scaled back to use only one generic ATC voice, and it is only used for new- hire screening in a 727 flight training device.¹⁰ The applicants are presented with relevant partyline information to evaluate their planning abilities during an approach, as well as automated frequency changes. New hire evaluators reportedly complain when the system is down because of the increased workload of providing frequency changes.

The lessons learned from IRAS are that for a system to be successful, it must be flexible, transparent, easy to use, easy to implement and maintain, and easy to integrate with different scenarios, simulators, and simulator systems (e.g., visual, audio, etc.). There may be a cost/benefit trade-off for the different aspects of realism required for different training and evaluation events that must be examined in this research. For instance, it may not be necessary to conceal instructor intervention from the pilots, a capability achieved by IRAS, but at a high price.

4.2 Ground and Air Traffic Environment System

Airlines have advanced the state-of-the-art by specifying requirements to the simulator manufacturers. The CAE product GATES came to be following a request from a foreign airline to provide a visual representation of traffic in the airport terminal environment. It soon became obvious to the developers and airline that correlated and meaningful radio communications would have to be an essential component of such traffic representation.

Several domestic airlines and training facilities as well as foreign airlines and military are currently installing GATES-equipped simulators. The product as it is today provides simulated aircraft traffic and associated relevant communications to and from those aircraft on the airport and in the terminal environment. Runway incursions can be seen as well as heard. The product also provides voice simulations of ground traffic from emergency equipment as well as maintenance, fuel, baggage, and aircraft handling.

GATES-simulated vehicles do not follow scripted scenarios. A continuous flow of arriving and departing traffic is generated. The traffic elements are aware of and will react to each other and the simulated own airplane. The I/E has limited ability to control specific situations like runway incursions and emergency vehicles. The I/E still provides all ATC communication to own airplane, however. Pre-departure clearances are provided by simulated ACARS.

The main reason a domestic airline cited for choosing GATES was to add realism and increase distraction factors for LOFT sessions. Specifically, they wanted to enhance TCAS scenarios, ground hazard avoidance, and runway incursions. A senior check airman responsible for GATES implementation stated that the improved overall sense of realism alone justified its purchase and support costs. Workload increase for the I/E is reportedly minimal. He added that interactive/reactive communications capability would enhance usability and realism.

4.3 Personal Computer Commercial Off-The-Shelf Systems

The PC COTS flight simulator manufacturers clearly perceive a market value for including realistic radio communications add-ons to their products. Evaluations included a PCATD product approved for limited credit in the presence of a flight instructor (FAA, 1997) and three PC-based entertainment software systems. Though not approved for training credit, the latter are marketed as ATC and instrument flight environment enhancements to instrument pilots. With respect to realistic radio communications, the methods of generating and presenting ATC and/or partyline communications as well as the perceived limitations and strengths of the design are described.

Typical features include concatenated recorded natural speech to own airplane and to/from other aircraft, partyline consisting of random chatter with no relevance for situation awareness and no visual traffic, and scenarios scripted with a simple decision tree.

Flexibility and controllability vary greatly among products. The different systems add varying degrees of realism and fun to the simulations, dependent on ease of use. They are summarized below, starting with the most significant one.

4.3.1 Jeppesen FS-200/Flightpro

FS-200/Flightpro is available, when packaged with the proper hardware, as an FAA approved PCATD, offering some credit in the presence of a certified instrument flight instructor (CFII) under AC61-126 (FAA, 1997). Specifically, it can be used to meet the requirement of an instrument rating in lieu of, but not more than, 10 hours of time that ordinarily may be acquired in an aircraft, flight simulator, or flight training device. According to a Jeppesen representative, the PCATD version is marketed toward flight schools as a low cost alternative simulator. The FS-200 and ATC add-on software are generally marketed toward private pilots as a tool to provide a more realistic simulation to maintain currency. About 50 percent of FS-200 owners purchase ATC packages. However, they are not flexible enough for flight schools, which rely on the instructor for simulating ATC. The ATC flight packages are low cost-scripted scenarios. They add natural sounding concatenated ATC audio recordings. The partyline is relevant. For example, an aircraft ahead is being given hold instructions due to weather and soon the pilot receives something similar. The script is identical for every flight and ATC communications are aborted if the pilot strays off course.

4.3.2 FlightSafety International Aviator PRO 98

Aviator PRO 98 adds ATC scripts to Microsoft's MS Flight Simulator that function much like the Jeppesen add-on scripts. The inflexible scripts do not permit deviations. One cannot request any changes from ATC. The script will terminate after a set number of deviations. ATC clearances are recorded natural speech. The partyline chatter is random and not relevant to the current situation, but are rich in features such as disfluencies, distortions, and pilots interrupting an ongoing exchange by keying their microphone.

4.3.3 Proflight

Proflight also adds features to MS Flight Simulator. Its interface generates a flight plan, weather, and ATC script. The scripts are somewhat more flexible than those of the previous PC COTS systems. Proflight accepts 50 different keyboard commands to acknowledge ATC and make requests. For example, a pilot can request a different altitude, declare a missed approach, or request vectors. ATC clearances are concatenated recorded natural speech. Partyline communications are limited to aircraft checking in with headings and altitudes and are not relevant to the current flight.

4.3.4 Flight Unlimited

Flight Unlimited is entertainment software that offers a non-scripted intelligent air traffic control system with vectoring abilities. There is a relevant partyline that corresponds to visual traffic in common situations. It has limited ability in some situations to interact with other traffic visually with correct communications to and from other aircraft. A pop-up menu provides the input choices, possible responses and requests to ATC partyline and ATC clearances are concatenated recorded natural speech.

In conclusion, all but one of these systems are scripted, which represents similar limitations here as encountered with IRAS. COTS products that are based on scripts limit users to whatever scenarios they purchased. They are often not tolerant of deviations from the script, even if these deviations involve accepted and sometimes even recommended navigation practices. Pilots must make requests from and respond to ATC via keyboard input, increasing their workload. The introduction of even basic artificial intelligence capabilities such as in Flight Unlimited greatly improves realism and flexibility of the system, significantly reducing user frustration.

4.4 Intelligent Systems

Intelligent systems, which were mainly developed for military applications, have the potential for a high degree of communication realism with no instructor intervention. They use rule-based models of goals and behavior of the speaker (e.g., controller). Messages are generated based on speaker task goals and situation awareness. This eliminates the need for complex "triggering" algorithms.

This kind of modeling has been successfully applied in manned and unmanned simulations. Rapid Intelligent Tutoring Development Shell (RIDES, Flemming and Horwitz, 1996) provides an intelligent advisor for simulator and airborne pilot training. RIDES is a tool for developing agent behavior and permits the author/instructor with little or no programming experience to quickly modify parts of the simulation.

TacAir SOAR (Jones, Laird, Nielsen, Coulter, Kenny, and Koss, 1999) simulates pilot behavior and decision-making in complex airborne tactical warfare distributed simulations. It integrates a wide variety of intelligent capabilities including real-time hierarchical execution of complex goals, communication and coordination with humans and simulated entities, maintenance of situational awareness, and the ability to accept and respond to new orders while in flight. It has been successfully applied in a 1997 Department of Defense Synthetic Theater of War simulation. One of its developers (Jones, personal communication, 1999) sounded optimistic about the feasibility of a civil application such as realistic radio communications simulation.

4.4.1 Component Agent Approach

Expert system development for large-scale computer simulations of environments, such as tactical air combat, is an expensive and time-consuming effort. Much of the development of TacAir-Soar and its predecessor, TacBrawler, took many years to develop because of the extraordinarily complex and dynamic environment that these systems are attempting to simulate. The development of this level of sophistication within a training simulation environment may not be cost effective.

An alternative approach to the application of intelligent systems which does not require the extensive development of war gaming simulations is to develop a component approach to the application of intelligent systems. In this approach, smaller, modular components (agents) are added into existing simulation applications and are programmed to provide expertise within a narrow range of parameters. Modularized agent technology could be developed and applied to emulate controller behavior within a given facility or sector or even within the confines of particular approach procedure. Each such agent is provided with only those rules necessary to provide the simulation environment with the needed communications in a timely fashion. Because of their small size, such agents could be readily adapted to a variety of simulation environments without a major re-design of the simulator's software.

This modular approach to intelligent agent development and application has become a cost-effective alternative to large-scale intelligent systems development, particularly when agents are intended as add-ons to existing software applications (Riecken, 1994).

4.5 Supporting Technologies

4.5.1 Rapidly Reconfigurable Line-Oriented Evaluation Scenario Generator (RRLOE)

Technologies that may support the automated provision of realistic radio communications during simulator training events also have been examined. The most promising of these for a script- based approach is the rapidly reconfigurable line-oriented evaluation scenario generator (Bowers, Jentsch, Baker, Prince, & Salas, 1997). LOE scenarios currently are developed and tested by the training organization and individually approved by the FAA. Training organizations can only produce a limited number of scenarios, which are valid for a limited time. The risk that the content of the scenarios will become known among pilots is high. RRLOE will use LOE event sets (modules) which, when FAA approved, can be assembled by the training organization into complete and valid LOE scenarios. In addition, the RRLOE scenario generator will provide the instructor with either an explicit script of ATC transmissions or the necessary information for all communications directed from ATC to the crew.

4.5.2 Speech Generation and Recognition Technology

Any successful automated radio communications simulation system would have to rely on speech generation and recognition systems. Only a summary of this technology and its application to flight simulation can be provided here. Significant progress in speech research over the past two decades permits the generation of high-quality speech regardless of whether the speech is synthesized "from scratch" (articulatory or formant synthesis, e.g., DecTalk) or concatenated from prerecorded and digitized natural speech (concatenative speech). In assessing the state of speech recognition technology with respect to its practical utility for airline pilot training and evaluation, several criteria need to be considered. First, the intelligibility of voice displays used in simulations needs to be provided at a level comparable to that of human speech to avoid introducing additional cognitive loads to trainees during either training or evaluation, especially in the absence of disambiguating semantic context. Human-speech like intelligibility is well within the capabilities of existing higher quality speech systems, at least in a quiet environment. In a noisy environment such as the cockpit, however, articulatory or formant synthesis may not yet be able to replicate the full richness of natural human speech with all its redundancy, which may be needed when some of the distinguishing properties of individual speech segments are masked by the ambient noise. Concatenative synthesis, which consists of natural speech segments, should maintain some of this redundancy and receive higher intelligibility ratings in noise.

A second consideration is how natural the synthetic voice sounds, which, for lack of an objective standard, is determined by subjective assessments by a sample of end users. Naturalness is a function of the degree to which both coarticulation and prosodic effects are captured. Coarticulation results in the "sound signature" of words excised from a sentence looking quite different from words spoken in isolation, while prosody mainly refers to pitch, timing, and intensity. Currently, only concatenative speech using units large enough to cover both coarticulation and prosodic effects meet extremely high quality criteria, at a high computer memory cost. Theoretical research in prosody, in particular with respect to pitch and timing, however, is progressing internationally and its results are being implemented by the industry. This will greatly improve user acceptance of synthetic speech, which is important to avoid pilot stress and distraction especially for advanced training and evaluation. There is no reason to rule out machine speech generation for future simulator applications.

The main stumbling block, however, may be automatic speech recognition, although recognition rates have been dramatically improved over the past 5 years. The main problems that need to be overcome by an automatic speech recognizer to be successful are coarticulation (words can occur in an unlimited number of contexts), and variability between and sometimes even within speakers in articulating the same word. The first problem may be less prominent in air traffic control, which uses a restricted and highly regulated phraseology. The second problem refers to the fact that each speaker will articulate the same word or phrase slightly differently. This may be due to physical differences in the vocal tract, such as found between men and women, or different linguistic experiences, such as geographic or social accents. Differences in articulation can be pronounced even within the same speaker, due to variations in speaking rate, a cold, or emotional state. Even with a highly regulated phraseology such as used in aviation, extensive "training" of the system with a large set of utterances of each speaker will be required to achieve satisfactory reliability.

With regard to speaking rate, speech recognition systems can now recognize rates near those of normal, conversational speech (e.g., about 180 words per minute), but no system has yet achieved the much higher speech rates often found in aviation operations (e.g., more than 250 words per minute, see Lee, 1999). This suggests that simulator applications of this technology may have to await further refinements. Additionally, the speaker training required of these systems to achieve very high levels of recognition accuracy may prove problematic for training management. However, the implementation of human speech perception and production research findings as well as advanced mathematical tools make it conceivable that these limitations may be overcome in the future. For this reason, the application of speech recognition technology in communications simulation, necessary for a fully automated communications simulation, should be considered as a potentially long-term enabling technology for future development in communications simulation.

5. Conclusions

The findings documented in this report are as follows:

• The literature emphasizes the need for a realistic radio communications environment as a key element in training and evaluation scenarios. Not only does it identify radio communications as a contributor to many incidents and accidents, but it also underscores its importance for effective CRM (where ATC and company are part of the crew) and task management training.
• Subject matter experts perceive realistic radio communications as an important element in pilot simulator training and evaluation, enhancing pilots' distraction management and situation awareness skills while breaking the "simulator mindset."
• Airlines mostly leave the simulation of radio communications to the individual I/Es, thus increasing I/E workload while reducing pilot workload because the already taxed I/Es cannot provide a realistic communications load. This impoverished simulation environment may compromise the transfer of skills between the simulator and the airplane.
• The technology to simulate ATC/Company and partyline automatically is still immature, although there is promising progress in intelligent systems and automated speech generation/recognition.

The underlying premise of this workthat the provision of realistic radio communications during training and evaluation of airline pilots would enhance safetyhas been confirmed thus far. Airlines, however, are unlikely to allocate the necessary funds for provision of a full radio communications environment during simulator training and evaluation unless there is a proven gain in safety and training effectiveness and a requirement from the FAA. The technology development seems to be largely market driven, but would of course be furthered by such a requirement. Therefore, a proof-of-concept is required that will empirically demonstrate the benefit of realistic radio communications for safety of operations. A first step may be to systematically compare the communications load and its effect on pilot workload and CRM behaviors during revenue operations in the air versus LOFT/LOEs. Another step may be an assessment of pilots who have just completed initial qualification in their first fleet. The question of interest would be how prepared such pilots are to handle ATC and company communications requirements during their first Initial Operating Experience (IOE) flights. This may be followed up by a carefully conceived study examining the effects of different levels and approaches to achieve radio communication realism on transfer of training acquired in the simulator to the airplane or at least the simulator with realistic radio communications as a stand-in for the airplane. Full skill transfer to and from the airplane is a critical issue, if simulator use for training and evaluation is to be mandated.

REFERENCES

Boothe, E., (January 9, 2000). Presentation Given at the Human Factors in Transportation 33rd Annual Workshop, Transportation Research Board 79^th Annual Meeting. Washington, D.C.

Bransford, J., and J. Franks (1976). Towards a Framework for Understanding Learning. The Psychology of Learning and Motivation. 10, 93-127.

Billings, C., and E. Cheaney (1981). Information Transfer Problems in the Aviation System. NASA technical paper 1875.

Bowers, C., F. Jentsch, D. Baker, C. Prince, and E. Salas (1997). Rapidly Reconfigurable Event-Set Based Line Operational Evaluation Scenarios. Proceedings of the Human Factors and Ergonomics Society 41^st Annual Meeting, Albuquerque, NM (pp. 912-915). Santa Monica, CA: Human Factors and Ergonomics Society.

Bürki-Cohen, J. (1995). An Analysis of Tower (Ground) Controller-Pilot Voice Communications. Final Report No. DOT-VNTSC-FAA-95-41.

Cardosi, K. (1993). An Analysis of En Route Controller-Pilot Voice Communications. Final Report No. DOT-VNTSC-FAA-93-2.

Chou, C., D. Madhavan, and K. Funk (1996). Studies of Cockpit Task Management Errors. The International Journal of Aviation Psychology. 6, 307-320.

Connell, L. (1994). Pilot and Controller Communications: Incidents Reported to the NASA Aviation Safety Reporting System. (SAE Technical Paper 942137) Paper presented at Aerotech '94, Los Angeles, CA.

Connell, L. (1996). Pilot and Controller Communications Issues. In B. G. Kanki & O. V. Prinzo (eds), Proceedings of Methods and Metrics of Voice Communications Workshop, San Antonio, TX.

Endsley, M., T. Farley, W. Jones, A. Midkiff, and R. Hansman (1998). Situation Awareness Information Requirements for Commercial Airline Pilots. International Center for Air Transportation, ICAT-98-1.

Federal Aviation Administration, U.S. Department of Transportation, (September, 1990). Crew Line Operational Simulations: Line Oriented Flight Training, Special Purpose Operational Training, Line Operational Evaluation. Advisory Circular No. 120-35B.

Federal Aviation Administration, U.S. Department of Transportation (July, 1995). Airplane Simulator Qualification. Advisory Circular No. 120-40A, B, C (draft).

Federal Aviation Administration, U.S. Department of Transportation (July, 1997). Qualification And Approval Of Personal Computer-Based Aviation Training Devices. Advisory Circular No. 61-126.

Federal Aviation Administration, U.S. Department of Transportation (October, 1998). Crew Resource Management Training. Advisory Circular No. 120-51C.

Federal Aviation Administration, U.S. Department of Transportation (July, 1998). Advanced Qualification Program. Advisory Circular No. 120-54A.

Federal Aviation Administration, U.S. Department of Transportation, (February, 2000). Advanced Qualification Program, AFS-230 Web site: http://www.faa.gov/avr/afs/aqphome.htm.

Fleming, J., and C. Horwitz (1996). Application of the Rapid Intelligent Tutoring System Development Shell (RIDES). ITS 1996 Workshop, Montreal, Canada.

Helmreich, R., and H. Foushee (1993). Why Crew Resource Management. In E. Wiener, B. Kanki, & R. Helmreich, (eds.) Cockpit Resource Management, San Diego, CA: Academic Press, 3-45.

Jones R., J. Laird, P. Nielsen, K. Coulter, P. Kenny, and F. Koss (1999). Automated Intelligent Pilots for Combat Flight Simulation. AI Magazine.

Jones, R. (November 2, 1999). Personal Communication.

Kanki, B. and M. Palmer (1993). Communication and Crew Resource Management. In E. Wiener, B. Kanki, and R. Helmreich (eds.) Cockpit Resource Management, San Diego, CA: Academic Press, 99-136.

Kanki, B. and G. Smith (in press), Training Aviation Communication Skills.

Kayten, P. (1993). The Accident Investigator's Perspective. In E. Wiener, B. Kanki, and R. Helmreich, (eds.) Cockpit Resource Management, San Diego, CA.: Academic Press, 283-314.

Khatwa, R. and R. Helmreich (November, 1999). Analysis of Critical Factors During Approach and Landing in Accidents and Normal Flight. Flight Safety Digest. Flight Safety Foundation.

Latorella, K. (1996). Investigating Interruptions: An Example from the Flightdeck. Proceedings of the Human Factors and Ergonomics Society 40^th Annual Meeting.

Lee, A. (1999). Advanced Technology Applications to Flight Simulation Training and Evaluation: Voice Generation and Recognition. (Tech. Report BRI-TR-121099). Los Gatos, CA: Beta Research, Inc.

Lintern, G., H. Taylor, J. Koonce, R. Kaiser, and G. Morrison (1997). Transfer and Quasi-Transfer Effects of Scene Detail and Visual Augmentation in Landing Training. International Journal of Aviation Psychology, 7, 149-169.

Longridge T. (1997) Overview of the Advanced Qualification Program. Proceedings of the Human Factors and Ergonomics Society 41^st Annual Meeting, 898-901.

Mangold, S., and D. Eldredge (1993). An Approach to Modeling Pilot Memory and Developing a Taxonomy of Memory Errors. In Proceedings from The Ohio State University Seventh International Symposium on Aviation Psychology. Columbus, OH.

Office of the Federal Register, Code of Federal Regulations, Title 14, Federal Aviation Regulations Part 121 Air Carriers and Commercial Operators, Appendix H: Advanced Simulation Plan, Vol 45 (1980), p. 44183.

Office of the Federal Register, Code of Federal Regulations, Title 14, Federal Aviation Regulations Part 121 Special Federal Aviation Regulations 58Advanced Qualification Program, Vol 55, (1990), p. 40275.

Predmore, S. (1991). Microcoding of Cockpit Communications in Accident Analyses: Crew Coordination in the United Airlines Flight 232 Accident. In R. S. Jensen (Ed.), Proceedings of the Sixth International Symposium on Aviation Psychology (pp. 350-355). Columbus, OH: Ohio State University.

Riecken, D. (1994). Intelligent Agents. Communications of the ACM, 37, 18-21.

Rodgers, M., and L. Nye (1993). Factors Associated with the Severity of Operational Errors and Air Route Traffic Control Centers. In M. Rodgers (ed.) An Examination of the Operational Error Database for Air Route Traffic Control Centers (DOT/FAA/AM-93/22), Oklahoma City, OK: Human Factors Research Laboratory, Civil Aeromedical Institute.

Rosenthal, L., R. Chamberlin and R. Matchette (1996). Flight Deck Confusion: A Survey of ASRS Reports Indicates That Better Communication Between Cockpit Crews and ATC Could Help Prevent Confusion-Related Incidents. Air Line Pilot. Vol 65, May.

Sumwalt, R., R. Morrison, A. Watson, and E. Taube (1997). What ASRS Data Tell About Inadequate Flight Crew Monitoring. In Proceedings from The Ninth International Symposium on Aviation Psychology 977-983.

Appendix A

Results of the Approach and Landing Accident Reduction Task Force Report and Related Studies

The importance of efficient interactions with ATC and company (Khatwa & Helmreich, 1999) are pointed out in the report of the Approach and Landing Accident (ALA) reduction task force of the Flight Safety Foundation. The task force based its conclusions on an analysis of 76 ALAs and serious incidents involving jet and turboprop aircraft and occurring between 1984 and 1997 (inclusive). "Incorrect or inadequate ATC instruction/advice/service" was a causal factor in 33 percent of all occurrences. It ranked eleventh among the most common causal factors after "Poor professional judgment/airmanship" in first, "Lack of positional awareness" in fourth, and long before "Interaction with automation" in seventeenth place. "[D]emanding ATC clearances" are also explicitly mentioned in context with even higher ranking causal factors such as the sixth-placed "Flight-handling difficulties," the eighth-placed "Press-on-itis," or the tenth-placed "Slow and/or low on approach." In many cases of "press-on-itis," "a breakdown in CRM between the flight crew and ATC" was observed.

Incorrect or inadequate ATC instruction/advice/service may include incorrect radar vectoring, incorrect (or absence of) essential traffic information, or inadequate controller technique for dealing with aircraft facing difficulties. Late runway assignment notifications or last-minute changes may cause rushed approaches. ATC may be unaware of the demands and limitations of the glass cockpit, where such clearances may require reprogramming of the flight management system (FMS). This greatly increases workload, undermines CRM and reduces situational awareness. Potential results are unstable conditions, overruns, CFIT (Controlled Flight Into Terrain), or loss of control. Several of the problems mentioned in this paragraph were recognized as potentially contributing factors to the 1995 American Airlines Flight 965, Boeing 757 CFIT in Cali, Columbia, which cost 159 lives (Simmon, 1998).

Other ATC-communications related causal factors in the report that are not included in the 33 percent are misunderstood or missed communications such as missed readbacks, call-sign confusions, and simultaneous transmissions (12 percent of occurrences). Instances of controllers and crews using non-standard phraseology are also mentioned. Communications can become especially difficult when non-native English speakers and listeners are involved, as the 1990 Avianca Airlines Boeing 707-321B crash over Long Island demonstrates. In this accident, the Colombian crew was unsuccessful in communicating a fuel emergency to the American ATC (NTSB, 1991). In addition to the ALA survey, Khatwa and Helmreich mention an ongoing study including line audits of 102 flight segments to and from destinations with non-native English speaking ATC (Klinect, Wilhelm, & Helmreich, 1999). One of the most frequent errors observed involved crew interaction with ATC. Language problems may also have exacerbated the communications problems encountered by the American Airlines crew over Cali (see above).

Both the Avianca and the American Airlines crashes show that efficient communication between flight crew and ATC/company is especially important during emergency situations. The ALA reduction task force relates several instances where ambiguous emergency messages from the crew were not cleared up by ATC follow-up, a factor that is also mentioned in the Cali accident (Simmon, 1998). In other cases, ATC may have contributed to confusion and distraction of the crew, e.g., with unnecessary requests for information especially during the "sterile cockpit" period below 10,000 feet (Sumwalt, 1993). The ALA task force also reports instances where crews repeatedly ignored urgent warnings from ATC.

The task force points out that such examples demonstrate poor CRM between flight crews and ATC and recommend that regulatory authorities should "[r]equire operators to provide all operational personnel (e.g., flight crew, cabin crew, air traffic controllers) effective training in CRM." It also recommends that operators "[i]nclude training scenarios that allow crews to experience overload, task saturation, loss of situational awareness, out-of-control and too-far-behind-the-aircraft situations, and communications in stressful circumstances." Joint training sessions should be held between pilots and air traffic controllers including scenarios that "promote mutual understanding of issues on both the flight deck and in the ATC environment, and foster improved communications during emergency situations." The use of standard ICAO phraseology should be emphasized (see also Jones and Tesmer, 1999, for training recommendations).

In summary, realistic radio communications in carefully constructed simulator training and evaluation scenarios give pilots the opportunity to practice how to recognize and deal with incorrect, inadequate, or too demanding ATC instructions. They also allow pilots to practice correct phraseology, especially during emergencies and/or with non-native English-speaking ATC.

REFERENCES

Jones, S., and B. Tesmer (1999). A New Tool For Investigating and Tracking Human Factors Issues in Incidents. In Proceedings from The Ohio State University Tenth International Symposium on Aviation Psychology. Columbus, OH.

Khatwa, R., and R. Helmreich (November, 1999). Analysis of Critical Factors During Approach and Landing in Accidents and Normal Flight. Flight Safety Digest. Flight Safety Foundation.

Klinect, J., J. Wilhelm, and R. Helmreich (1999). Threat and Error Management: Data from Line Operations Safety Audits. Proceedings from The Ohio State University Tenth International Symposium on Aviation Psychology. Columbus, OH.

National Transportation Safety Board, (1991). Aircraft Accident Report-Avianca, The Airline of Colombia, Boeing 707-321B, HK 2016, Fuel Exhaustion, Cove Neck, New York. Report PB91-910404.

Simmon, D. (May, 1998). Boeing 757 CFIT Accident at Cali, Colombia Becomes Focus of Lessons Learned. Flight Safety Digest. Flight Safety Foundation.

Sumwalt, R. (June, 1993). The Sterile Cockpit. ASRS Directline.

Appendix B

Realistic Radio communications For Task Management Training

The Need for Task Management Training

In an intuitive summary on the cognitive underpinnings of task management based on Norman and Shallice (1986; see also Shiffrin & Schneider, 1997), Dismukes et al. explain that humans have two cognitive systems with which they perform tasks, one involving conscious control, the other operating in an automatic fashion (Dismukes, Young, and Sumwalt, 1998). The conscious system operates slowly and methodically, performing one operation at a time. Learning a new skill typically requires conscious processing. Dismukes et al. explain that "this is why learning to drive a car or flying an airplane at first seems overwhelming: the multiple demands of the task exceed conscious capacity." As a learner of a skill progresses, the repetitive aspects of a task will become automatic, requiring little or no conscious effort, whereas novel aspects continue to require conscious attention. Since any human language allows the generation of infinitely many novel sentences (see, e.g., Chomsky 1965), communication is one activity that allows little automatic processing, even if it is somewhat regulated as are communications between pilots and ATC. Aside from novelty, other attributes of a task that require conscious control listed by Dismukes et al. are difficulty, criticality, or danger. Finally, the presence of competing activities (presumably of equal urgency) and "when an automatic process must be overridden to prevent habit capture" (such as when trying to catch a turnoff when taking a different route home from work) require a shift from automatic to conscious control.

Given that conversations have the power to capture all our conscious capacity, it is not surprising to learn that radio communications have been found to be a close second contributor to monitoring errors (Sumwalt, Morrison, Watson, and Taube, 1997). In their study, Sumwalt et al. reviewed 200 of 800 ASRS reports on monitoring errors submitted between 1992 and 1996. Monitoring errors were defined as a "failure to adequately watch, observe, keep track of, or cross check" the aircraft's trajectory or systems. Safety consequences ranged from taxi or flight path deviations and runway incursions to speed violations to system or equipment damage or shutdown to loss of control. For the 170 reports that indicated the tasks or functions accomplished shortly before or during the error, 45 percent reported being involved with cockpit automation/navigation systems and 42 percent with radio communications (ATC, company, or obtaining ATIS). Cockpit documentation (checklists, etc.) was in third place with 39 percent (some reporters listed several activities). The majority of errors occurred during climb, followed by descent, descent transition, and approach.

Similar findings are reported by McElhatton et al. who observed that crossing restriction altitude deviations on Standard Instrument Departures (SIDs) and Terminal Arrival Routes (STARs) tend to occur most frequently during descent (77 versus 33 percent of 172 ASRS reports studied, McElhatton, Buchanan, and Drew, 1998). McElhatton et al. hypothesize that the preponderance of deviations during descent can be traced back to increased workload due to, among others, ATIS and company communications. Cockpit workload was indeed cited as a factor in 44 percent of the reports. Again, "high quantity radio communications with ATC" were the second most frequently mentioned workload issue with just one less citation than "FMS programming," and tied with "lack of planning on the part of the flight crew that led to time compressions (such as cabin attendant in cockpit)." In fact, overall radio communications issues may have even surpassed automation issues, given that company communications were included in "Other" in third position.

One might object that ASRS reports represent a biased sample due to their voluntary nature. The fact that task management and related flight path errors increase with the number of concurrent tasks, however, has been confirmed in controlled experiments as well. Chou et al. investigated the significance and nature of the task management process by first conducting a review of NTSB accident and ASRS incident reports and then testing their findings in a controlled simulator experiment (Chou, Madhavan, and Funk, 1996). They describe cockpit task management (CTM) as consisting of the following seven functions: 1) task initiation when appropriate conditions exist; 2) task monitoring; 3) task prioritization; 4) resource allocation; 5) task interruption when resources need to be allocated to a higher priority task; 6) task resumption when priorities change or resources become available; and 7) task termination when task has been completed, cannot be completed, or has become irrelevant. CTM errors are classified to correspond to the aforementioned CTM functions.

In their accident report study, they found CTM errors in 23 percent of the 324 accidents reviewed. This number rose to 49 percent of 470 reports reviewed for the incident study. In both studies, task initiation errors, such as early descents, late configurations, and failures to tune navigation and communication radios, were in the lead. Task prioritization errors including distractions by weather and traffic watches were a close second, especially for the incidents. Last were termination errors (such as early autopilot disengagements, altitude overshoots, and improperly continued landings under unsafe conditions).

The objective of the simulator study was to elicit the CTM errors found in the previous analyses and to identify the contributing factors. Mainly non-pilot participants flew a low-fidelity simulator after training, in one of six scenarios of varying degrees of difficulty. Degree of difficulty was defined as a function of resource requirements, number of concurrent tasks, and flight path complexity. Performance was measured by assessing response times to system faults, flight path errors, task prioritization, and late task initiation. Average response time and late task initiation were significantly increased only by visual, manual, and mental resource requirements, but not by the number of concurrent tasks or flight path complexity. The only flight deviation measures that were affected by any of the scenario manipulations were heading, where deviations increased by the combination of flight path complexity and the number of tasks, and altitude deviations, which were increased by increased mental resource requirements. Task prioritization, however, was greatly degraded by both resource requirements and the combination of flight path complexity and number of concurrent tasks.

One last study to be mentioned here that specifically investigated interruption management experimentally used ATC clearances to interrupt commercial airline pilots as they performed procedures during approaches in the simulator (Latorella, 1996). Latorella found that ATC interruptions significantly increased procedure performance errors as well as flightpath management activity (number of attitude lateral control inputs).

The Method for Task Management Training

Now that it has been established, via both anecdotal and empirically evidence, that task management skills need to be trained and evaluated, how can this best be accomplished?

Simulator Task Management Training Can Transfer to In-Flight Performance/Behavior

A first finding is that task management trained even in a simulation with low physical fidelity may indeed transfer to flight (Gopher, Weil, and Bareket, 1994). The training device used by Gopher et al. consisted of an extremely complex computer game requiring subjects to control the movement of a spaceship while aiming and firing missiles to destroy a space fortress.¹¹ Limited resources, constant enemy attacks, and the requirement to verbalize one's own actions ensured task difficulty. Similarity of task demands with real flight was seen in the shared requirement for continuous and discrete manual control, visual and spatial orientation, procedural knowledge involving long- and short-term memory, high attention demands under severe time constraints, and the presence of in-flight communication (verbalizations). Gopher et al. compared actual flight performance of two groups trained with different strategies (to be discussed below) in the space-fortress game with a control group without training and found a clear advantage of the space-fortress trained groups over a control group of matched ability. Gopher et al. make a point of explaining that it is unlikely that these results could be due to the so-called Hawthorne effect, which purports that any group receiving special treatment over what another group receives should show an advantage. All subjects in this study were highly motivated volunteers from an exclusive military flight school, but participation in the study was only a small and unimportant fraction of their daily activities that was not welcomed with enthusiasm. The space-fortress training was regarded as an extra burden, not a privilege. A cover story was aimed at eliminating further bias. Participation or lack of such in the space-fortress training was not evident to the flight performance evaluators. Interestingly, the computer game was perceived as such a successful training tool that it was later incorporated into the regular curriculum of the Israeli Air Force.

Whole-Task Versus Part-Task Training

The 1994 Gopher et al. study also served to compare the effects of two different training strategies for the two space-fortress game groups. The so-called full-training group was first trained with a sophisticated hierarchical part-task technique of working its way up through a series of simplified "part-task" games of increasing complexity before the integrated game with its full task load was presented, all the while receiving verbal tips and individual coaching. This technique has been developed by Frederiksen and White and had been shown to lead to superior performance than simple "whole-task" training, at least in the context of the space-fortress game or related contexts (1989). Although some effort was taken to test transfer to a different context, subjects still had to manipulate an, albeit somewhat differently behaving, space ship on a screen. No transfer to a real-life task requiring the trained skills was attempted. In addition to the hierarchical part-task training, the full training group in the Gopher et al. 1994 study received the emphasis-change training described below.

The second space-fortress game group, dubbed "emphasis-only" group, was trained under an emphasis-change regime developed by Gopher and colleagues specifically for complex skill acquisition in an earlier study (Gopher, Weil, and Siegel, 1989). This group practiced the whole game with its full load at all times, but was "led through instructions and auxiliary feedback indicators to vary their focus of attention in different game trials on different aspects of the game." In the earlier study, this technique had proven superior to exposure to the whole task without instruction, but only in the context of the space fortress game. No transfer to real-life tasks was tested in this earlier work.

As is known from the previous section, both game groups outperformed the no-game group in the flights of the 1994 study, showing high transfer of training for the acquired attention management strategies. Moreover, despite the fact that the part-task trained group achieved considerably higher game scores than the emphasis-change group, the two groups did not differ in subsequent flight performance. This may indicate that the whole-task training with emphasis-change experienced by both groups "promoted the development of skill components that could be transferred and generalized to flight, whereas the part-task training and verbal tips contributed only to skill elements that were exclusively relevant to space fortress performance."

This interpretation is supported by work by Fabiani et al. who trained three space-fortress game groups with the whole task, either with or without emphasis training, or the hierarchical part-task training method (Fabiani, Buckley, Gratton, Coles, Donchin and Logie, 1989). Again, the part-task group performed considerably better than the emphasis-change group in the space-fortress game after training. However, when a second unrelated task, such as rapid foot tapping, was introduced in addition to the space-fortress tasks, the emphasis-change group outperformed the part-task group. In all conditions, the whole-task control group lagged behind the two other groups.

In general, careful whole-task practice may thus generalize and transfer better than part-task practice to novel situations, especially if they require multi-tasking skills such as flying an airplane. This was also shown, for a non-aviation multi-task application, by Detweiler and Lundy (1995). They compared the effects of single and dual task training on dual task performance in visual search tasks involving word-category and pattern matching. During the eight training blocks, the learning curve was steeper for the single task group, although this advantage tended to diminish during the course of training at least for the most important variable of hits. When transferring to the dual task condition after training, however, although the single task trained group did show some effect of prior training, the dual task trained group showed a substantial performance advantage over the single task group. This advantage was maintained over at least four training blocks, although the single task trained group showed a trend for catching up later. Detweiler and Lundy attribute this advantage to an opportunity of 1) developing strategies for focusing attention on two tasks at once; 2) acquiring skills in performing both type of memory comparisons together; and 3) learning how to select and execute different responses in close sequence.

In aviation, the motivation for part training is often to complete part of the training in a low-cost part-task simulation and thus saving time in the costly high-fidelity device. This is often supported by the intuitively attractive reasoning that at least initial training may be enhanced by sparing trainees from being exposed to too many tasks all at once. This perception was shared by one of the participants of the instructor/evaluator survey who felt that "realistic radio communications are more important for LOE and LOFT than for initial training." The literature, however, does not appear to support these intuitions for aviation, where unsuccessful multi-tasking has led to tragic accidents such as the 1978 DC-8 crash into a Portland suburb, when a crew distracted by a landing gear problem ran out of fuel (Diehl, 1992).

References

Chomsky, N. (1965). Aspects of the Theory of Syntax. The MIT Press, Cambridge, MA.

Chou, C., D. Madhavan, and K. Funk (1996). Studies of Cockpit Task Management Errors. The International Journal of Aviation Psychology. 6, 307-320.

Detweiler, M., and D. Lundy (1995). Effects of Single and Dual-Task Practice on Acquiring Dual-Task Skill. Human Factors. 37, 193-211.

Diehl, A. (1992). The Effectiveness of Training Programs for Civil and Military Cockpit Management. In Proceedings from FSF 45th IASS & IFA 22nd International Conference. Long Beach, CA.

Dismukes, K., G. Young, and R. Sumwalt (December, 1998). Cockpit Interruptions and Distractions. ASRS Directline. 10, 4-9.

Fabiani, M., J. Buckley, G. Gratton, M. Coles, E. Donchin, and R. Logie (1989). The Training of Complex Task Performance. Acta Psychologica. 71, 259-299.

Frederiksen, J., and B. White (1989). An Approach to Training Based Upon Principled Task Decomposition. Acta Psychologica. 71, 89-146.

Gopher, D., M. Weil and T. Bareket (1994). Transfer of Skill from a Computer Game Trainer to Flight. Human Factors. 36, 387-405.

Gopher, D., M. Weil, and D. Siegel (1989). Practice Under Changing Priorities: An Approach to the Training of Complex Skills. Acta Psychologica. 71, 147-177.

Latorella, K. (1996) Investigating Interruptions: An Example from the Flightdeck. Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting.

McElhatton, J., P. Buchanan, and C. Drew (December, 1998). Crossing Restriction, Altitude Deviations on SIDs and STARs. ASRS Directline. 10, 10-18.

Norman, D., and T. Shallice (1986). Attention to Action: Willed and Automatic Control of Behavior. In R. Davidson, G. Schwartz, and D. Shapiro (Eds.), Consciousness and Self-Regulation, Advances in Research, Vol IV. New York: Plenum Press.

Shiffrin, R., and W. Schneider (1977). Controlled and Automatic Human Information Processing: II. Perceptual Learning, Automatic Attending, and a General Theory. Psychological Review. 84, 127-190.

Sumwalt, R., R. Morrison, A. Watson, and E. Taube (1997). What ASRS Data Tell About Inadequate Flight Crew Monitoring. In Proceedings from The Ninth International Symposium on Aviation Psychology. 977-983.

Appendix C

Instructor/Evaluator Questionnaire

 

Realistic Radio Communications Simulation Questionnaire

 

First Name:

Last Name:

Title:

Company:

Please mail the completed questionnaire to:

Dr. Alfred T. Lee
Beta Research, Inc.
P.O. Box 66423
Scotts Valley, CA 95067

 

Thank you very much for participating in our study. Your input is critical for evaluating the effect of current radio communications simulation practices on both instructor workload and training/evaluation efficiency; and to determine the need for automated tools to provide a realistic radio communications environment.

Questions or comments? Contact Dr. Lee at atlee@ix.netcom.com or 408-353-2665

1) Types of Communications Simulated:

First, we would like to know which types of communications are being simulated in your company's simulator training and evaluation events (LOFT, LOE, and SPOT). Note: At this point, we are only asking about communications with own aircraft.

In the tables below, check the boxes to indicate which types of communications are incorporated into your company's Line Operational Simulation (LOS).

2) How are Communications Simulated?

We would like to learn exactly how these communication simulations are implemented. Please check the boxes that apply.

I/E provides - one instructor provides all simulator operation including radio communications
Live "controllers" - an additional staff member focuses on radio communications simulation
Recorded controller voice - prerecorded, automated controller voice
Synthetic controller voice - prerecorded, automated synthetic controller voice
If you check "Other", please explain in the comments section below.

 

3) Effect and Overall Importance of Radio Communications Simulations:

Here we would like your opinion on how your company's radio communication simulation practices impact instructor workload compared to instructing/evaluating pilots in the air with real radio communications (to own airplane).¹

We also would like your opinion on how those practices impact pilot workload, again compared to what the pilot experiences in the air.

Finally, we would like you to rate the overall importance of realistically simulating each type of communication on training/evaluation effectiveness.

Please circle the ratings that apply.

4) Simulation of ATC Communications With Other Aircraft (partyline):

Now, we would like to find out about your company's simulations of ATC communications with other aircraft (and ground vehicles). In addition, we would like to know if visual simulations of aircraft and ground vehicles are incorporated through out-the-window views or TCAS.

Please indicate below which simulations of other traffic are implemented in your company's training and evaluation events.

5) How are ATC Communications to Other Aircraft Simulated?

(See Question 2 for meanings of ratings.)

6) Effect and Overall Importance of Partyline Simulation:

a) Please rate the effect of your company's partyline simulation practices on instructor workload and on pilot workload, again compared to instructing/evaluating or piloting in the air with a real partyline.²

 

b) Whether or not your company engages in partyline simulation, please rate whether you think the simulation of the partyline increases or decreases training and evaluation effectiveness and how important you feel it is for training and evaluation.

 

7) Radio Communications in Scenarios:

Please indicate how frequently the skills below rely on radio communications as part of the scenario context for team performance. Also, indicate how important you believe the radio communications simulation is to training effectiveness.

 

8) A Few Final Questions

a) Please indicate the percentage of time and effort you spend:

 

b) If the following radio communications simulations were automated, what do you think the effect would be on instructor workload and training effectiveness?

c) What other types of radio communications would you like to see automated, and what do you think the effect would be on instructor workload and training effectiveness?

___________________________________

___________________________________

___________________________________

 

9) Please complete the following background and contact information:

Your Simulator Training/Evaluation Responsibilities:

___________________________________

___________________________________

Your length of employment in this position (include time with your current employer as well as any time in a similar position with a previous employer):
___________________________________ (years, months)

In case we need to contact you for follow-up questions:

Telephone: (_____) (_______-_________)

E-mail, if available:___________________________________

 

If you have any additional comments, please enter them below.

___________________________________

___________________________________

___________________________________

___________________________________

___________________________________

End of Survey!

Thank you very much!

Please mail the completed form to:

Dr. Alfred T. Lee
Beta Research, Inc.
P.O. Box 66423
Scotts Valley, CA 95067

---

Footnotes

1. This I/E did, however, later answer questions regarding the effect of his company's radio communications practices on workload. His data was included assuming the same practices as indicated by his three colleagues from the same airline. [ back ]

2. One of the two I/Es that indicated not simulating ARTCC communications does so during LOEs. The other indicated participating only in LOFT and has two colleagues from the same airline who did indicate simulating ARTCC. [ back ]

3. From his answer to a question later in the survey, it can be concluded that the I/E that skipped the question also does not perform LOEs. [ back ]

4. As is evident from his answer to a later question, the I/E omitting his answer to this question also does not participate in SPOT. [ back ]

5. All but one of the remaining four pilots from this same company reportedly instruct three-person crews. One of these started to add a similar comment into the space provided, but crossed it out. [ back ]

6. Only nine of these had previously indicated that they provided partyline communications in Question 4. [ back ]

7. Only two of these reported percentages for LOE (20 percent) and SPOT (15 percent). [ back ]

8. Given that I/Es consistently indicated simulating company communications via I/E role play (with one exception of dispatch), this analysis includes the one instance each in dispatch, ramp/gate, and cabin where an I/E gave his opinion without having indicated his method of simulation. [ back ]

9. In conjunction with these same inquiries, I/Es were also asked about training effectiveness of radio communications in simulations. Unfortunately, the wording of this question failed to distinguish whether I/Es should answer this question for their method of providing radio communications, or for realistic radio communications in general, as they were asked for overall importance. Due to this ambiguity, the results for training effectiveness in Question three are not included here. [ back ]

10. United Airlines discussed plans to install this audio system on a DC-10 simulation for new-hire screening. [ back ]

11. All the studies using the space fortress game are part of a concerted effort funded by a DARPA contract to the University of Illinois, principal investigator Emanuel Donchin. [ back ]

 

1. For this comparison, you may have to imagine instructor workload with real radio in the air. [ back ]

2. For this comparison, you may have to imagine instructor workload with real radio in the air. [ back ]

---

ERRATA

DOT/FAA/AR-00/62
DOT-VNTSC-FAA-00-13
Office of Aviation Research
Washington, DC

Realistic Radio Communications in Pilot Simulator Training

Judith Bürki-Cohen
Andrew J. Kendra

U.S. Department of Transportation
Research and Special Programs Administration
John A. Volpe
National Transportation Systems Center
Cambridge, MA 02142-1093

Barbara G. Kanki
NASA Ames Research Center
Moffett Field, CA 94035-1000

Alfred T. Lee
Beta Research, Inc.
Scotts Valley, CA 95067

Final Report
December, 2000

Page 5, Section 2.1, line 6: The hyperlink to the FAA AQP Management Web "http://www.faa.gov/avr/afs/agphone.htm" site should read "http://www.faa.gov/avr/afs/aqphome.htm"

Page 22, ATC Communications to Own Airplane, fourth bullet. The last phrase "and two (8 percent) considered it as low (2 percent)" should read "and two (8 percent) considered it as low (2)."

Page 28, Section 4.2.
Line 9: The sentence "Runway incursions can be seen as well as heard" should read "Runway incursions can be seen, with associated communications planned as a future enhancement."
Line 9: The phrase "The product also provides voice simulations of ground traffic from emergency equipment as well as maintenance, fuel, baggage, and aircraft handling" should read "The product also provides visual simulations of ground traffic from emergency equipment as well as maintenance, fuel, baggage, and aircraft handling."

Page 42, Appendix A, line 11: the phrase "communication in stressful circumstances" should read "communications in stressful circumstances."