Summary

The Federal Aviation Administration (FAA) is spending billions of dollars to modernize software-intensive air traffic control systems. GAO examined the processes used to acquire software, using models developed by Carnegie Mellon University's Software Engineering Institute to define and determine an organization's software process maturity. FAA did not fully meet any of the criteria to achieve a "repeatable" level of maturity, the second of a five-level maturity scale. FAA's processes for acquiring costly and complex software are ad hoc, sometimes chaotic, and not repeatable across projects. Moreover, serious process weaknesses have prevented FAA from satisfying the one criterion for the third level of maturity, called "defined." FAA is committed to increasing process maturity in its air traffic control modernization effort, but despite four years of effort it still lacks an effective management approach for improving software processes. In addition, FAA lacks an effective plan to prioritize improvements and measure progress. As a result, it has launched a "hodge podge" of software acquisition improvement efforts without any analytical justification.

GAO noted that: (1) because of the number and severity of FAA ATC modernization software acquisition process weaknesses, FAA did not fully satisfy any of the seven key process areas (KPA) necessary to achieve the "repeatable" level of process maturity; (2) as a result, its processes for acquiring software, the most costly and complex component of ATC systems, are ad hoc, sometimes chaotic, and not repeatable across projects; (3) in addition, serious process weaknesses prevented FAA from satisfying the one KPA specified under the Software Engineering Institute's "defined" maturity level; (4) while FAA showed process strengths, primarily in the solicitation and evaluation KPAs, GAO found extensive weaknesses in these and the remaining six KPAs; (5) some of these weaknesses were systemic, recurring in each of the KPAs; (6) for example, no software project teams measured or reported to management on the status of activities performed, and management never verified that critical activities were being done; (7) these types of problems are some of the reasons for FAA's frequent failures to deliver promised ATC system capabilities on time and within budget; (8) FAA has stated its commitment to increasing ATC modernization process maturity; (9) however, despite 4 years of activity in this area, FAA lacks an effective management approach for improving software acquisition processes; (10) currently, the Software Engineering Process Group (SEPG) is responsible for process improvement, but the SEPG has neither organizational nor budgetary authority over the product teams that acquire software, and, therefore, cannot effectively implement or enforce process change; (11) instead, it can only recommend and encourage change; (12) additionally, FAA does not have an effective plan to correctly target and prioritize improvements and measure improvement progress; (13) in the absence of this plan, it has initiated a "hodge podge" of software acquisition improvement efforts without any analytical justification; and (14) as a result, FAA's process improvement activities have yet to produce more repeatable, better defined, more disciplined software acquisition processes.