ai97047

8/8/2019 ai97047

1/148

United States General Accounting Office

GAO Report to the Chairman, Subcommitteeon Transportation, Committee onAppropriations, House of Representatives

March 1997 AIR TRAFFICCONTROL

Immature SoftwareAcquisition ProcessesIncrease FAA SystemAcquisition Risks

GAO/AIMD-97-47

8/8/2019 ai97047

2/148

8/8/2019 ai97047

3/148

GAO United State sGeneral Accounting OfficeWashington, D.C. 20548Accounting and InformationManagement Division

B-271531

March 21, 1997

The Honorable F rank R. Wolf Chairman, Subcommittee on

TransportationCommittee on AppropriationsHouse of Representatives

Dear Mr. Chairman:

This report re sponds to your request that we review the Feder al Aviation Administrations ( FAA)

air traffic con trol moder nization software acquisition maturity and improvement activities. FAAplans to spend billions of dollars replacing its existing air tr affic con trol systems, bu t has ahistory of performing poorly when acquiring these software -intensive systems. We found thatFAAs software acquisition processes are immature, and are making recommendations to theSecretary of Transportation for strengthening them.

We are se nding copies of this report to the Secreta ry of Transpor tation, the Director of theOffice o f Management and Budget, the Administrator of the Federal Aviation Administration,and other congressional committees . We will also make copies available to othe r interestedparties upon request. If you have questions or wish to discuss the issues in this repor t, pleasecontact me at (202) 512-6412. Major con tributors to this re port are listed in append ix II.

Sincerely yours,

Dr. Rona B. StillmanChief Scientist for Compute rs

and Telecommunications

8/8/2019 ai97047

4/148

Executive Summary

Purpose The Fe deral Aviation Administration ( FAA) is mo dernizing the air tra fficcontrol ( ATC ) systems upon wh ich it will rely to ensure safe, order ly, andefficient air tr avel well into the 21st century. Since software is the mostexpensive and c omplex component of these systems, FAA must use definedand disciplined processes when it acquires software.

Recognizing softwares growing importanc e and prevalence in ATCsystems, the Chairman, Subcommittee on Transportation and RelatedAgencies, House Committee on Appropriations, asked GAO to determine(1) the maturity of FAAs ATC modernization software acquisition processes,and (2) the steps/actions FAA has underway or planned to improve theseprocesses , including any obstacles tha t may impede FAAs progress.

Background To accommodate forecasted growth in air traffic and rep lace agingequipment, FAA embarked on an ambitious ATC modernization program in1981. FAA estimates that it will spend about $20 billion to replace andmodernize software -intensive ATC systems between 1982 and 2003. Ourwork over the years has chronicled many FAA failures in meeting ATCprojects cost, schedule, and performa nce goals, largely because of software -related p roblems. As a re sult of these failures as we ll as thetremendous cost, complexity, and mission criticality of FAAs ATCmodernization program, we designated the pr ogram as a high-risk information technology initiative in our 1995 and 1997 repor t se ries onhigh-risk programs. 1

Software quality is governed largely by the quality of the proce ssesinvolved in developing or acquiring, and maintaining it. Carnegie MellonUniversitys Software Engineering Institute ( SEI), recognized for itsexpertise in software proce sses, has developed models and m ethods thatdefine and dete rmine organizations software p rocess matu rity. Together ,they provide a logical framework for base lining an organizations cur rentprocess capa bilities (i.e., strengths and weaknesses ) and providing astructured plan for incremental process improvement.

Using SEI s so ftware acquisition capab ility maturity mode l ( SA-CMM ),2 SEI ssoftware c apability evaluation method, and SA-CMM authors as consultants,

1High-Risk Series: An Overview (GAO/HR-95-1 , Feb. 1995); High-Risk Series: Information Managementand Technology (GAO/HR-97-9 , Feb . 1997).

2We used a dr aft version of the model for our evaluation (vers ion 00.03, dated April 1996). The firstpublished version of the model was released on October 1996, after we performed our evaluation.According to the mode ls autho rs, the pub lished version differed only editorially from the draft weused.

GAO/AIMD-97-47 Air Traffic Contro lPage 2
http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1

8/8/2019 ai97047

5/148

Executive Summary

GAO staff trained at SEI evaluated FAAs ATC modernization softwareacquisition matu rity in the seven key process a reas ( KPA) necessary toattain a repea table level of process capab ility, and one KPA associatedwith the defined level of process matur ity. 3 Repeatability ensures that anorganization has the necessary process discipline in place to repeat ear liersuccesses on projects in similar domains. Repeatable processes are at thesecon d level on SEI s five-level scale of process maturity. Organizationsthat do not sat isfy the requirements for the repea table level are bydefault judged to be a t the initial level of maturity, meaning that theirprocesses ar e ad hoc, sometimes even chaotic, with few of the proc essesdefined and success de pendent mainly on the heroic efforts of individuals.The one KPA assoc iated with the third level of proce ss matu rity, which iscalled the defined level, is acquisition risk management. It was includedbecause ma ny software experts consider it to be a very important processarea.

As pa rt o f its evaluation, GAO examined five ongoing ATC modernizationprojects selected by FAA .4 These were the Automated Radar TerminalSystem, Display System Replacement , National Airspace SystemInfrastructure Management System, Voice Switching and Contro l System,and the Weather and Radar Processor. (See chapter 1 of this report formore detailed information on GAOs evaluation sc ope and methodology.)

Results in Brief Because of the number and severity of FAA ATC modernization softwareacquisition process weaknesses, FAA did not fully satisfy any of the sevenKPAs nece ssary to ac hieve the repea table level of process maturity. As aresult, its processes for acquiring software , the most costly and complexcomponent of ATC systems, are ad hoc, sometimes chaotic, and notrepeatable across projects. In addition, serious process weaknessesprevented FAA from sa tisfying the one KPA specified under SEI s definedmaturity level. While FAA showed process strengths, primarily in thesolicitation a nd e valuation (i.e., testing) KPAs,5 GAO found extensiveweaknesses in these and the rem aining six KPAs ( i.e., software acquisition

3The seven KPAs relating to the repeatable level are software acquisition planning, solicitation,requirements development and management, project office ma nagement, contract trac king andoversight, evaluation, and transition and support.

4GAO asked FAA to choo se five projects th at are : (1) major efforts with large software a cquisitioncompon ents, (2) mana ged by different FAA product teams , (3) at different life cycle stages , and(4) amon g FAAs bes t ma naged.

5According to the SA-CMM , solicitation is the process of ensuring that award is made to the contractormost capable of satisfying the specified requirements, and evaluation is the process of determiningthat acquired software products and services satisfy contract requirements prior to acce ptance.


8/8/2019 ai97047

6/148

Executive Summary

planning, requirements development and ma nagement, project officemanagement, contract tracking and oversight, transition and support, andacquisition risk management). 6 Some of these weaknesses were systemic,recurring in each of the KPAs. For example, no software project teamsmeasured or reported to management on the status of activitiesperformed, and management never verified that cr itical activities werebeing done. These types of problems are some of the reasons for FAAsfrequent failures to deliver promised ATC system capabilities on time andwithin budget.

FAA has stated its commitment to increasing ATC modernization proce ssmaturity. However, despite 4 years of ac tivity in this ar ea, FAA lacks aneffective management approach for improving software acquisitionprocesses . Currently, the Software Engineering Process Group ( SEPG) isresponsible for process improvement; but the SEPG has neitherorganizational nor budgetary authority over the product teams that acquiresoftware, and, therefore, cannot effectively implement or enforce processchange. Instead, it can only recommend a nd encourage cha nge.Additionally, FAA does not ha ve an effective plan to corr ectly target andprioritize improvements and measure improvement progress. In theabsence o f this plan, it has initiated a hodge podge of softwareacquisition improvement efforts without any ana lytical justification. As aresult, FAAs proce ss improvement a ctivities have yet to produce morerepeatable, better defined, more disciplined software acquisitionprocesses.

Principal Findings

ATC ModernizationSoftware AcquisitionProcesses Are Immature

To attain a given SEI -defined maturity level, an organization m ust satisfythe key practices for the KPAs assoc iated with that level. FAAs ATCmodernization organization had too many weaknesses to sa tisfy any of therepeatable KPAs ( i.e., software acquisition plann ing, solicitation,requirements development and management, project office management,

6According to the SA-CMM , software acquisition planning is the process for ensuring that reasonableplanning for all elements of the software acquisition occur; requirements development andmanagement is the process for establishing an unambiguous and agreed upon set of softwarerequirements; project office management is the process for effective and efficient management of project office ac tivities; contract tracking and oversight is the process of ensuring that contractoractivities, products, and services satisfy contract requirements; transition and support is the process of transferring acquired software products to the eventual support organization; and acquisition risk management is the process of identifying software risks early and adjusting the acquisition strategy tomitigate those risks.


8/8/2019 ai97047

7/148

Executive Summary

contract tracking and oversight, evaluation, and transition and support),nor doe s it satisfy the acquisition risk management KPA from the definedor th ird matu rity level.

For FAA to sa tisfy any of these eight KPAs, it must eliminate the key practiceweaknesses identified in this report. 7 Each practice that is performedeffectively constitutes a strength, and each practice not performed orperformed poorly constitutes a wea kness. While FAAs ATC modernizationhas some strengths, it has more weaknesses . Table 1 tallies these s trengthson the five projects tha t GAO evaluated. In summary, of the tota l numbe r of KPA practices rated, 38 percent constituted strengths, 50 percent wer eweaknesses, and 12 percent were observations. An observation indicatesthat the e vidence was inconclusive and did not clearly support adetermination of either strength or weakness.

Table 1: Collective Number of KPAStrengths, Weaknesses, andObservations on the Five Projects Key Process Area

Number ofstrengths

Number ofweaknesses

Number ofobservations

Software acquisition planning 16 37 7

Solicitation 36 28 14

Requirements development andmanagement

17 35 6

Project office management 26 35 6

Contract tracking and oversight 26 32 6Evaluation 43 21 8

Transition and support 27 32 8

Acquisition risk management 16 46 7

Totals 207 266 62

Additionally, GAO found that wh ile the five projects varied as to prac ticestrengths, weaknesses, and observations under three of the five commonfeatures or practice groupings (commitment to per form, ability toperform, and activities performed), the projects were consistently weak inall practices under the rem aining two groupings (measurement and

analysis and verifying implementation). As a r esult, software project teamsand FAA manageme nt consisten tly lack reliable information on projectteam performance.

7SEI groups e ach of its KPA pract ices into one of five common featur es or practice cate gories. Theseare commitment to perform, ability to perform, activities performed, measurement and analysis, andverifying implementa tion.


8/8/2019 ai97047

8/148

Executive Summary

FAAs Approach forImproving ATCModernization SoftwareAcquisition Processes IsNot Effect ive

To be e ffective, the FAA organization responsible for software acquisitionprocess improvement must have (1) organizational and/or budgetaryauthority over the ATC moder nization units acquiring the software ; and(2) an effective plan to guide improvement efforts and me asure p rogresson each. The FAA organizational entity currently respons ible for ATCmodernization software acquisition process improvement, the SEPG , hasneither. As a re sult, little progress has been made over the last 4 years ininstituting definition and discipline into ATC modernization softwareacquisition processes.

The SEPG is a multilevel committee structure chaired by a member of FAAsChief Information Officers ( CIO) staff. The SEPG is directed by the SoftwareEngineering Executive Committee , which is chaired by the head of the ATCmodernization program. The SEPG has no authority to implement andenforce pr ocess ch ange. Consequent ly, it can only attempt to encourageand persuade software acquirers to establish and follow defined anddisciplined software acquisition processes.

The SEPG and its predecessors have advocated and initiated a collection of efforts intended to strengthen ATC modernization software-relatedprocesses , including software a cquisition processes. However, there is noanalytical basis for the focus, conten t, timing, and interrelationships of these efforts. Specifically, the efforts (1) are not ba sed on any assessme ntof current software acquisition proce ss strengths and weaknesses; and(2) are not d etailed in a formal plan tha t specifies measura ble goals,objectives, milestones, and needed resources, prioritizes efforts, fixesrespo nsibility and accounta bility, and defines metrics for measuringprogress. Instead, these e fforts were undertaken with no sound analyticalbasis and, rathe r than being part of a comprehensive plan, are discussed ingeneral terms without detail and spec ificity in briefing documents, minutesof meetings, and work ing group recommendations. While the SEPG is nowtaking steps to establish the ana lytical basis needed to formulate acomprehensive software pr ocess improvement plan, that plan does not yetexist, and no schedule has been established for completing it.

Recommendations Given the importance and the magnitude of information technology at FAAGAO reiterates its earlier re commendation that a CIO management structuresimilar to the depar tment -level CIOs as presc ribed in the Clinger-Cohen Actof 1996 be established for FAA .8

8Air Traffic Control: Complete and E nforced Architect ure Neede d for FAA Systems Modernization(GAO/AIMD-97-30 , February 3, 1997).

http://www.gao.gov/cgi-bin/getrpt?AIMD-97-30http://www.gao.gov/cgi-bin/getrpt?AIMD-97-30http://www.gao.gov/cgi-bin/getrpt?AIMD-97-30http://www.gao.gov/cgi-bin/getrpt?AIMD-97-30

8/8/2019 ai97047

9/148

Executive Summary

To improve FAAs software acquisition capability for its ATC modernization,GAO recommends that the Secretary of Transportation direct the FAAAdministrator to:

assign respons ibility for software acquisition process improvement toFAAs CIO;

provide FAAs CIO with the authority needed to implement and enforce ATCmodernization software acquisition process improvement;

require the CIO to develop and implement a formal plan for ATCmodernization software acquisition process improvement that is based onthe software capability evaluation results contained in this report andspecifies measurable goals and time frames , prioritizes initiatives,estimates resource requirements, and assigns roles and responsibilities;

allocate adequate resources to ensur e that planned initiatives areimplemented and enforced; and

require that, before be ing approved, every ATC moder nization acquisitionproject have software acquisition processes that satisfy at least SA-CMMlevel 2 requirements .

Agency Commentsand GAOs Evaluation

In its written comments on a dr aft of this report, the Department o f Transportation recognized the importance of mature software acquisitionprocesses, agreed that FAAs processe s are insufficiently matur e,acknowledged that FAA process improvement activities have yet toproduce greater software acquisition process discipline, and reaffirmedFAAs com mitment to impro ving its software acquisition capab ilities usingthe SA-CMM . However, the Department did not st ate what, if any, specificaction it would take on GAOs r ecommenda tions. Additionally, it t ook thepositions that (1) the SA-CMM by itself is inadequate to evaluate ATC systemacquisition capa bilities, is too new to use a s an auth oritative source of guidance, and may have been misapplied by GAO, (2) the repor t does notsufficiently recognize FAAs process improvement organization andprogress nor the difficulties and time required to affect proce ssimprovement change, (3) the SEPG , which is FAAs designated agent for

software acquisition process change, is organized as the Departmen tunderstands other SEPGs to be or ganized, and (4) the re port leads thereader to errone ously conclude that the five programs reviewed are introub le relative to attainment of cost and schedule goals.

None of these positions are valid. First, the SA-CMM , like the SW-CMM(another SEI software -specific capability matur ity model), focuses onsoftware because it is widely recognized as the most d ifficult and costly


8/8/2019 ai97047

10/148

Executive Summary

component of modern computer systems; the one most frequentlyassoc iated with late deliveries, cost overruns, and performance shortfalls;and the one in greatest need of special management attention. Further,while the SA-CMM is relatively new, the p rocesses it requires are wellestab lished, exper ience-based t enets of effective software ac quisition thatare widely supported throughout industry and government. Moreover, GAOapplied the SA-CMM at FAA prope rly and with extraordinary diligence: Everymember of the evaluation team was tra ined at SEI ; the team leader wascert ified by SEI as a lead evaluator; and three SEI professionals, includingtwo authors of the SA-CMM , participated in the evaluation and concurredwith every prac tice determ ination (e.g., strength, weakness).

Second, FAAs many software acquisition process improvement a ctivitieswere unde rtaken without assessing current software acquisition processstrengths and weaknesses, and were not part of a com prehensive plan forprocess improvement. Therefore, FAA had no analytical basis for decidingwhat improvement ac tivities to initiate, or wha t priorities to assign them.Further, although FAA began drafting a plan during the course of GAOsevaluation, it has no sche dule for completing it. In describing FAAsprogress to da te in improving its processes, the repor t delineates a widearray of FAA proce ss improvement ac tivities, but distinguishes theseactivities from ac tual progress . In fact, after 4 years of act ivity, FAA couldnot point to a s ingle case in which it had instituted a more disciplinedsoftware a cquisition process. Since SEI published statistics show that themedian time to improve from software development CMM level 1 to level 2is 26 months , and from level 2 to level 3 is 17 months , it is ent irelyreasonable to expect FAA to be a ble to demonstrate som e improvement inits processes after 4 years.

Third, the issue is not whe ther FAAs SEPG is organized as the Departmentunderstands other SEPGs to be organized, but whether the SEPG , or anyFAA organizational entity responsible for implementing and enforcingsoftware acquisition process change, has the authority needed toaccomplish the ta sk. Current ly, no organizational entity in FAA has the

requisite author ity.

Last, the report addresses the maturity of FAAs software acquisitionprocesses and concludes that these processes are ad hoc andundisciplined, reducing the proba bility that software-intense ATCmodernization projects will consistently perform as intended and bedelivered on schedule and within budget. The repor t does not address theoverall status o f the projects covered by GAOs review, and , there fore,


8/8/2019 ai97047

11/148

Executive Summary

provides no basis for drawing conclusions about the p rojects overall costor schedule performance.

The Departments comments and GAOs evaluation of them ar e presented ingreater detail in chapter 11 of this repor t.


8/8/2019 ai97047

12/148

8/8/2019 ai97047

13/148

Contents

Chapter 7Evaluation

8FAA Is Strong in Most but Not All Evaluation KPA Practices 8Conclusions 9

Chapter 8Transition andSupport

10Transition and Support Not Being Performed Effectively 10Conclusions 11

Chapter 9Acquisition Risk Management

11

ATC Modernization Software Risk Management Is Ineffective 11Conclusions 12

Chapter 10FAA Lacks anEffect ive Approachfor Improving ATCModernizationSoftware AcquisitionProcesses

12FAA Organization Responsible for ATC Software Acquisition

Process Improvement Lacks Authority to Affect Change12

FAA Planning for Software Process Improvement Has Not BeenEffective

12

Improvement Initiatives Have Thus Far Not Instilled SoftwareProcess Discipline

13

Conclusions 13

Chapter 11Overall ConclusionsandRecommendations

13Recommendations 13Agency Comments and Our Evaluation 13

Appendixes Append ix I: Comments Fr om the Department of Transportation 13Append ix II: Major Contr ibutors to This Repor t 14

Tables Table 1: Collective Number of KPA Strengths, Weaknesses, andObservations on the Five Projects

Table 1.1: SA-CMM KPAs Used to Assess FAA SoftwareAcquisition Matur ity

2

Table 2.1: Software Acquisition P lanning F indings for ARTSIIIE 3Table 2.2: Software Acquisition Planning Findings for DSR 3


8/8/2019 ai97047

14/148

Contents

Table 2.3: Software Acquisition Planning Findings for NIMS 3Table 2.4: Software Acquisition Planning Findings for VSCS 3Table 2.5: Software Acquisition Planning Findings for WARP 3Table 3.1: Solicitation Findings for ARTSIIIE 4Table 3.2: Solicitation F indings for DSR 4Table 3.3: Solicitation Findings for NIMS 4Table 3.4: Solicitat ion F indings for VSCS 4Table 3.5: Solicitat ion F indings for WARP 5Table 4.1: Requirements Development and Management F indings

for ARTSIIIE5

Table 4.2: Requirements Development and Management F indingsfor DSR

5

Table 4.3: Requirements Development and Management F indingsfor NIMS

5

Table 4.4: Requirements Development and Management F indingsfor VSCS

5

Table 4.5: Requirements Development and Management F indingsfor WARP

6

Table 5.1: Pro ject Office Management Findings for ARTSIIIE 6Table 5.2: Pro ject Office Management Findings for DSR 6Table 5.3: Pro ject Office Management F indings for NIMS 6Table 5.4: Pro ject Office Management F indings for VSCS 7Table 5.5: Pro ject Office Management F indings for WARP 7Table 6.1: Contract Track ing and Overs ight Findings for ARTSIIIE 7Table 6.2: Contract Tracking and Oversight Findings for DSR 7Table 6.3: Contract Track ing and Oversight Find ings for NIMS 8Table 6.4: Contract Track ing and Oversight Find ings for VSCS 8Table 6.5: Contract Track ing and Oversight Find ings for WARP 8Table 7.1: Evaluation Findings for ARTSIIIE 9Table 7.2: Evaluat ion Findings for DSR 9Table 7.3: Evaluat ion Find ings for NIMS 9Table 7.4: Evaluation Findings for VSCS 9Table 7.5: Evaluation Findings for WARP 9Table 8.1: Transition and Support Findings for ARTSIIIE 10

Table 8.2: Transition and Support Findings for DSR 10Table 8.3: Transition and Support Findings for NIMS 10Table 8.4: Transition and Support Findings for VSCS 11Table 8.5: Transition and Support Findings for WARP 11Table 9.1: Acquisition Risk Management Findings for ARTSIIIE 11Table 9.2: Acquisition Risk Management Findings for DSR 11Table 9.3: Acquisition Risk Management Find ings for NIMS 12


8/8/2019 ai97047

15/148

Contents

Table 9.4: Acquisition Risk Management Findings for VSCS 12Table 9.5: Acquisition Risk Management Findings for WARP 12

Figures Figure 1.1: Summary of ATC Over the Continental United Statesand Oceans

1

Figure 1.2: ARA Organization Chart 2Figure 1.3: SA-CMM Levels and Descriptions 2Figure 2.1: Software Acquisition Planning 3Figure 3.1: Solicitation Summary 4Figure 4.1: Requirements Development and Management

Summary5

Figure 5.1: Project Office Management Summary 6Figure 6.1: Contract Tracking and Oversight Summary 7Figure 7.1: Evaluation Summary 8Figure 8.1: Transition and Support Summary 10Figure 9.1: Acquisition Risk Management Summary 11


8/8/2019 ai97047

16/148

Contents

Abbreviations

AAR Office of Aviation ResearchAAS Advanced Automa tion SystemACT William J. Hughes Technica l CenterAIMD Accounting and Information Management DivisionAIT Office of Information Technology

AND Office of Communication, Navigation, and SurveillanceSystemsARA Associate Administrator for Research and AcquisitionsARINC Aeronautical Radio IncorporatedARTS Automated Radar Terminal SystemASD Office of System Architecture and Investment AnalysisASU Office of AcquisitionsATC air traffic contr olATCSCC Air Traffic Contr ol System Command CenterATS Associate Administrator for Air Traffic ServicesAUA Office of Air Traffic Systems DevelopmentCIO Chief Information Officer

CMM Capab ility Maturity ModelCOTS commercial, off-the-shelf DSR Display System ReplacementFAA Federal Aviation AdministrationGAO General Accounting OfficeIPT Integrated Product TeamISSS Initial Sector Suite SystemKPA key process areaNAS National Airspace SystemNDI non-development itemNIMS NAS Infrastructure Management SystemSA-CMM Software Acquisition Capability Maturity Mode lSE-CMM Systems Engineering Capability Maturity ModelSCE Software Capability EvaluationSEEC Software Engineering Executive CommitteeSEI Software Engineer ing InstituteSEPG Software Engineering Process GroupSW-CMM Capability Maturity Model for SoftwareTRACON Terminal Radar Approach Contr olVSCS Voice Switching and Control SystemWARP Weather and Radar Processor


8/8/2019 ai97047

17/148


8/8/2019 ai97047

18/148

Chapter 1

Introduction

The Federal Aviation Administrations ( FAA) primary mission is to ensuresafe, order ly, and efficient air travel in the national airspace . FAAs abilityto fulfill this mission depends on the adequacy and reliability of thenations air t raffic control ( ATC ) system, a vast network of computerhardware, software, and communications e quipment. 1 The ATC system,however, is being strained by aging equipment, much of which is 1960stechn ology, and growing air traffic. This growth should cont inue as thenumbe r of passengers t raveling on U.S. airlines is expec ted to increase by38 percent between 1995 and 2003, from about 580 million to nearly800 million.

To accommodate the forecaste d growth in air traffic and to relieve theproblems o f the aging ATC system, FAA embarked on an ambitious ATCmodernization program in 1981. FAA estimates that it will spend about$20 billion to replace and modern ize so ftware-intensive ATC systemsbetwe en 1982 and 2003. Our work over the years has chron icled many FAAfailures in meeting ATC projects cost, schedule, and performance goals. 2

As a result of these failures a s well as the treme ndous cost, complexity,and mission criticality of FAAs ATC modernization program, we designatedthe program as a high-risk information te chnology initiative in our 1995and 1997 repor t ser ies on high-risk programs. 3

Overview of ATC Automa ted information proc essing and display, communication,navigation, surveillance, and wea ther r esources pe rmit air trafficcontr ollers to view key information, such as aircra ft location, aircraft flightplans, and prevailing weather c onditions, and to comm unicate with pilots.These resources reside at, or are associated with, several ATCfacilitiesair traffic control towers, terminal radar approach control(TRACON ) facilities, air route tra ffic control center s (en route cente rs),flight se rvice stations, and the Air Traffic Control System CommandCenter ( ATCSCC ). These facilities ATC functions are described below.

Airport towers cont rol aircraft on the ground and before landing and after

take-off when they are within about 5 nautical miles of the airport, and upto 3,000 feet above the a irport. Air traffic controllers re ly on a combination

1The ATC system is a major c ompone nt of the Nationa l Airspace System (NAS).

2Air Traffic Control: Status of FAAs Modernization Program (GAO/RCED-95-175FS , May 26, 1995); AirTraffic Control: Status of FAAs Modernization Program (GAO/RCED-94-167FS , Apr. 15, 1994) ; AirTraffic Control: Status of FAAs Modernization Program (GAO/RCED-93-121FS , Apr. 16, 1993).

3High-Risk Series: An Overview (GAO/HR-95-1 , Feb. 1995); High-Risk Series: Information Managementand Technology (GAO/HR-97-9 , Feb . 1997).

http://www.gao.gov/cgi-bin/getrpt?RCED-95-175FShttp://www.gao.gov/cgi-bin/getrpt?RCED-95-175FShttp://www.gao.gov/cgi-bin/getrpt?RCED-95-175FShttp://www.gao.gov/cgi-bin/getrpt?RCED-95-175FShttp://www.gao.gov/cgi-bin/getrpt?RCED-95-175FShttp://www.gao.gov/cgi-bin/getrpt?RCED-95-175FShttp://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?HR-95-1http://www.gao.gov/cgi-bin/getrpt?RCED-95-175FShttp://www.gao.gov/cgi-bin/getrpt?RCED-95-175FShttp://www.gao.gov/cgi-bin/getrpt?RCED-95-175FS

8/8/2019 ai97047

19/148

Chapter 1Introduction

of technology and visual surveillance to direct aircra ft departu res andapproaches, maintain safe distances between aircraft, and communicateweather-related information, clearances, and other instructions to pilotsand other personnel.

Approximately 180 TRACON s sequence and separate aircraft as theyapproach and leave busy airports, beginning about 5 nautical miles andending about 50 nautical miles from the a irport, and generally up to 10,000feet above the airport , where en route cente rs control begins.

Twenty en route center s control planes over the continental United Statesin transit and during approaches to some airports. Each en route centerhandles a different r egion of airspace , passing contr ol from one to anothe ras respective borders are reache d until the aircraft reaches TRACONairspace . En route cente r controlled airspace usually extends above 18,000feet for commercial aircraft. En route centers also handle lower altitudeswhen de aling directly with a tower, or when agreed upon with a TRACON .

Two en route cente rsOakland and New Yorkalso contr ol aircraft overthe oc ean. Controlling aircraft over ocea ns is radically different fromcontr olling aircraft over land beca use radar sur veillance only extends 175to 225 miles o ffshore . Beyond the rada rs sight, cont rollers mus t re ly onperiodic radio communications through a third partyAeronautical RadioIncorporated ( ARINC ), a private organization funded by the airlines and FAAto oper ate r adio stationsto dete rmine aircraft locations.

About 90 flight service stations provide pre-flight and in-flight services,such a s flight plan filing and wea ther r eport updates, primarily for generalaviation aircraft.

See figure 1.1 for a visual summary of air traffic contro l over thecontinental United States and oceans.


8/8/2019 ai97047

20/148


Figure 1.1: Summary of ATC Over the Continental United States and Oceans

Airport Tower

TRACON

Flight ServiceStation

En RouteCenter

TRACON

OceanicEn Route

Center

ARINC

En RouteCenter

Airport TowerDeparture ControlApproach Control

ContinentalUnited States

Local and Ground Control


8/8/2019 ai97047

21/148


The ATCModernizationProgram Is Complex,Costly, andHistoricallyProblematic

FAA faced two problems in continuing to fulfill its mission to ensur e safe,orde rly, and efficient air travel in the nat ional airspace. First, the ATCsystem of the late 1970s was a blend of several generations of automatedand manual equipment, much of it labor -intensive and obsolete. Second,air traffic was projected to increase d ramatically as a result of airlinederegulations o f the late 1970s. FAA recognized that it could increase ATCoperating efficiency by increasing automation. It also anticipated thatmeet ing the de mand safely and efficiently would require improved andexpanded services, additional facilities and equipment , improved work force product ivity, and the o rder ly replacem ent of aging equipment.Accordingly, in December 1981, FAA initiated its plan to modern ize,automate, and consolidate its enormous ATC system infrastructure by theyear 2000. In do ing so, it chose to acquire new ATC systems by contractingfor systems development ser vices from vendors ra ther tha n building newATC systems in-house .

This ambitious modernization program includes the ac quisition of newsurveillance, data processing, navigation, and communication equipmentin addition to new facilities and suppor t equipment. Totaling over 200separ ate projects, the modernization is estimated to cost over $34 billionthrough the year 2003. Software-intensive ATC systems make up a largeportion of this total, accounting for 169 projects costing $20.7 billion. TheCongress will have provided FAA with approximately $14.7 billion of the$20.7 billion through fiscal year 1997. Many of these projects, for examplethe Display System Replaceme nt and the Voice Switching and ControlSystem, each involve the acquisition of over a million lines of code.Moreover, because the software must operate in a real-time environmentin which human life is at stake , it must be fault tolerant , meaning that itmust be able to monitor its own execution and rec over from failureswithout losing any data.

Over the past 15 years, FAAs moder nization projects have exper iencedsubsta ntial cost overruns, lengthy schedu le delays, and significantperformance shortfalls. To illustrate, the long-time centerpiece o f this

modernization programthe Advanced Automation System ( AAS)wasrestr uctured in 1994 after est imated costs t ripled from $2.5 billion to$7.6 billion and delays in putting significantly less-than-promised systemcapabilities into operation were expected to run 8 years or more overoriginal estimates . Similarly, increases in costs for three other ATC projectshave ranged from 51 to 511 percent , and sche dule delays have averaged

4The three projects and their respective percentage increase in unit costs are the Voice Switching andControl System (511 percent), the Integrated Term inal Weathe r System (129 percent), and t he AviationWeather Observing System (51 percent).


8/8/2019 ai97047

22/148


almost 4 years. For example, the pe r-unit cost e stimate for the VoiceSwitching and Control System increased 511 percent , and the first siteimplementation was delayed 6 years from the original estimate .

AAS and other ATC projects ha ve also experienced shortfalls inperformance. For example, the critical Initial Sector Suite Systemcomponent of AAS , which was intended to replace controllersworkstations at en r oute centers, faced so many technical problems thatits functionality was severely scaled back. In addition, difficulties indeveloping the Air Route Surveillance Radar-4 software and integrating itwith other ATC systems de layed its implementation for years.

ATC Modernizat ionNow ProceedingUnder a NewAcquisitionManagement System

Because of FAAs conte ntion that ma ny of its modern ization problems wereroote d in the Federal Acquisition System, the Congress enacted legislationin October 1995 that exempted FAA from most federal procurement andpersonnel laws and regulations and directed FAA to develop and implementa new acquisition system that would address the unique needs of theagency. 5 At a minimum, the system was to provide for more t imely andcos t-effect ive acquisitions. On April 1, 1996, in response to the Act, the FAAAdministrator began implementation of a new ac quisition managementsystem.

The new system is intended to reduce the time and cost to field newproducts and ser vices by introducing a new investment managementsystem that spans the investments entire life cycles, a new procurementsystem tha t provides flexibility in selecting and managing contractor s, andorganizational learning and cultural reform that supports the ne winvestment management and procurement systems.

This high-level policy promulgated by the new acquisition m anagementsystem is intende d to be supplemented by guidelines in three ar eas:software /systems acquisition, facilities acquisition, and servicesacquisition. These guidelines will be available to FAA staff via the Internet

and were scheduled to be online by October 1, 1996. As of February 1,1997, these guidelines were st ill in draft form and n ot available to FAA staff

5Departme nt o f Transport ation and Related Agencies Appropriat ions Act 1996, P.L. No. 104-50, sec.348, 109 Stat. 436, 460 (1995).


8/8/2019 ai97047

23/148

8/8/2019 ai97047

24/148


Figure 1.2: ARA Organization Chart

Office of AviationResearch

AAR

Office of AcquisitionsASU

Office of Air TrafficSystems

DevelopmentAUA

Acquisition Policyand Procedures

DivisionASU-100

Quality AssuranceDivision

ASU-200

Integrated ProductTeam for En Route

AUA-200

Integrated ProductTeam for Terminal

AUA-300

Integrated ProductTeam for Weatherand Flight ServiceSystems, AUA-400

Integrated ProductTeam for Oceanic

AUA-600

Office ofComm., Navigation,

and SurveillanceSystems, AND

Integrated ProductTeam for

Aircraft/AvionicsAND-600

Office of InformationTechnology

AIT

Office of SystemArchitecture and

Investment AnalysisASD

William J. HughesTech Center

ACT

Chief Scientistfor Human Factors

DivisionAAR-100

ResearchDivision

AAR-200

Research andAcquisitions

International DivisionAAR-300

Airport and AircraftSafety, Researchand DevelopmentDivision, AAR-400

Aviation SecurityResearch andDevelopment

Division, AAR-500

Integrated ProductTeam for Air Traffic

ManagementAUA-500

ContractsDivision

ASU-300

Integrated ProductTeam forCommunications

AND-300


SurveillanceAND-400

Integrated ProductTeam for GPS/

NavigationAND-500


InfrastructureAND-100

Integrated ProductTeam forInformation Systems

AIT-200

Integrated ProductTeam for IT

ServicesAIT-300

Integrated ProductTeam for ITAcquisitions

AIT-400

CorporateInformation

Management Div.AIT-100

Architectureand SystemEngineering

ASD-100

Evaluation andConfigurationManagement

ASD-200

NAS Programmingand FinancialManagement

ASD-300

Investment Analysisand Operations

ResearchASD-400

ResourceManagement

DivisionACT-100

ATC Engineeringand Test Division

ACT-200

CNS Engineeringand Test Division

ACT-300

Aviation Simulationand Human

Factors DivisionACT-500

FacilitiesManagement

DivisionACT-400

Airport Managementand Emergency

Operations DivisionACT-600

AssociateAdministrator

for Research andAcquisitions

ARA

Object ives, Scope,and Methodology

The Chairman, Subcom mittee on Transportat ion and Related Agencies,House Committee on Appropriations, requested that we review FAAsability to acquire software for ATC systems. Our objectives were todetermine (1) the maturity of FAAs ATC moder nization software acquisitionprocesses; and (2) the steps/actions FAA has underway or planned toimprove these processes, and a ny obstacles that may impede FAAsimprovement actions.

To determine FAAs software acquisition process matur ity, we applied theSoftware Engineering Institutes Software Acquisition Capability MaturityMode l ( SA-CMM )6 and its Software Capability Evaluation ( SCE ) method. SEI sexpertise in software process assessment is accepted throughout theindustry. Our evaluators were all SEI -trained software specialists. In

6We used a dr aft version of the model for our evaluation (vers ion 00.03, dated April 1996). The firstpublished version of the model was released in October 1996, after we performed our evaluation.According to the mode ls autho rs, the pub lished version differed only editorially from the draft weused.


8/8/2019 ai97047

25/148


addition, we employed SEI consultants, two of whom are authors of themode l, as advisors to ensure p roper application of the model.

The SA-CMM ranks organizational matu rity according to five levels (seefigure 1.3). Maturity levels 2 through 5 require the verifiable existence anduse of certa in software acquisition processes, known as key process areas(KPA). According to the SEI , an agency that has these acquisition processesin place is in a much bette r position to successfully acquire softwar e thanan organization that does not have these processes in place. We evaluatedFAAs software acquisition proc esses against all level 2 KPAs and one level 3KPA (see table 1.1). We included one level 3 KPAacquisition risk managementbecause it is considered by software experts to be a veryimportant process area.


8/8/2019 ai97047

26/148


Figure 1.3: SA-CMM Levels and Descriptions

The acquisition organization's software acquisitionprocess is documented, standardized, and establishedas the standard software acquisition process. Allprojects use an approved, tailored version of theorganization's standard software acquisition process

for acquiring their software products and services.

Level 3 - Defined

Detailed measures of quality of the software acquistionprocesses, products, and services are collected. Thesoftware processes, products, and services arequantitatively understood and controlled.

Level 4 - Managed

Continuous process improvement is empowered byquantitative feedback from the process and from pilotinginnovative ideas and technologies.

Level 5 - Optimizing

Basic project management processes are establishedto track performance, cost, and schedule. Thenecessary process discipline is in place to repeatearlier successes on projects in similar domains.

Level 2 - Repeatable

Continuouslyimprovingprocess

Predictableprocess

Standardconsistant

process

The software acquisiton process is characterized asad hoc, and occasionally even chaotic. Few processesare defined and success depends on individualeffort.

Level 1 - Initial

Disciplinedprocess


8/8/2019 ai97047

27/148


Table 1.1: SA-CMM KPAs Used toAssess FAA Software AcquisitionMaturity

SA-CMM Level 2 Keyprocess areas Description

Software acquisition planning Ensuring that reasonable planning for the softwareacquisition is conducted and that all elements of theproject are included.

Solicitation Ensuring that award is made to the contractor mostcapable of satisfying the specified requirements.

Requirements developmentand management

Establishing a common and unambiguous definition ofsoftware acquisition requirements understood by theacquisition team, system user, and the contractor.

Project office management Managing the activities of the project office and

supporting contractor(s) to ensure a timely, efficient, andeffective software acquisition.

Contract tracking andoversight

Ensuring that the software activities under contract arebeing performed in accordance with contractrequirements, and that products and services will satisfycontract requirements.

Evaluation Determining that the acquired software products andservices satisfy contract requirements prior toacceptance and transition to support.

Transition and support Providing for the transition of the software products beingacquired to their eventual support organization.

CMM Level 3Key process area

Description

Acquisition risk management Identifying risks as early as possible, adjusting acquisitionstrategy to mitigate those risks, and developing andimplementing a risk management process as an integralpart of the acquisition process.

As esta blished by the mode l, each KPA contains five common attributesthat indicate whether the implementation and institutionalization of a KPAcan be effective, repea table, and lasting. The five common features a re:

Commitment to perform. Commitment to perform d escribes the actionsthat the organization must take to e stablish the proc ess and ensure that itcan endure . Commitment to pe rform typically involves es tablishingorganizational policies and sponsor ship.

Ability to per form. Ability to perform descr ibes the preconditions thatmust ex ist in the project or o rganization to implement the softwareacquisition proce ss compe tently. Ability to perform typically involvesresources, organizational structures, and training.

Activities performed. Activities per formed descr ibes the roles andprocedures nec essary to implement a KPA . Activities performed typicallyinvolve establishing plans and procedures, performing the work, trackingit, and taking appropriate management actions.


8/8/2019 ai97047

28/148


Measur ement a nd ana lysis. Measur ement and ana lysis describes ac tivitiesperformed to measure the process and analyze the measurements.Measur ement a nd ana lysis typically includes defining the measurements tobe taken and the ana lyses to be conducted to deter mine the status andeffectiveness of the ac tivities per formed.

Verifying implementation. Verifying implementation descr ibes the steps toensure tha t the activities are performed in compliance with the processthat has been established. Verification typically encompasses r eviews bymanagement.

In accordance with SEIs SCE method, for each KPA in level 2 and the one KPAin level 3 (risk management ), we evaluated institutional FAA policies andpractices and compared project-specific guidance and practices againstthe five comm on attr ibutes. This project-specific comparison can result inone of four possible outcom es: (1) project st rengthan effectiveimplementation of the key practice; (2) project we aknessineffectiveimplementation of a key pract ice or failure to implement a key pract ice;(3) project obser vationkey prac tice evaluated but evidence inconclusiveand cannot be characterized as either strength or weakness; and (4) notratedkey practice not c urrently relevant to pro ject, therefore notevaluated.

We pe rformed the project-specific evaluations on five ongoing ATCmodernization projects, each of which is described be low. We asked FAA tochoose these projects using the following criteria: (1) the project s aremajor efforts with large software acquisition componen ts; (2) the projectsare m anaged by different integrated pr oduct teams, (3) the projects are indifferen t stages of their life cycles, and (4) the projects are among FAAsbest-managed acquisitions. The projects tha t FAA selected for ourevaluation are:

Automa ted Radar Terminal System ( ARTS ) IIIE: ARTS gathers informationfrom surveillance sensors, processes it, and sends it to air trafficcontrollers in terminal radar approach control facilities and control towers

at airport s. A series of improvements to ARTS have provided increasedprocessor ca pacity and the ability to support a greater number of contr oller displays. The ARTS IIIE improvements p rovide for morecontr oller positions and surveillance sensor inputs at se lected largefacilities. ARTS IIIE is operational at New York, Chicago, and Dallas/FortWorth with additional systems planned for Southern California andDenver. FAA estimates tha t the enhancement will cost $383.8 million todevelop and dep loy.


8/8/2019 ai97047

29/148

8/8/2019 ai97047

30/148


initiative; (3) the r elative priority of each; (4) progress made on ea chinitiative; and (5) obstacles, if any, impeding progress. We also analyzedprocess improvement plans, meeting minutes, and related docum entationto further address these areas. Finally, we interviewed representativesfrom the ATC modernization product teams and the SEPG to obtain theirperspectives in assessing process improvement support, activities,progress, and obstacles.

The Department of Transportation provided written comments on a draftof this report. These comments are prese nted and evaluated in chapter 11,and are reprinted in appendix I. We performed our work at FAAheadquarters offices in Washington, D.C. between March 1996 andFebruary 1997, in accordanc e with generally accepted governmentauditing standards.


8/8/2019 ai97047

31/148

Chapter 2

Software Acquisition Planning

The purpose of software acquisition planning is to ensure tha t reasonableplanning for the software ac quisition is conduc ted and tha t all aspec ts of the total software acquisition effort are included in these plans. Accordingto the SA-CMM , a repeatab le software acquisition planning process, amongother things, includes (1) addressing software life cycle support inacquisition plans, (2) preparing life cycle software cost e stimates,(3) having a written so ftware acquisition policy, (4) measuring andreporting on the sta tus of software acquisition planning activities, and(5) having guidance on software training and experience requirements forproject personnel.

FAAs SoftwareAcquisition PlanningProcess Is NotEffective

All five p rojects have some ability and/or activity strengths in this KPA . Forexample, every project addresses software life cycle support in planningdocuments a nd software life cycle cost estimates we re prepa red for fourof the projects. However, we found many more process weakne sses thanstrengths. For example, FAA has a systems acquisition policy, but thepolicy does no t specifically address or p rovide guidance on softwareacquisition. There fore, FAA management has not forma lly recognized theimportance and uniqueness of software ac quisition issues in the systemacquisition process, and has not formally committed to managing softwareacquisition in a disciplined ma nner. Also, the product tea ms do notmeasure or report on the status of software acquisition planning activities.As a result, management is not always aware of problems in projectperformance, and cannot always take corrective action expeditiously.Additionally, none of the five projects has specific guidance on softwar etraining or experience requirements for pr oject part icipation. As a resu lt,software training is ad hoc, and decisions about project perso nnelassignments are subjective.

Figure 2.1 provides a comprehensive listing of the five projects strengths,weaknesses, and observations for the software acquisition planning KPA .The spec ific findings suppor ting the pra ctice r atings cited in figure 2.1 arein tables 2.1 through 2.5.


8/8/2019 ai97047

32/148

8/8/2019 ai97047

33/148

Chapter 2Software Acquisition Planning

Table 2.1: Software Acquisition Planning Findings for ARTSIIIEAutomated Radar Terminal System IIIE

Key Practice Finding Rating

Commitment 1 The acquisition organization has a writtenpolicy for planning the software acquisition.

The system acquisition policy does notadequately address software, e.g., it doesnot address items that should be inc ludedin software planning such as contracttracking and oversight, requirementsdevelopment, evaluation, and riskmanagement.

Weakness

Ability 1 Personnel are assigned the responsibility

for performing software acquisition planning.

No personnel are assigned the

responsibility for software acquisitionplanning.

Weakness

Ability 2 The acquisition organization hasexperienced software acquisitionmanagement personnel.

The acquisition organization has noguidance regarding training or experiencerequirements for project participation.

Observation

Ability 3 Software acquisition managementpersonnel are experienced in the domain ofthe project.

There are no guidelines that define domainknowledge or experience.

Weakness

Ac tivity 1 Software ac quisition p lanning personnel areinvolved in system acquisition planning.

No one on the product team is specificallyassigned responsibility for softwareacquisition.

Weakness

Ac tivity 2 The projec ts software acq uisition planningis documented and the planningdocumentation is maintained over the life ofthe project.

There is no documented softwareacquisition plan.

Weakness

Activity 3 The software acquisition strategy for theproject is developed.

There is no software acquisition strategy. Weakness

Activity 4 Software acquisition planning includesprovisions for ensuring that the life cyclesupport of the system is included inplanning documentation.

The product team ensures that life cyclesupport is included in planningdocumentation.

Strength

Activity 5 A life cycle cost estimate for the softwareactivity is prepared and independentlyverified.

The life cycle cost estimate was preparedbut not independently verified.

Weakness

Measurement 1 Measurements are made and used todetermine the status of the softwareacquisition planning activities.

No internal process measurements aretaken and used to determine the status ofactivities for any of the key process areas.

Weakness

Verification 1 The ac tivities for software acquisitionplanning are reviewed by acquisitionorganization management on a periodicbasis.

While the Integrated Product Team leaderreviews the status of the contract and thecontractors cost and schedule, he does notreview the status of the activities that arerequired to be performed for any of the keyprocess areas.

Weakness

Verification 2 The ac tivities for software acquisitionplanning are reviewed by the projectmanager on both a periodic andevent-driven basis.

While the product team leader reviews thestatus of the contract and the contractorscost and schedule, he does not review thestatus of the activities that are required tobe performed for any of the key processareas.

Weakness


8/8/2019 ai97047

34/148

8/8/2019 ai97047

35/148


Table 2.3: Software Acquisition Planning Findings for NIMSNAS Infrastructure Management System




Weakness



The team members are assigned the

responsibility for software acquisitionplanning.

Strength



Observation



Weakness


The team members for software acquisitionare assigned collective responsibility andare actively involved in system acquisitionplanning.

Strength


At this early stage in the program, thesoftware acquisition planningdocumentation is being written but is notcomplete.

Observation


Officials stated that the software acquisitionstrategy will be contained within theacquisition plan.

Strength


Software acquisition planning includesprovisions for ensuring that life cyclesupport is included in planningdocumentation.

Strength


A life cycle cost estimate for software hasbeen prepared and independently verified.

Strength


No internal process measurements aretaken to determine the status of activities forany of the key process areas.

Weakness



Weakness

(continued)


8/8/2019 ai97047

36/148


NAS Infrastructure Management System




Weakness


8/8/2019 ai97047

37/148


Table 2.4: Software Acquisition Planning Findings for VSCSVoice Switching and Control System




Weakness



Personnel are assigned the responsibility


Strength



Observation



Weakness


Software acquisition personnel are involvedin system acquisition planning.

Strength


The projects software acquisition planningis not documented.

Weakness

Activity 3 The software acquisition strategy for theproject is developed. No software acquisition strategy exists. Thesystem acquisition strategy does notaddress software.

Weakness


The life cycle support of the system isincluded in the acq uisition planningdocumentation.

Strength


The life cycle cost estimate has beenprepared and independently verified.

Strength


No internal process measurements aretaken or used to determine the status ofactivities for any of the key process areas.

Weakness

Verification 1 The ac tivities for software acquisition

planning are reviewed by acquisitionorganization management on a periodicbasis.

While the Integrated Product Team leader

reviews the status of the contract and thecontractors cost and schedule, he does notreview the status of the activities that arerequired to be performed for any of the keyprocess areas.

Weakness



Weakness


8/8/2019 ai97047

38/148


Table 2.5: Software Acquisition Planning Findings for WARPWeather and Radar Processor




Weakness



Although the product team stated that they

are assigned collective responsibility forsystems acquisition, they could not providedocumentation to show a specificassignment for software acquisition.

Observation



Observation



Weakness


Although the product team is responsiblefor systems acquisition, there is no onespecifically assigned for software nor doesany document expressly state that software

is part of systems acquisition.

Weakness


There is no software acquisition plan. Weakness


There is no software acquisition strategy forthe project. The system acquisition strategycovers only software enhancements.

Weakness


The acquisition plan includes provisions forensuring that life cycle support is included.

Strength


The life cycle cost estimate is prepared andindependently verified.

Strength



Weakness



Weakness

(continued)


8/8/2019 ai97047

39/148


Weather and Radar Processor




Weakness

Conclusions Effective planning is the cor nerstone of succe ssful software acquisition.While FAA showed some strengths in this KPA , its many weaknesses renderthe software acquisition planning capability ad hoc and chaotic. Therefore,it is unlikely that projects ar e effectively measur ing and monitoringsoftware acquisition progress and taking corre ctive actions expeditiously.


8/8/2019 ai97047

40/148



8/8/2019 ai97047

41/148

8/8/2019 ai97047

42/148

Chapter 3Solicitation

Figure 3.1: Solicitation Summary

Commitment 1

Commitment 2

Commitment 3

Ability 1

Ability 2

Ability 3

Activity 1

Activity 2

Activity 3

Activity 4

Activity 5

Activity 6

Activity 7

Activity 8

Measurement 1

Key Practice ARTSIIIE DSR NIMS VSCS WARP

The acquisition organization has a written policy forthe conduct of the solicitation. Strength Strength Strength Strength Strength

Responsibility for the software portion of thesolicitation is designated.

Observation Strength Strength

A selection official has been designated to beresponsible for the selection process and thedecision.

Strength Not rated Strength Strength Not rated

A group that is responsible for coordinating andconducting the solicitation activities exists.

Strength Strength Strength Not rated Strength

Adequate resources are provided for t he solicitationactivities.

Individuals performing the solicitation activities haveexperience or receive training.

Observation Observation Observation Observation Observation

The project team documents its plans for solicitationactivities.

Strength Not rated Strength Observation Strength

The project's solicitation activities are performed inaccordance with its plans.

Observation Not rated Strength Strength

The project team documents its plans for proposalevaluation activities.

Strength Strength Observation Strength

The project team's proposal evaluation activities areperformed in accordance with its plans.

Not rated Strength Strength

A cost estimate and schedule for the softwareactivity being sought are prepared.

Strength Strength Strength Strength Strength

The software cost estimate and schedule areindependently reviewed for comprehensiveness andrealism.

Strength Observation Strength Strength Strength

The groups supporting the solicitation (e.g., enduser, systems engineering, support organization, andapplication domain experts) receive orientation onthe solicitation's objectives and procedures.

Not rated Strength Observation

The project team and the offeror review the project'ssoftware requirements during negotiations to ensuremutual understanding.

Observation Strength Not rated Observat ion Strength

Measurements are made and used to determine thestatus of the solicitation activities.

Weakness Weakness

Weakness Weakness Weakness Weakness Weakness

Weakness

Weakness

Weakness Weakness

Weakness Weakness



8/8/2019 ai97047

43/148


Verification 1

Verification 2

Key Practice ARTSIIIE DSR NIMS VSCS WARP

Key practice not implemented

Key practice effectively implemented

Key practice evaluated but evidence inconclusive. Cannot characterize as either strength or

Weakness

Strength

Observation

=

=

=weakness

Key practice not currently relevant to project, therefore not evaluatedNot rated

The activities for solicitation are reviewed by thedesignated selection official or acquisitionorganization management on a periodic basis.


The activities for solicitation are reviewed by theproject manager on both a periodic and event-drivenbasis.


=


8/8/2019 ai97047

44/148


Table 3.1: Solicitation Findings for ARTSIIIEAutomated Radar Terminal System IIIE


Commitment 1 The acquisition organization has a writtenpolicy for the conduct of the solicitation.

FAA Order 1810.1F is the written policy. Strength

Commitment 2 Responsibility for the software portion of thesolicitation is designated.

Officials gave conflicting answers as to whois responsible for the software portion of thesolicitation, and could not provide adocument that formally designatesresponsibility.

Weakness

Commitment 3 A selection official has been designated tobe responsible for the selection processand the decision.

The Administrator was the selection officialfor the sole-source contract.

Strength

Ability 1 A group that is responsible for coordinatingand conducting the solicitation activitiesexists.

A group (matrix team) is responsible forcoordinating and conducting the solicitationactivities.

Strength

Ability 2 Adequate resources are provided for thesolicitation activities.

No mechanism exists for identifyingresources required and for ensuring that theneeded resources are provided to theproject.

Weakness

Ability 3 Individuals performing the solicitationactivities have experience or receivetraining.


Observation

Activity 1 The project team documents its plans for

solicitation activities.

The team documents its plans for

solicitation activities.

Strength

Activity 2 The projects solicitation activities areperformed in accordance with its plans.

While officials stated that solicitationactivities are performed in acc ordance withits plans, they could not providedocumentation to support this.

Observation

Activity 3 The project team documents its plans forproposal evaluation activities.

The team does not document its plans forproposal evaluation activities.

Weakness

Activity 4 The project teams proposal evaluationactivities are performed in acc ordance withits plans.

No evaluation plan exists, therefore, theteam could not perform in accordance withits plan.

Weakness

Activity 5 A cost estimate and schedule for thesoftware activity being sought are prepared.

A cost estimate and schedule wereprepared.

Strength

Activity 6 The software cost estimate and scheduleare independently reviewed forcomprehensiveness and realism.

The cost estimate and schedule wereindependently reviewed.

Strength

Ac tivity 7 The group s sup porting the solic itation (e.g .,end user, systems engineering, supportorganization, and application domainexperts) receive orientation on thesolicitations objectives and procedures.

No orientation briefing occurred. Weakness

(continued)


8/8/2019 ai97047

45/148


Automated Radar Terminal System IIIE


Activity 8 The project team and the offeror review theprojects software requirements duringnegotiations to ensure mutualunderstanding.

Officials stated that meetings were held withthe contractor to ensure mutualunderstanding, however, they could notprovide documents to support this.

Observation

Measurement 1 Measurements are made and used todetermine the status of the solicitationactivities.


Weakness

Verification 1 The activities for solicitation are reviewed bythe designated selection official oracquisition organization management on a

periodic basis.

While the Integrated Product Team leaderreviews the status of the contract and thecontractors cost and schedule, he does not

review the status of the activities that arerequired to be performed for any of thevarious key process areas.

Weakness

Verification 2 The activities for solicitation are reviewed bythe project manager on both a periodic andevent-driven basis.


Weakness


8/8/2019 ai97047

46/148


Table 3.2: Solicitation Findings for DSRDisplay System Replacement



There is an FAA policy that addressessolicitation conduct.

Strength


Officials stated that responsibility for thesoftware portion of the solicitation has beenassigned, however, they could not providedocuments to support this.

Observation


Not applicable. DSR was a change orderfrom an existing larger contract that wentthrough the acquisition phase in 1984.Current team members joined the teamafter the change order was negotiated and,therefore, could not address this keypractice.

Not rated


A group responsible for the solicitationexists.

Strength



Weakness



Observation

Activity 1 The project team documents its plans forsolicitation activities.


Not rated


Not applicable. DSR was a change orderfrom an existing larger contract that wentthrough the acquisition phase in 1984.Current team members joined the teamafter the change order was negotiated and,therefore, could not address this key

practice.

Not rated


The produc t team uses a change orderevaluation plan to document plans forproposal evaluation activities.

Strength

(continued)


8/8/2019 ai97047

47/148


Display System Replacement




Not rated


A cost estimate and schedule weregenerated.

Strength

Activity 6 The software cost estimate and schedule

are independently reviewed forcomprehensiveness and realism.

Officials could not produce documentation

that supported their statement that thesoftware cost estimate and schedule wereindependently reviewed.

Observation



Not rated


A series of scheduled meetings were heldto ensure mutual understanding ofrequirements.

Strength



Weakness

Verification 1 The activities for solicitation are reviewed bythe designated selection official oracquisition organization management on aperiodic basis.


Weakness

Verification 2 The activities for solicitation are reviewed bythe project manager on both a periodic andevent-driven basis.


Weakness


8/8/2019 ai97047

48/148


Table 3.3: Solicitation Findings for NIMSNAS Infrastructure Management System



The Acquisition Management System is thewritten policy.

Strength


Responsibility for the software portion of thesolicitation has been designated to softwareexperts on the team.

Strength


A selection official has been d esignated tobe responsible for the selection processand dec ision.

Strength


The contracting officer, support staff, andthe parent ASU organization areresponsible for coordinating andconducting the solicitation activities.

Strength



Weakness



Observation

Activity 1 The project team documents its plans forsolicitation activities.

The product team documents its plans forsolicitation activities.

Strength


Officials stated that solicitation activities willbe performed in acc ordance with its plans.NIMS is in the presolicitation phase.

Strength


The project team has documented its plansfor proposal evaluation activities.

Strength


In accordance with the plan,prequalification was completed andvendors were down-selected from it.

Strength


The Acquisition Program Baseline includesa cost estimate and schedule for thesoftware acquisition.

Strength

Activity 6 The software cost estimate and scheduleare independently reviewed forcomprehensiveness and realism.

An independent assessment was done. Strength


Solicitation activities orientation wasconducted for NIMS personnel.

Strength


The NIMS project has not yet reached thisstage, therefore, this activity was not rated.

Not rated

(continued)


8/8/2019 ai97047

49/148


NAS Infrastructure Management System



No internal process measurements aretaken to determine the status of activities forany of the key process areas.

Weakness

Verification 1 The activities for solicitation are reviewed bythe designated selection official oracquisition organization management on aperiodic basis.


Weakness

Verification 2 The activities for solicitation are reviewed by

the project manager on both a periodic andevent-driven basis.

While the product team leader reviews the

status of the contract and the contractorscost and schedule, he does not review thestatus of the activities that are required tobe performed for any of the key processareas.

Weakness


8/8/2019 ai97047

50/148


Table 3.4: Solicitation Findings for VSCSVoice Switching and Control System



FAA Order 1810.1F is the written policy. Strength


Officials gave conflicting answers as to whois responsible for software acquisition, andcould not p rovide documentation thatformally designates responsibility.

Weakness


The Administrator was the selection official. Strength


Officials stated that a group wasresponsible for coordinating andconducting solicitation activities; however,they could not

Documents

ai97047