K gallagher tosborne

Overview of GAO Quicklook

PM Challenge

February 2012

Katie Gallagher and Tracy OsborneOCFO/SID

Outline• Background

– Why is this important? • GAO’s Approach

– What projects are involved?– Why does GAO request technical data?

• What should I expect if my project is in Quicklook?

• What does NASA’s data say?• Agency Messages

1/22/2012 2

Background• The GAO’s Assessment of Large-Scale [NASA] Projects is

Congressionally-mandated

• GAO sends its findings to Congress in an annual “QuickLook Book” (QLB) Report focusing on changes in project cost and schedule and providing GAO’s explanations for these changes

• GAO is frequently called upon to testify about the results.

• Engagement objective: “Identify and gauge the progress and potential risks associated with selected NASA acquisitions.” (i.e., projects)

• GAO uses data from QuickLook Book (QLB) in other audits, such as the ongoing High Risk audit, and vice versa.

31/22/2012

4

Why the Stakeholder Interest?

• NASA’s historical inability to control cost and schedule growth– JWST, CxP, MSL growth cited in several provisions establishing new

reports.– A 2004 GAO finding that “NASA lacks discipline in cost estimation.”– Multiple Operating Plans per year to address cost growth or phasing.

• Based on past Performance NASA is required to report to OMB, Congress and GAO routinely on Project cost and schedule. – All appropriations since FY 2008 have included direction for GAO to

“Identify and gauge the progress and potential risks associated with selected NASA acquisitions.”

1/22/2012

Typical Results from Quicklook Reports

• Most of NASA’s cost growth for the current portfolio is attributed to projects baselined prior to 2009

• NASA projects do not meet GAO-identified best practice standards for technology maturity and design stability

• Other challenges that affect NASA project outcomes include Funding, Launch Vehicle, Development Partners, Parts Quality, Contractor Management

1/22/2012 5

GAO’S APPROACHWhy do they ask about TRL’s and drawings?

1/22/2012 6

GAO’s Report is in Two Parts• Agency Observations

– Examines adherence to NASA policies such as 7120.5E and 1000.5A

– Examines potential systemic reasons for NASA’s cost growth

– Assesses NASA’s efforts to improve cost and schedule growth, including JCL policy

• Project Assessments– Provides an update on status of each project

included in the audit– Outlines challenges specific to each project that

GAO believes may lead to cost growth

1/22/2012 7

GAO Quicklook assesses performance of NASA’s major projects and the Agency’s management of those projects.

How does a project get picked?• GAO evaluates all spaceflight projects in formulation if a contract is

released that has development content and is greater than $25 M– Can include projects in Phase A and Pre-Phase A if they have such a contract

• GAO evaluates all spaceflight projects in development with a life cycle cost greater than $250 M

• GAO continues to track projects in Operations until NASA negotiates removal from Quicklook– Minimal data collection on Operations projects; main focus is cost and

scientific results

• For projects in formulation, GAO expects NASA to provide cost and schedule ranges

• For projects in development, GAO expects NASA to provide baseline cost, schedule, design and technical information

1/22/2012 8

Projects Included in Audit2008 2009 2010 2011

* Estimated Lifecycle Cost >$250M* In development* In formulation if it also has released a development contract valued at >$50M

(consistent with other external reporting requirements)

* Estimated Lifecycle Cost >$250M* In development* In formulation if it also has released a development contract valued at >$25M* If in operations, it is not included

* Estimated Lifecycle Cost close to $250M (if it is called out in the budget as a discreet project then it may be included)* In development* In formulation if it also has released a development contract valued at >$50M * In operations, in the prime mission phase

* Estimated Lifecycle Cost close to $250M (if it is called out in the budget as a discreet project then it may be included)* In development* In formulation if it also has released a development contract valued at >$50M * In operations, in the prime mission phase

1 Aquarius Development X X X X2 Ares Formulation X X X3 Dawn Operations X X X4 Fermi/ GLAST Operations X X X5 GPM Development X X X X6 Glory Development X X X X7 GRAIL Development X X X8 Herschel Operations X X X X9 ICESat II Formulation X X

10 JWST Development X X X X11 J uno Development X X X12 Kepler Operations X X X X13 LADEE Development X X14 LDCM Development X X X X15 LRO Operations X X X X16 MAVEN Development X X17 Mars 2016 (TGO) Formulation X18 MMS Development X X X19 MPCV Pre- Formulation X20 MSL Development X X X X21 NPP Development X X X X22 OCO/ OCO-2 Development X X X23 Orion Formulation X X X24 Phoenix Closed Out X25 RBSP Development X X X26 SDO Operations X X X X27 SLS Pre- Formulation X28 SMAP Formulation X X29 SOFIA Development X X X X30 Solar Probe Plus Formulation X X31 TDRS Development X X32 Wise Operations X X X X

Requirement for Inclusion

Project

Lifecycle Phase as of

Entrance Conference (April 2011)

1/22/2012 9

10

GAO “Business Case” Model

• Knowledge Point 1: Resources and Requirements Match– Indicated by a high level of technology maturity

• Heritage is technology is appropriate in form, fit and function• New Technologies at least at TRL-6 (for NASA)

• Knowledge Point 2: Product Design is stable “about midway through development”– Best practice is to achieve design stability by midpoint of development– GAO cites 90% of engineering drawings releasable as figure of merit

• Knowledge Point 3: Manufacturing Processes are mature– Product shown to be producible within cost, schedule, and quality targets and have demonstrated

reliability and Design demonstrates needed performance through realistic system-level testing

Pre-phase A

Phase A

Phase B Phase C Phase D Phase E

KDP-A KDP-B KDP-C KDP-D KDP-D

PDR CDR SIR

Concept and Technology Development Production

Technology Development Engineering Manufacturing and Development Production and Deployment Operations and

Support

MS-B MS-CPDR

NASA

DOD

GAO

Knowledge Point 1

Knowledge Point 2

Knowledge Point 3

1/22/2012

CDR

Engineering Manufacturing and Development

Production Rarely Applies to NASA Projects

Data Requested AnnuallyData Category Element 2008 2009 2010 2011 Frequency

Design Stability X X X X Annual + UpdatesCritcal Technologies X X X X Annual + Updates

Heritage Technologies X X X Annual + UpdatesSoftware Complexity X X AnnualQuality Parts Issues X X Annual

MPAR Baseline X X X X As OccursARRA Funding X X Semi-annual

KDP-B Est. LCC Range X X X X Semi-annualKDP-C Baseline X X As Occurs

ICEs X X As OccursJ CLs X X As Occurs

Project-held UFE X X Monthly (in MSRs)MD-held UFE by year X Annual

Key Milestones X X X X Semi-annualAgy vs Mgmt Schedule Comparison X Semi-annual

Basic Information X X X X Semi-annualAward Fee Strucuture X X X Semi-annualEVM (CPR Formats) X X Monthly

FAD/PCA X X X X As OccursProject Plan X X X X As Occurs

Control Plans X X X X As OccursPDR/CDR Packages X X X X As Occurs

CADRes X X As OccursSRB Reports X X As Occurs

KDP-C Decision Memos X As OccursKDP-C Datasheets X As OccursMSR Presentations X X Monthly

Documentation

Technical

Cost

Contracts

Reporting Year

Schedule

11GAO reviewed this list for 2011, and agreed to reduce frequency of some requests1/22/2012

WHAT SHOULD I EXPECT?So, my project is in Quicklook…

1/22/2012 12

DCI-C&s

The Quicklook Annual Cycle (2011)

MarchFeb

Entrance Conference

May DecAug

JuneOct

Exit Conference

DCI-proj

AprilJuly Sept

NovJan

Site Visits – JPL &GSFC

Action Item Response

Other Data Requests

2-Pgr Cmts Agency Cmts

Addt’l Data

Addt’l Actions

Final Report

Site Visits – MSFC

&JSCDCI-C&s

DCI-contract

DCI-C&s

DCI = “Data Collection Instrument” Three types:1.Project DCI – for technical data2.Cost and Schedule (C&S) DCI – provided by OCFO3.Contracts DCI – provided by OP

131/22/2012

14

GAO transmits request

• The TPOC and relevant Audit Liaisons (ALR) should be copied on all requests

• PM please forward requests to ALR and QLB TPOC when necessary

Project drafts response

• Provide document and/or explanation

• When document or data is not available, explain why

• Draft response provided to PE, ALR, and QLB TPOC

Response reviewed by HQ & Ctr Mgmt

• Possible reviewers include PE, MD, PCFO, Procurement, OCE, OSMA, LSP

• QLB TPOC and MD ALR will coordinate the appropriate reviews.

QLB TPOC transmits response to GAO

• ALR, PM, PE, and reviewers will be copied

Process for Working with GAO

Standard 3 week turn-around when not otherwise stipulated by GAO

Quicklook Technical POC (TPOC) discusses request with GAO to

understand purpose for requested items

Extra explanation may be provided to help contextualize the data or to better assist the GAO in answering

their questions

1/22/2012

15

• Legislatively, the GAO has right of access to the data, regardless of procurement sensitivity or classification

• NASA can negotiate with GAO regarding the provision of data; if there is a disagreement, GAO’s recourse is legislative action

• GAO is bound by the same rules as NASA for handling the data, and NASA can specify data restrictions

• NASA actively works with GAO on data formats, types and context for interpretation

GAO Data Access Rights

1/22/2012

NASA provides the data requested and works with the GAO to add context and help the GAO analysts to understand NASA’s data.

Data Collection Instruments• GAO asks PMs to fill in a

Data Collection Instrument (DCI) annually and following PDR and CDR

• Data collection includes– Critical and heritage

technology TRLs– Drawing release– Parts Quality issues,

including impact ($)– Software TLOC and

Blocks– Partners

• GAO requests data as of PDR, CDR, and current

• Procurement and OCFO fill in similar DCIs on contracts, cost, and schedule1/22/2012 16

PMs should NOT leave any sections blank; if the data is not available or not applicable, say so and explain why.

Example Technical DCIGAO Data Collection Instrument:

Date Submitted to NASA Project Office:

NASA Project Office Response Required by:GAO Contact:Project:NASA Project Office Instructions

KP ESTIMATE PDR CDR September 2010 update LatestDate 7/12/2010 Scheduled for July 2011 9/16/2010 5/3/2011No. of Releasable

DrawingsNo. of Total

Drawings ExpectedSource Project Office Project Office Project Office Project Office

Key Event Dates Defined: The PDR column should reflect information as of the Preliminary Design Review prior to KDP C (NAR). The CDR column should reflect information as of the Critical Design Review. The "Latest" column should reflect the the number of drawings releasable and expected as of the date this spreadsheet is returned to GAO.

Overall Design Knowledge

See below for Spacecraft and Instruments comments. The project assesses design stability primarily thru the Systems Engineering organization. A weekly Systems Design Team regularly addresses design, design margins, analyses, interfaces, subsystem and instrument design topics, and proposed changes. Proposed design changes are then processed thru standard Configuration Control Boards. In addition, weekly status reviews as well as a Monthly Management Review take place that addresses overall status and issues.

0

1537 sheets expected for spacecraft, 818 drawings expected for instruments.

10 released for spacecraft, 16 released for instruments.

947 spacecraft sheets released to date. 299 drawings released for instruments.

Comments & Issues: Enter comments and issues below or on a separate sheet.

1879 sheets expected for spacecraft.

2437 sheets expected for spacecraft, 1023 drawings expected for instruments.

If the total number of drawings is not known or if the total number of drawings has changed, please provide an explanation in the space below. If the project has completed CDR, please describe any measures and metrics used in addition to engineering drawings to assess design stability. If the project has yet to complete its CDR, please identify the measures and metrics the project intends to use to assess design stability.

2437 sheets of spacecraft drawings planned, 1023 drawings expected for instruments.

Joe AnalystAwesome

Please provide or update the total number of releasable engineering drawings and the total number expected at key dates listed below. The term "releasable" refers to those drawings (physical hard copy or computer-generated packages) that are sent to be manufactured. Please calculate the number of drawings by using the number sheets within a package, not just the number of packages. Sometimes these drawings are considered the "build to" drawings. These drawings include details on the parts and work instructions needed to make the product and reflect the results of testing. Change any dates or data as needed.

Design Stability

5/3/2011

5/6/2011

Site Visits• GAO will prepare for site visits with the projects by reviewing project monthly

status review (MSR or PSR) packages; discussion questions will be transmitted in advance of the visit.

• GAO may not visit every project included in the Quicklook; projects not visited will receive written questions based on the content of their monthly review packages.

• PMs are invited to bring any charts or personnel that they need to help answer the questions; however, GAO prefers to have an informal discussion – NOT a lecture.

• When preparing responses, keep in mind that Quicklook is about cost growth against the Agency’s MPAR Baseline; responses should keep in mind the full mission cost – including launch vehicle and UFE

• Refer questions about things you are not responsible for to correct organization– LV questions should generally be directed to LSP– MD-held UFE questions should be directed to the MD, although it is ok to state the value

• When in doubt, it is ok to take an action to provide a response after the visit.

1/22/2012 17

This is the project manager’s opportunity to add context and explain all the good work their team is doing.

2-Page Project Summary Anatomy

1/22/2012 18

Project Challenges Overview

Project Essentials

Contract Info

Project Performance Summary

Project Timeline Project Update

Project Office Comments

Draft 2-Page Project SummariesWhat to look for

• Overview and Project Essentials – Is the information correct?

• Project Update and Summary– Does the GAO accurately portray the status?– Are there any errors in the analysis?

• Project Challenges– GAO’s challenge categories are fixed and will

not change– Assertion of a challenge should be supported

by the analysis described in the Project Update

• Contract Info, Project Performance, and Project Timeline – no action from PM required; Procurement and

OCFO will review to ensure consistency with other external reporting

How to Respond1. Project comments

– Will be printed in the “Project Office Comments” section of the final report

– Limited to 500 characters (appr 100 words)2. General Comments

– Focus on errors in understanding or interpretation of information previously provided to GAO as well as points of disagreement with the GAO’s analysis.

– These comments will not appear in the report but may help drive the discussion with the GAO about how they can revise

3. Technical Comments– Comments provided in this section will be used

to correct any factual errors in the report. – Corrections must be accompanied by evidence

1/22/2012 19

GAO will transmit draft summaries directly to project managers with response requested in 5 business days; Response must go through full agency review process in this time.

WHAT DID THE DATA SAY?So we gave GAO all this data…

1/22/2012 20

GAO Quicklook – Mar 2011• Maintained recommendation for GAO’s “Business Case Approach,” but recognized

improvements due to JCL policy• Found average cost growth of 14.6% and schedule growth of 8 mo; nearly all growth

attributed to projects baselined before 2009• Main Observations:

– Technology Challenges: • NASA begins development with insufficient technology maturity• Evidence: Majority of projects passed PDR with critical technologies at TRL < 6

– Design Challenges: • Projects pass CDR with unstable design• Evidence: Nearly all projects passed CDR with <90% releasable drawings

• Additional Observations– Funding Challenges: JWST, Ares, Orion funding “did not align with plan”– Launch Vehicle Challenges: mainly medium class – addressed more fully in another GAO report– Development Partner Challenges– Parts Challenges – addressed more fully in GAO Quality Parts Audit– Contractor Management Challenges

• Report stated that GAO will conduct a more thorough analysis of NASA’s EVM data in future audits and that [cost] transparency is insufficient during formulation1/22/2012 21

22

Mar 2011 QLB - Cost Growth

NASA agrees with GAO’s cost and schedule growth figures

The majority of NASA’s cost and schedule growth is attributed to projects baselined prior to 2009

1/22/2012

Funding Issues Launch Issues Parts Issues Technology Issues

Design Issues Contractor Issues

Development Partner Issues

0

2

4

6

8

10

122009

2010

2011

# Pr

ojec

ts w

ith

GA

O-id

entifi

ed Is

sue

Challenges SummaryQuicklooks published in 2009, 2010, 2011

Subjects of other GAO audits1/22/2012 23

Data Provided During 2011

Technology Maturity and Drawing Release

• Self-reported data from DCIs submitted to GAO appear to support the hypothesis that immature technologies at PDR leads to cost growth for NASA missions

– The distribution does not support a strong correlation

– JWST and MSL drive the correlation, but have cost growth clearly attributable to other factors.

• The data shows low correlation between releasable drawings at CDR and cost growth

1/22/2012 24

MSL did not provide drawing data; MAVEN is excluded from CDR analysis because CDR was not complete at time of DCI submission

0% 5% 10% 15% 20% 25% 30%-10.0%

10.0%

30.0%

50.0%

70.0%

90.0%

110.0%

130.0%

150.0%

Class A

Class B

Class C

Class D

% Immature Technologies at PDR

Perc

enta

ge D

evel

opm

ent C

ost

Gro

wth MSL

NPP

JWST

0.00%

20.00%

40.00%

60.00%

80.00%

100.00%

120.00%-30.0%

-10.0%

10.0%

30.0%

50.0%

70.0%

90.0%

110.0%

130.0%

150.0%

% releaseable drawings at CDR

Perc

enta

ge c

ost g

row

th

NPP

GPM

SOFIA

Juno

JWST

GAO hypothesis


Drawing Release

CDR Now# Releasable A C

Total # anticipated B D

NASA: % releasable = A/BGAO: % releasable = A/D

NASA’s 90% guidance is based on what is known at the time

• There appears to be some disagreement between how NASA and GAO calculate the percent of releasable drawings

• NASA calculates the percentage of releasable drawings at CDR based upon the total number of drawings anticipated as of CDR

• Drawing growth after CDR typically occurs as a result of risk realization; it is impossible to predict which risks will come to pass and which of those will require drawing changes

• By NASA’s calculation, several projects did meet the best practice guideline of 90% releasable drawings at CDR

Aquarius GPM GRAIL Juno JWST LADEE LDCM MMS MSL NPP OCO-2 RBSP SOFIA TDRS0.00%

10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%

100.00%

% of Drawings Releasable at CDR

GAO Calculation Assumed NASA calculation

1/22/2012 25


Software

• 13 projects provided complete software data– 4 projects - Aquarius, MSL, NPP, and JWST - reported growth in TLOC greater than 25%– 5 projects reported a decrease in TLOC since PDR – Juno, LDCM, RBSP, SOFIA, TDRS

• After removing projects whose cost growth is clearly attributed to other factors, the mathematical correlation between development cost growth in growth in TLOC is high (.91) but the lack of significant distribution does not support a relationship between cost growth and software growth

26

-40% -20% 0% 20% 40% 60% 80% 100% 120%-20.0%0.0%

20.0%40.0%60.0%80.0%

100.0%120.0%140.0%160.0%

NPPSOFIA

JWST

GPM

LDCM

MSL and Aquarius Excluded

% Growth in TLOC, PDR to Current

% D

evel

opm

ent C

ost G

row

th

-100% 0% 100% 200% 300% 400% 500% 600% 700%-20.0%0.0%

20.0%40.0%60.0%80.0%

100.0%120.0%140.0%160.0%

NPPSOFIA Aquarius

MSL

JWST

GPM

All Quicklook Projects

% Growth in TLOC, PDR to Current% D

evel

opm

ent C

ost G

row

th

1/22/2012

AGENCY MESSAGESWhat is Headquarters saying to GAO?

1/22/2012 27

Formulation• Formulation is an iterative

process that aims to balance requirements and resources.

• At KDP-C the Decision Authority agrees that the design is sufficiently mature and the requirements match the resources

• The confirmation decision results in an agreed-to cost and schedule baseline; budget is committed to the project as part of the PPBE process

• The balance of requirements and resources is continuously assessed throughout a project’s lifecycle

ResourcesRequiremen

ts

Changes in schedule or projected LCC during formulation should not be treated as slips or cost growth; rather, this is a normal part of the formulation process.1/22/2012 28

Road to Confirmation(For SMD Projects to date)

PDR

SRB C&S (JCL) Assessment

SRB Technical Assessment

Program Office Recommendation

Division (Theme) Recommendation

Center Recommendation

ICEs or ICAs1

Project Report

Project Response to SRB

DPMC APMCCMC



SMD Recommendation

ICEs or ICAs1

Project Report


OCFO/SID Analyst Assessment



Project Report


ICEs or ICAs1

Approval to Proceed (Cat 2&3)2

Approval to Proceed

(Cat 1)

The Confirmation decision takes into account many inputs. There is no “right answer”

JCL JCL JCL

1. when available2. Cat 2 and 3 missions do not require an APMC, but Cat 1 missions do. With 7120.5E, the AA will approve entry to Implementation for all projects

1/22/2012 29

Center Recommendation

JCLs and UFE• The use of JCL in decision making represents a shift in the

paradigm of how project cost estimates, budgets and schedules are established.

• This method of estimation moves beyond prior practices of calculating a “point estimate” and adding an additional level of reserves.

• Because the UFE is a product of the probabilistic basis of estimates, any reduction to the UFE will reduce the probability of achieving the project cost and schedule targets

• Usage of UFE should not be interpreted as a “funding issue;” the UFE is part of the full amount that NASA budgets for a mission.

• The term “UFE” should be used rather than “reserves” to denote the more rigorous cost estimating policy NASA now uses

1/22/2012 30

With the release of 7120.5E, NASA no longer uses the term “reserves.” PMs should update charts to use the term

“Project-held UFE.”

Management vs Agency CommitmentsCost Schedule

Management Agreement• Managed by PM• Based upon Project’s

own Cost Estimate• Should be at 50% JCL or

greater

Agency Baseline Commitment• Based upon JCL and other

inputs• At 70% JCL or as

determined by the DA• Difference between ABC

and MA is budgeted as MD-held UFE

Agency Baseline Commitment• Based upon JCL and other

inputs• At 70% JCL or as

determined by the DA• Difference between ABC

and MA is additional schedule margin held by the Agency

Project-held UFE

MD-Held UFE

Project Schedule

Margin

Agency Schedule

Margin

Management Agreement Managed by PM

Based upon Schedule developed by PM

Should be at 50% JCL or greater

In some cases, the Decision Authority deems it prudent for the Agency to hold additional schedule margin above the project’s schedule. In these cases, the Management Schedule should not be viewed as acceleration of the Agency’s schedule

Not to scale

1/22/2012 31

What happens when funds are not appropriated or performance is poor?

o When requested funds are not appropriated, or performance is poor, work is pushed to future years, and budget requirements increase in the out years.

o This may lead to cost growth and operating plan changes.

o If reductions happen in subsequent years, the profile becomes unsustainable. FY10 FY11 FY12 FY13 FY14

400

450

500

550

600

650

700Notional Example

FY10 Budget request

FY10 Appropriation + FY11 request



1/22/2012 32

EVMTotal Project (Combined In- and Out- of House)

IIn-House

Out-of-House

ContractsPartnersCP & FPI

>$50MCP & FPI >$20M

CP & FPI <$20M FFP

7 Principles

Certified System

32 Guidelines NA NA NA

7 Principles applied across all elements of Total Project

• NASA’s projects are accomplished through combined efforts of suppliers and NASA in-house efforts– Total In-House with targeted procurements– Prime contract with government providing Level-Of-Effort (LOE) oversight– Many unique mixtures of the above two scenarios

• NASA applies EVM to contractors at the project level and requirements are flowed down to subcontractors

NASA projects tend to be complex; EVM is not necessarily applied to all suppliers but it is addressed at

the total project level.

EVM is one of many project management tools that NASA uses. NASA recognizes EVM implementation as an area of concern and is committed to improvement in this area

1/22/2012 33

Technical Leading Indicators

• NASA is enhancing the ability to monitor and assess the stability of programs and projects as well as the maturity of their products through the development process

• Comprehensive 3-Step Approach1. The Entrance and Success Criteria for Lifecycle reviews are being

enhanced to further account for product maturity• These will be documented in the Rev B update to NPR 7123.1

2. A required set of 3 parameters that will address product stability and maturity will be added as a NID to NPR 7123.1A• Mass Margins• Power Margins• Review item closures

3. A set of recommended Common Leading Indicators is being developed• Added to NPR 7120.5E Formulation Agreement and Program/Project Plan

templates

1/22/2012 34

Any Questions?

1/22/2012 35