D brown gbezos_shirshorn

National Aeronautics and Space Administration

www.nasa.gov

Risk Management during Integrated Systems Research Technology Development

NASA Project Management (PM) Challenge 2012February 22-23, 2012

Presented by: Douglas Brown, Environmental Responsible Aviation (ERA) Risk ManagerGaudy Bezos-O’Connor, ERA Deputy Project ManagerSteven Hirshorn, Integrated Systems Research Program Systems Engineering & Integration Manager

2

AbstractCo-Authors: Justin Hornback (former ERA RM), Gaudy Bezos-O’Connor (ERA DPM), Steve Hirshorn (ISRP SE&I Mgr), and Douglas Brown (ERA RM)

The Environmental Responsible Aviation (ERA) Project, through system-level analyses, will select promising N+2 vehicle and propulsion concepts and technologies to develop based on their potential benefit toward simultaneously reducing fuel burn, noise and emissions.

These concepts and technologies will then be matured and their performance will be evaluated at the system and sub-system level in relevant environments.

Risk management is a set of activities aimed at achieving success by proactively risk-informing the selection of decision alternatives and then managing the implementation risks associated with the selected alternative.

In technology development efforts the measure of success to apply risk management assessment criteria is not easily defined. The ERA Risk Management Process developed a process to define and assess risks to achieving project success across a portfolio of independent sub-project technology development, that, when combined are required to maximize likelihood of achieving ERA project goals. ERA project goals are to select promising N+2 vehicle and propulsion concepts and technologies based on their potential benefit toward simultaneously reducing fuel burn, noise and emissions. The ERA risk management process added another layer of risk assessment; identifying a contribution factor based on subject matter expert input for each technology weighing each technology potential contribution to ERA goals. This was then applied to risks identified to the ERA project. The additional weighting factor improves identification of the true impact of project risks to achieve goals compared to traditional risk management processes. This presentation describes the process used to develop the weighting factor and share ERA's experience during implementation.

N+1, N+2 and N+3 reflect time periods for technology insertion into the aircraft fleet:• N+1: 2015-2020, N+2: 2020- 2025, N+3: 2025 +

3

Presentation Focus: The Risk Management Challenge

• What is the appropriate risk management construct for an Aeronautics Technology Development Project responsible for maturing airframe and propulsion technologies from TRL 3 to 5/6 through Integrated Systems Research?

• Factors:– Cultural and Project Execution Paradigm Shift from Fundamental Research to

Technology Development– Technology Development Projects are finite life, not enduring– Geographically dispersed government team – High Degree of Industry/OGA Partnerships/Collaborations with significant

costshare across Project Portfolio (FY10-12)– Varied or limited experience with project risk management outside of

airworthiness risks for aeronautics flight research projects.– The large number of independent tasks part of the portfolio of ERA

• No project integrated critical path– Must balance portfolio content versus schedule margin and budget reserves

• Budget reserves address prototype test article challenges and unique capability investments.

• Schedule margin address technical risks and facility challenges

4

Presentation Outline

• What is Aeronautics Integrated Systems Research– How does it differ from Aeronautics Fundamental Research?

• Overview of ARMD, ISRP and ERA Project• Aeronautics National Goals and ERA Project Technical Challenges• ERA Project Goals, Approach and Deliverables• ERA’s Risk Informed Decision Making Strategy

5

What is Aeronautics Integrated Systems Research?

N+1, N+2 and N+3 reflect time periods for technology insertion into the aircraft fleet:• N+1: 2015-2020• N+2: 2020- 2025• N+3: 2025 +

NASA Aeronautics Portfolio

6

Conduct cutting-edge research that will produce innovative concepts, tools, and technologies to enable revolutionary changes for vehicles that fly in all speed regimes.

Directly address the fundamental ATM research needs for NextGen by dev-

eloping revolutionary concepts, capabilities, and technologies that

will enable significant increases in the capacity, efficiency and

flexibility of the NAS.

Fundamental Aeronautics Program Airspace Systems Program

Conduct cutting-edge research that will produce innovative concepts, tools, and technologies to improve the intrinsic safety

attributes of current and future aircraft.

Preserve and promote the testing capabilities of one of the United States’ largest, most versatile and comprehensive set of flight and ground-based research facilities.

Aeronautics Test Program

Integrated Systems

Research Program

Conduct research at an integrated system-level on promising concepts and

technologies and explore/assess/demonstrate the benefits in a relevant environment

Aviation Safety Program

Program Goal: Conduct research at an integrated system level on promising concepts and technologies and demonstrate the benefits in a relevant environment

Integrated Systems Research Program Overview

7

Environmentally Responsible Aviation (ERA) Project

Explore and assess new vehicle concepts and enabling technologies through system-level experimentation to simultaneously reduce fuel burn, noise, and emissions

Unmanned Aircraft Systems (UAS) Integration in the National Airspace System (NAS) Project

Contribute capabilities that reduce technical barriers related to the safety and operational challenges associated with enabling routine UAS access to the NAS

ERA Project Technical Challenges (FY10-15)

Innovative Flow Control Concepts

for Drag Reduction

Advanced Composites for

Weight Reduction

Advanced UHB Engines for SFC & Noise Reduction

Advanced Combustors for

LTO Oxides of Ni reductions

Airframe & Engine Integration for

Community Noise Reduction

ERA Project Goals: Simultaneous Achievement of

the NASA Subsonic Transport System-Level Metrics (N+2 Timeframe)-75% LTO & -70% Cruise NOx Emissions

below CAEP6

-42dB below Stage 4 Community Noise

-50% Aircraft Fuel/ Energy Consumption

National R&D PlanEnergy and Environment

Enhance Mobility National Security

Traceability from National R&D Plan to ERA Project Technical Challenges

8

ERA Project Goals, Approach and Deliverables

9

• Project System-Level Performance Metrics:• Simultaneous achievement of the community noise, emissions and fuel burn

metrics defined in the NASA Subsonic Transport System Level Metrics in the N+2 timeframe

• Project Approach: • Combine rigorous systems analysis with large-scale, integrated systems

research demonstrations of promising airframe and propulsion technology solutions to TRL 5/6 by 2015

• Increase the viable trade space of vehicle configurations that can simultaneously meet the goals

• Project Definition: • 6-year life; • 2 Phases: Phase 1 (FY10-12); Phase 2 (FY 13-15)

• Project Deliverables: Key Performance Parameters (KPPs)• Technology Readiness Level (TRL) Maturation Maps• Product Transition Opportunities: Technology Transition Maps• Vehicle-level system metrics to measure progress towards the Project Goals• Technical data to validate/enhance system and physics-based assessment

tools

10

ISRP and ERA Risk Management Plan

Implement both Continuous Risk Management (CRM) and Risk Informed Decision Making (RIDM)– 8000.4A, Agency Risk Management Procedural Reqs.– 7120.08, NASA R&T Program & Project Mgmt. Reqs.

• Research and Technology Risk Management – Flight hardware focused– Risk management must balance the need

to conduct challenging technology development that will realize significant gains.

Advanced Combustor Testing

Lightweight Integrated Structures Testing

Propulsion Airframe, Aeroacoustic, and Propulstion Airframe AeroAcoustic Wind

Tunnel Testing

Flight Testing and Advanced Vehicle Concepts

Advanced Propulsor Testing

11

ISRP and ERA Project Risk Management Strategy

• Risk management at the program level will tend to be strategic and focused on ensuring the success of the projects, while risks that focus on the tactical technical / cost / schedule execution risks will be largely managed at the project-level with program insight.– ISRP Projects will manage their technical performance, schedule, and cost risks

according to their Risk Management Plans. – Significant project risks or risks requiring resources beyond those available to the

Project will be “Tracked” by or “Elevated” to the Program.

• A common frame of reference for Likelihood and Consequence (L&C) exists between ISRP and its Projects and across Projects:– Stems from the need to be able to reference both Program and Project risks in an

apples-to-apples comparison when communicating risks to the ISRP Program Director and to the Mission Directorate.

– Enable traceability of risks from Project task level to ISRP Programmatic Risks

• In order to ensure both Program and Project processes remain in sync and remain aware of risks at both levels, frequent good communications must be maintained .

12

Assessment of Risk in a TechnologyDevelopment Project• Technology risks follow a different risk pattern than other types of risks• The level of maturity for a technology affects the risk profiles of that technology,

i.e. lower maturity levels come with higher risks• The technology landscape is constantly changing with new technologies coming

online promising increased performance. Risk assessment must weigh the promise of new performance against the confidence of what has been done before

• These considerations do not require us to change likelihood and consequence criteria but consider the lens through which we view these scales

13

Technological Considerations For Risk Assessment

Technology Maturity Support BaseRating Value

ERA Likelihood Ratings

Technical

Some Research Completed/ Never Done Before

No Other Program Developing Similar Technology 5 Very High PTCS > 75%

New Design Based On Existing Technology

One Other Program Developing Similar Technology 4 High 50% < PTCS <= 75%

Major Redesign Of Existing Technology More Than One Program Developing Similar Technology 3 Moderate 20% < PTCS <= 50%

Minor Redesign A Few Parallel Programs 2 Low 5% < PTCS <= 20%

Existing Multiple Parallel Programs 1 Very Low PTCS <= 5%

Rating 5 4 3 2 1

ERA Risk Ratings Consequences

Major impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, and KPP Goals

Moderate impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, and KPP Goals

Some impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, and KPP Goals

Minor impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, and KPP Goals

Negligible or no impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables, and KPP Goals

Contingency Solutions

No Acceptable Alternatives

Some Possible Alternatives

Single Acceptable Alternative

A Few Known Alternatives

Several Acceptable Alternatives

Reliability Factor Reliability May Not Be Increased

Fairly Confident Reliability Will Increase Somewhat

Highly Confident Reliability Will Increase Somewhat

Fairly Confident Reliability Will Increase Significantly

Highly Confident Reliability Will Increase Significantly

Considerations

Con

sid

era

tion

s

14

ERA Project’s Risk Assessment Approach

• ERA has applied a tailored continuous risk management process that enable risk- informed decision making– For Phase 1 Portfolio (FY10-12)– For Phase 2 Portfolio Development in FY12 for authorization to proceed through a

Key Decision Point R(KDP) Review Process

• Risk Management Process defined:– Risk Factors for technical, cost and schedule– Consequence and Likelihood Definitions and Scoring– Parent-Child Risk Construct– ERA Project Risk Reporting– ERA Risk Factor Weighting

• Current Phase 1 Portfolio• Assessment of Phase 2 Portfolio Opportunities

15

ERA Project Phase 1 Portfolio Continuous Risk Management Process

• Risk Factors:– Technical Risk

• Industry/OGA contributions• Technical Complexity (Test Article, Experiment/Test)• Technical Benefit /System Impact

– Cost Risk:• Workplan cost estimate maturity/fidelity• Technical Complexity• Workplan resource availability• Acquisition/Procurement cost • Industry/OGA collaboration dependency

– Schedule Risk:• Workplan maturity/WBS fidelity• Technical Complexity• Workforce resource availability• Facility/Flight test asset availability• Acquisition/Procurement schedule • Industry/OGA collaboration dependency

16

ERA Risk Consequence Criteria

Rating Value

ERA Risk Ratings Consequences

Technical Cost ScheduleDecision/

NotificationCommunication

5Major impact to achievement of

Subsonic Transport System Level Metrics, Technical Deliverables,

and KPP Goals

Greater than 20% increase over that

allocated budget (Sub-Project, Element or

Task level)

Level 1 (APGs) any impact

Level 2 Milestone(s):< 1 month impact

Level 3,4 Milestone(s): ≤ 1 month impact

Integrated Systems Research

Program (ISRP) & Centers

Aeronautics Research Mission

Directorate

4Moderate impact to achievement of Subsonic Transport System Level Metrics, Technical Deliverables,

and KPP Goals

Between 15% and 20% increase over allocated budget (Sub-Project,

Element or Task level)



ERA Project Management

(PM)ISRP & Centers

3Some impact to achievement of


and KPP Goals





ERA PM Centers

2Minor impact to achievement of


and KPP Goals





ERA Sub-Project

Managers (SPM)

ERA Project Manger (PM)/ Deputy PM

(DPM)

1Negligible or no impact to

achievement of Subsonic Transport System Level Metrics, Technical

Deliverables, and KPP Goals





SPM DPM, Element & Task Leads

Cost and Schedule easy to quantify and assess.

Challenge to define Negligible, Minor, Moderate, and Major impact

17

ERA Project Phase 1 Portfolio Continuous Risk Management Process• Consequence Elements:

– Technical Risk• Achievement of Subsonic Transport System Level Metrics, • Technical Deliverables, • Key Performance Parameters (KPPs)

– Technical Challenges– TRL Maturation– Technology/Product Transition Roadmap

– Cost Risk :• % increase over that allocated budget (Sub-Project, Element or Task level)

– Schedule Risk:• Level 1 Milestones: ISRP Program Level• Level 2 Milestones: Project Level• Level 3,4 Milestones: Sub-Project and Task Level

18

ERA Project Phase 1 Portfolio Continuous Risk Management Process• Likelihood Criteria may be defined in either Generalist terms or Probabilistic

terms:• Generalist: Improbable Unlikely May Likely Very Likely• Probabilistic: % or occurrence, Probability distribution (i.e. 10-5), etc.

19

ERA Parent and Child Risks

• All identified project risks start as Child Risks. Risks that apply to project level metrics are then elevated to the Parent level and are then managed by the Risk Management Board.– Child risks reported to Risk Management board, votes to make the risk a parent

based on the risks assessment at the project level.– Allows risk owner at the sub-project, element or task level to assess the risk in terms

of their approved plan.– Allows ERA project management to assess the risk at the project level where they

have a better understanding of the entire project • Preventing filtering or tweaking of the risk at the sub-project/element/task level.

WBS Parent Parent Risk TrendID # Open Date Risk Title Risk Statement L C Affinity Group Owner

1.00 PM Y ERA Schedule and Resources

5 68 14-Sep-11 ERA Schedule and Resources

Given the high cost and schedule uncertainty inherent to technology development projects, there is a possibility that planned funding and schedules will not support meeting ERA goals.

4 3Schedule (3)

Cost (3) Fay CollierTim Warner

3.0.0, 3.1.3, 3.3.4,

PT N ERA Schedule and Resources g

6 15-Jun-11 Shared Personnel & Services at GRC Facilities

Given that the large wind tunnel facilities at GRC share personnel and services, there is a possibility of schedule conflicts arising between ERA and other test programs (e.g., FAP: SUP, FAP: HYP), resulting in potential ERA propulsion test schedule slips.

5 3Schedule (3)

Cost (3) Ken Suder

1.00 PM N ERA Schedule and Resources g

24 15-Jun-11 Continuing Resolutions

Given the expectation for Continuing Resolutions during FY12, there is possibility that ERA technology development activities will be delayed, resulting in schedule slips and loss of productivity

5 3Technical (3)

Cost (3)Schedule (3)

Fay Collier

20

ERA Risk ReportingRank Trend

Risk ID #

Affinity GroupApproach(M,W,A,R)

Risk Title

1 h 11Cost (5)

Schedule (5)Technical (5)

M

Discrete Roughness Elements Laminar Flow Glove Experiment

(DRE) Programmatic Planning and Control

2 h 13Cost (3)

Schedule (5)W

Pultruded Rod Stitched Efficient Unitized Structure (PRSEUS) Programmatic Estimates and

Control

3 g 65Technical (5)Schedule (5)

M Combustor Development and Test

4 g 66 Technical (5) MUHB Geared Turbo Fan Engine

Development Noise Characteristics

5 g 76 Schedule (5) RERA Key Decision Point (KDP)

Schedule

6 5 77 Schedule (5) RFlow Control Experiment for AFC

Rudder

7 g 68Schedule (3)

Cost (3)A ERA Schedule and Resources

8 g 69 Technical (3) M ERA Technical Challenges

9 i 67 Technical (3) R

Insufficient Resources to Mature Vehicles Concepts and Associated

Technologies to Simultaneously meet ERA Goals

10 i 64 Schedule (5) WHybrid Wing Body (HWB)

Community Noise Assessment

Risk Matrix

LIKELIHOOD

5 11,13

4 68, 69

3 . . . . . . . 65,66,76,77

2 . .67

. . . . . . .

1 64

1 2 3 4 5

CONSEQUENCE

Note: The numbers on the risk matrix refer to the Risk ID numbers.

Approach

M - Mitigate

W - Watch

A - Accept

R - Research

Med

High

Low

CriticalityL x C Trend

Decreasing (Improving)

Increasing (Worsening)

Unchanged

New Since Last Period

Project “Parent” Risks

Denotes assessment and number of “Child” risks.

21

ERA Project Phase 1 Portfolio Weighting of RiskTranslating Child to Parent Risks

ERA WBS MatrixTitle

Sub-project Resource

Level Milestone Milestone

02.01.4

Demonstrate low-weight, damage-tolerant stitched composite structural concept on curved panel subjected to combined tension and internal pressure loads.(COLTS Large Scale Pressurized Fuselage Test Complete) (Proposed FY12 APG)Complete

AT 1

Complete Noise Transmission Assessment of PRSEUS panels

Milestone Deliverable KPPKPP Goal APG

Technical Challenge

Technical Maturation

Subsonic Transport System Level Metric

Technical Deliverable and Validation Method Cost Weight

Report development of predictive noise transmission models for like structural concepts

N/A N/A Yes N/A .75

ERA Risk Assessment Matrix

Very Low Low Medium High Very High5 10 16 20 23 254 7 13 18 22 243 4 9 15 19 212 2 6 11 14 171 1 3 5 8 12

1 2 3 4 5

Very Low Low Medium High Very High5 8 12 15 17 194 5 10 14 17 183 3 7 11 14 162 2 5 8 11 131 1 2 4 6 9 1 2 3 4 5

Weighted table, based on WBS task weight. *Values rounded

In this example, if this risk was assessed at a Likelihood of 4 and Consequence of 5 by the task lead. LxC: 4 x 5. The risk would be scored at 24, red, at the ERA project level. By creating the weighting factor to reflect the task’s contribution to ERA goals, the risk would be scored at LxC: 4x3 as the likelihood would not change, only the potential consequence. This allows the task lead to asses the risk based on their understanding of their task and creates a tool for the ERA risk manager a quantifiable and traceable method to accurate assess task risks and communicate at the project level.

ERA Project Phase 2 Portfolio Selection Criteria – Risk Posture

22

RISK

Cost Risk Score (1/3 weighting) Schedule Risk Score (1/3 weighting)Technical Risk Score (1/3

weighting)

WorkforceAvailability

Industry

Component

Procur-ement

Facility Availabil

ity

ITD Comple

xity

Industry Compon

ent

Procure-ment

WorkforceAvailability

Facility Availabi

lity

ITD Comple

xity

Technical

Benefit

ITD Complexity

Industry Compon

ent

Cum Score

(Weightin

g)10% 25% 10% 20% 35% 25% 10% 15% 20% 30% 25% 35% 40% 100%

ITD #1 Low Low Low Low Low High High High High High High High High 4.8

ITD #2 Low Low Low Low Low Low Low Low Low Low Low Low Low 5.0

ITD #3

Medium

High High High High Low Low Low Low Low Low Low Low 2.8

ITD #4 Low Low Low Low Low Low Low Low Low Low Low Low Low 4.6

ITD#5 High High

Medium

Medium

Medium

Low Low Low Low Low Low Low Low 2.5

ITD#6 High

Medium

Low Low Low Low Low Low Low Low Low Low Low 2.7

ITD#7 Low Low Low Low Low

Medium

Medium MediumMediu

mMediu

mMedium

Medium

Medium 5.0

ITD#8 Low Low Low Low Low Low Low Low Low Low Low Low Low 4.4

ITD#9

Medium

Low Low Low Low Low Low Low Low Low Low Low Low 4.2

ITD#10 Low Low

Medium

MediumMediu

mLow Low Low Low Low Low Low Low 3.9

• Example of Risk scoring• Supporting rationale will be developed for each Low, Medium or High score

Low = 5, Medium = 3, High = 1

23

Concluding Remarks

Opening Question: • What is the appropriate risk management construct for an Aeronautics

Technology Development Project responsible for maturing airframe and

propulsion technologies from TRL 3 to 5/6 through Integrated Systems

Research?

Lessons Learned to date:• Tailoring Risk Management Processes to Aeronautics Integrated Systems Research

Technology Development poses different challenges than spaceflight development

• The technology landscape is constantly changing with new technologies coming online promising increased performance. Risk assessment must weigh the promise of new performance against the confidence of what has been done before

• The realization of a risk is not failure, the knowledge gained identifies new foci for integrated systems research to continue the technology’s maturation path to enable technology transition into an aircraft system.

• These considerations do not require us to change the Project’s likelihood and consequence risk criteria but consider the lens through which we view these scales

24

Questions?

• Douglas Brown, ERA Risk [email protected]

757.864.3515

LaRC

• Gaudy Bezos-O’Connor, ERA Deputy Project [email protected]

757-864-5083

LaRC

• Steven Hirshorn, Integrated Systems Research Program Systems Engineering & Integration Manager

[email protected]

202.358.0775

HQ

mailto:[email protected]



25