~;, university o f

Central Florida

M&S at NASA

Martin J. Steele, Ph.D.

November 20, 2009

Overview

• Constellation's Discrete Event Simulation - DES?

- Analysis

• NASA's Modeling & Simulation Standard - Analysis/Results Focused

11/19/2009

1

https://ntrs.nasa.gov/search.jsp?R=20110008329 2018-06-24T10:08:36+00:00Z

+

CONSTELLATION

CONSTELLATION'S DISCRETE EVENT SIMULATION

Discrete Event Simulation

• Definition: - Process & System Analysis, through time-based & resource

constrained probabilistic simulation models, providing insight into operational system performance.

• "Competing" types of Analysis - Spreadsheets

- Scheduling Software

- Probabilistic Risk Assessment

11/19/2009

2

Current End-to-End CxDES Process Flow

Prod & Rorurtl Standa!oneOps

lntogOpI

A.,oo' + Ascent Abort

CxDES Architec$ 'e

PAIAriana

PooOps

1\ A

~ ~

Pott-FlightProceaslrlg

"of'"

ISS Depsrture ISSDeorblt

~~aa~~~l~~~ser If running the model with Inputs ptior to run Manifest Data and/or

F;;;~~::::::::::::;:=i:;!:=:!!!::==:;:::;!!::;!!!;;!J!~=1 Order Lead TImes, that

RS Arena ::11~~7:eo:~c:·/:r~he DES Model

_ .. ..,.. c-_·- -- -..:...~. :"- : ".:::J'1X.----

-=;:" I "::=-~. -:' ... " - .

i]AyE~

)

Doscont& umding

~-. P05t-Flight P/'OCCInlng

"of"'"

11/19/2009

3

Inputs: • Production Rates • Process Times • Transport TImes • Event Probabilities • Policies (shifting)

CxDES Analyst

I Prod&Refufb r... __ S\.Ind-'oM0p6 _000 - --

Outputs: • Mission Rate & Distribution

• Cycle Times • Utilizations • Waiting TImes

DES Analysis

Cycle

Understanding System Performance

• Critical Path • Risk to Launch Rate • Margin

Manufacturing through Launch Duration Comparisons

---.... ........... --. 155 OOp;lr'll' ~c ........

ISS Oeor.::'t "\

001(:001 I!.

, I Lt:rod~

'" I

J ~-.

7

POS1·F I'sIIlIProcO$S)i)o~ RQ("rtl Pc, t-Ftyhl. PI;)Wuhll;l

R~"ro

J

8

11/19/2009

4

Manufacturing through Launch Duration Comparisons

765 days Asse~bly I Integration

,---_-.--JA~_~, . _~ ( __ "' A,c<' I \

1_ _ __ .J._

1 . ~ ._ . ~-~ I~-':· .i I Prod& Rofurb r... __ Stand&:oneOp$

lntogOp$ - -- Pod"" ___ J

Time

9

AlIt.( Ares I/Orion shall be able to launch every 45 days & -- Baseline (With Scrubs/Rollbacks) W

Cumulative Probability of Achieving X or More Launches 4 5~Dily Mission Request; Pild Scrubs <1nd Rollb<1cks On;

Integration: 5x3 Shifting 100% r·········-········~·····~~·· .. ········-·····-----.... ---il-- ... --.... 90% +---------.---...~

RO\l1 t-~··~·~·~·~·~~····-·-·-··-·-···~-···_

7(),;.£ +--...... -... ~---... --.-.J,------------.. --.---.-----------.

GO':;' +---.-----I--=.---~ .. --------------, (N +·· .. ········ .. ··················-··· .. ~······· .. ··I ~O% +--...... ~ ... ~ ... - .... ~--.... I-

<Oli. + ................................... .. 1 O~1. j ...... --..... ","--

0% I ---------.------......... . ................. - ......... ~~ .... ------.

3

!'lumber of l~un(hes _ r crcr nt<lgc

.... Frf'(~ . l('rr:V

o

• 22% probability of 5 launches during one year • Average of 4.01 launches per year 10

11/19/2009

5

e. Ares I/Orion shall be able to launch every 45 days & Baseline (With Scrubs/Rollbacks) W

Time Between Launches 35J - 120.00%

30J 100.00%

25J

80.00')(,

~ 20)

~ 60.00%

~ .... 15J

40.~

10J

.-., .. ~, II zO.OO%

-IJlll . IJI~. , I , .~ __ . ~ u.uu"'

• Average is 91.35 days between launches I I

e. Ares I/Orion shall be able to launch every 45 days & Baseline (No Scrubs/Rollbacks) W

Launches/yr:4 4 4 4.3 4.3 4.3 4.6 5 5 .8 5.8 5.2 7.1 8 .1 100.00 l '1.~~ · '1 ,lUg H.J~ ·T··· .....

90..00 · •. - '--l ~".ti ~· ' -C..&2

IO .. 1-.1-+-• . - + I.-!-.I-+

. .. . . ..

14.55 " . ----.

19.10

72.61

" .. t-. l-t-a-t-- +-:-,:-:0, ... :+;,""".0;-' .. " .71-+'_-j_--I -1·- -f--~-"-'-

! n .51

" .. ., ..

--I

t·••··• .~ .. I

I

+­t + I

. --.. ,))C---

B.IS@Une Intetrr~Uon SC:he<Iull! X

6,] Integntkln SChedule

Orion tll"I\e av~lIabll! for Int~on

fwd Any Manufadurftll hdUty CapKtty l~aSf'

ARF-Fwd Any CaP«tty Increase

All FHE are rudy to Integate whft\ needed

• Change in Integration Shift Schedules 12

11/19/2009

6

Conclusions

• 2 & 4 Launches per Year possible with Baseline Assumptions

• ~ 90% of Cycle time is in Manufacturing & Assembly

• Dependencies to 4S-day launch-to-Iaunch cycle: - Integration & Pad Shifting Policy - FHE readiness for Integration

• Manufacturing • Assembly • Off-Line Ground Ops

- Aft Skirt quantity (of reusable FHEs)

• i-time 30-day launch-to-Iaunch cycle not possible using current model data

Future Work

• Input Data Refinement - Level 3 Projects Data

• Automate Chart Production

• Refine Analyses

• Logic for minimum launch spacing

• Adjust manufacturing start time based on system behavior (manage ETE Cycle Time)

• Shelf Life of FH Es

• Lunar SRR

13

14

11/19/2009

7

NASA'S MODELING & SIMULATION STANDARD (NASA-STD-7009)

Thoughts to Discuss

• M&S Practices

• Reporting to Decision Makers

• Credibility discussion - V&V,VV&A

• Placarding results

11/19/2009

8

Why a New Standard?

• Why Aren't Software Standards Enough? - Don't cover models developed only in hardware

• With simulations carried out as an exercise using the hardware models

- M&S use is focused towards understanding a system for the purpose of decision making

Why NASA? / Why Now?

• Feb 1,2003

• Resulting Columbia Accident Investigation Board (CAIB) developed set of Recommendations, Observations, & Findings (R-O-Fs) - Directed towards the Space

Shuttle Program - Some were related to Models

& Simulations

11/19/2009

17

18

9

Findings of Shuttle Accident Investigat· Related to Modeling & Simulation

• Operating a model • Model Management outside known - Maintenance limits - Support - Conditions are - Configuration

outside known Control limits

• Data V&V (I & 0) • Model Operator - Model Verified with

- Training Real Data - Experience - Model Data is

• Assumptions Current

Communicated - Sensitivity Analysis

- Also, Abstractions Performed

Basic Ideas

~ Documentation ofM&S Activities (Sections 4.1 - 4.6)

~ Credibility Assessment (Section 4.7 & Appendix B)

~ Reporting to Decision Makers (Section 4.8) - M&S Analysis Results

- A statement on the uncertainty in the results

- Credibility of M&S Results

- Identify • Unfavorable outcomes

• Violation of assumptions

- Unfavorable Use Assessment • Difference Between V&V & Use Assessment

OSE-SIW-076 20

'-->

19

11/19/2009

10

Accreditation Results

Depth of I Depth of I Depth of Development Analysis Su~ort

A. fi-r v- -v-

Development Operations Supporting Evidence

Ver I Val Input I Uncertainty I

Pedigree Quant. Robustness

Technical Review

'-----------------------~-----------------------~

Use Assessment

I Within

FideUty Validated Domain

Depth of Review

Use I History

V&V Foundation

Ol'tllliuutal / 'Vtlic1Atha.

,. I

" :!.xperinutatiol

I /

Comp,*ru:c4 Uo401

Verificntiln

© Sargent, R. G. (c. 1980).

GOICQtu.,J Modol

V3!i.dfltiCil

.U;1.1,:Jil: I.Di

Model

I Mgt

11/19/2009

"

People Qual.

11

Verification & Validation

Verification

• Structure

• Flow • Fidelity

• How: - Comparing to

Conceptual Model Entity (Code) Tracing Primitive Tests (All l 's)

- Min/Max Value Tests

Validation: .... . determining the degree to which a model or a simulation is an accurate representation of the real world .....

Rcru .!

EuvitullllltmL

UK

SiwiJulityof Analogous System mll~ t. h I"; C';1!I~fnlly

c:nmdlit";rt: n

Caution

Sim ilar S)utem & Environment Environment dAta. fidelity

mus t be carefully considered

I R<.1tP.

Best

Data from a p~vioUIS

flight of the same .o;pacecndt in the Sftllle

type of SpAce environ~nt

OK

Sim:l1o.rityof An41og:ms Environment

must be carefully considered

SLm tiar 8imDarity Real Ry~t.~m of Syste m. Syst.p.m

Input:

Input Pedigree Input Form:

• Source - Notional - Subject Matter Expert - Applicability to current

problem • Referent Quality

relative to current problem

- Referent System - Referent Environment

- Authoritative Data

• Quantity of Source Data

• What's the character of your analysis?

- Average

- Uniform

- Triangular ~

- Estimated PDF (from min, mOde~

95%) ) ~

- PDF from adequate real-world data

11/19/2009

12

Accuracy & Uncertainty • Accuracy:

True Value 'Modeled' Value

A ~ I } Uncertainty in

-25

Decrease

Uncertainty:

+-- 'Modeled' Value

Uncertainty in True Value

Low

~ O K

~At lCQstyou know That you have wide

lI nc~rtn i t1 ty

Worst

UO$ure of wide lIT u:t'! r1.n in l,.v

-

High Confidence: of a Narrow uncertainty

OK

Low confidence of a narrow uncertainty

may at least have some rnn.neuvering

Wide Results Narrow

Uncertalnty

• Types • Epistemic • Aleatory

• Sources • 'Size' (Le., how big)

How Confident

Reducible

Subjective Model Form Assumptions Abstractions Incomplete Information

Irreducible (Na tural) Variability

Inherent

Stochastic

Uncertainty

• 2 Types Epistemic

• Reducible • Subjective • Model Form • Lack of Knowledge • Incomplete Information

Aleatory • Variability • IrreduCible • Inherent • Stochastic

• Uncertainty Occurrences - Parameters of the model - Accuracy of the model - Sequence of possible event

• Parametric Uncertainty - Aleatoric - Stochastic Parameters

• Model Form - Epistemic - Model Structure/Selection

• Why M&S Results may not be correct - Variability

Uncertainty - Error

• Methods Representation Aggregation

- Propagation - Interpretation of Results

More Experiments

More System Knowledge

1----+1 Less Epistemic Uncertainty

11/19/2009

13

Robustness

Robustness of Results. Le .. Sensitivity of:

• The Real World System (RWS)

• The M&S

Insensitive To Changes

SenSitive To Changes

Worst Situation Best Situation

M&Sshows a RWS 1.5 robust (Insensitive Robustness not. present to Changes)

in the R\VS & - Valtdation Issue the M&s matches

- M&5 not so useful theRWS

"'~ <?f,<::'

Not a Good Situation OKSfluaUon

M&S Is not robust. but RWS Is sensitive to change RWSls

& the M&S matches - Validation Issue theRWS - Results will be overly

conservative

SenSitive To Changes

RWS Insensitive To Changes

Use History & Management

Use History:

• Similarity of Uses - Analogous Systems - Exact Systems

• Length of Time in Use Just Developed

• Just Updated

Long-Term Successful Use

M&S Management:

• Models & Data under Configuration Control

• Models are Maintained Sustained

11/19/2009

14

People Qualifications & Tech Review

People Qualifications: Technical Review:

• Education • When accomplished

• Training - DUling M&S Development

• Experience - InM&S - With the Modeled (Real

World) System

• Use of Recommended Practices

- During M&S Operations

• Qualifications & Independence of the 'Peer' Review Group: - Self - Internal Organization - External - Non-Expert to Expert

• Level of Formalism - Planning - Documentation

Sample Report Formats

BarChart Radar Plot

V.rtllcatlon

R .. ,,1\a Rabustne ••

nus briefing Is for status only and does not represent complete engineering data analysis

30

11/19/2009

15

Scope of the M&S Standard

• Standard covers the use ofM&S affecting:

n ~c. . . As defined by each Program C ·t· al {. Human Safety } DeCISIons • MISSIon Success

M&S Results Influence

-----;:-::---Sample Risk Matrix

Models / Modeling

Modeling Aspects: • Incidents (events , activities) • Lifecycle (phases) • Functions

Model Dynamics • Social • Physical • Environmental • Economic • Organizational • Infrastructure • Other (e.g., Engineering

Processes

Model Representations: • Conceptual • Mathematical • Dynamic • Programming Paradigms • Analytical Techniques

Interaction Methods: • Live • Virtual • Constructive

Uses / Objective: • Decision Support • Planning • Analysis • Systems Engineering • Training / Gaming • Performance Measure • Component / Module

11/19/2009

32

16

Questions to Ask

• Type of Analysis • Level of Detail • Type ofM&S • Application S /W • Uncertainty • Use History • Config Mgt • V&V Domain/Range • Analysis Domain/Range

Model Types

Behavior Mimicking (S imulations)

Y=X2 Mathematical, Physical,

or Chemical Formula (Algebraic Equations, ode. pde. Physical Formulas &

Chemical Reaction Equations)

2H, +0, ~ 2H,G

Behavioral

Listing & Relating Pieces oflnformatlon

(Databases. ObJect­Oriented

Hierarchy. Organizational.

a~~

Physical

Visual Form or Representation (Pictures. Graphs)

Physical/ Tangible (Abstract. Scaled)

Versions (Model Cars. Dolls)

t

11/19/2009

33

34

17

Sim Types

Process Analysis

F,(t)~

Computational t! Science & I H(t)

Engineering

Scenario Analysis (Autonomous

Entity Interaction in

an Environment)

(Physics-based,

Process Feasibility

(Visualizing, Form, Fit, Function)

Military View of M&S

pde)<'EM Fa(t)

Sys & S/W Validation

Missing Element Testing

Training

(from an 'Interaction Modes' perspective)

Real System

Live

This is currently not defined, but lcavl:s room fur AutOllOlDOIIII I

Robotic systems operating in a

real environment

Simulated System

Virtual

ColiltrliCt1Ye

-,,-

• This looks at M&S from an 'Interaction Mode' perspective

• Description of categorization from: - McLean, et al. - Taxonomy

paper - S1S0 2008 - Lee Lacey (DRC) - OneSAF

2008 Conference

• Pink box is from conversation with Lee Lacey (DRC)

36

11/19/2009

18

Analysis Methods

System

Experiment with Actual System

Physical Model

Experiment with Model of System

Mental Model

Analytical Model

Law & Kelton (2000), Sim.llarioD Modeling and ~ 3rd ed , McGraw-Hill, Inc.

~ Modified by Steele with added detail

Static QI

Dynamic

M&S Uses:

Analysis

Prediction

Training

Testing

Gaming

Experiencing Visualizing Analyzing

Numerical/Computational (including Simulation)

Deterministic QI Continuous Stochastic QI

/ Probabilistic Discrete

Visualization Sensory Immersion

Simple

ill Complex

Level of Detail

, .

•

'!' 0 ~ . . 0 ,

• 0

8

11/19/2009

19

Network Layered Protocol Approach

Send er

Application

Presentation

Session

Transport

08E-SIW-076

Like the Layered Network Protocol

Model

39

Layered M&S View (Influences in M&S Results)

Receiver

User Input including I

Run

Analyzing Output

M&SV&V t and

Credibility Assessment

Industry Standards { and

Broad Use

08E-SIW-076

Setup I t including Post-Processing

put Data I of Out

MtS Input I M/SOuqmt

: Model/Simulation :

I Application Software I dperating System SOftwo/e

L Com....E.ute~~dw~ J

40

Need for a Clearinghouse for Commercial & Open Source M&S Languages & Application Software

11/19/2009

20

BACKUPS

Martin's Response 'Measured'Value = M&S Result

Comparing Values that have Uncertainty

True Value

Short Definitions

Accuracy - Agreement between a measurement (M&S Result) & the True Value

Uncertainty - A range of values likely to enclose the True Value

Validation - Process of determining the accuracy of aM&S

co 75 2 .., '0 ~ 1:> ~ 25 w

0

·25

Decrease

To know how much agreement there is between a measured & true value, the uncertainty of each must be evaluated.

- 'Measured'Value

+-- Uncertainty in 'Measured'Value

Uncertainty in 42

True Value

11/19/2009

21

Expected Output Range

, , , : . ,

I I I I Va 'dated

Use Assessment

Envelope of VaUd<lt iQn PQI(\1~

: 0 tput Range I I I

, , ,

Prediction Point OutsIde-

V"id"T~IOP'

* Prediction Point

InsidE! Validation E.nvelope

·_- --------- Intended Input Domain

OSE-SIW-076 Note - this is a 2-dimensional example of a potentially multi­dimensional input domain & multi-dimensional output range

Information Reported to Decision-makers

Section .04 ,1 Supplement : CAS ()penJtional Concepr

_ 1". ... ,.,,*,,,. ' T"'~""'r s.:.,1mtIrIt ' l"' C~iIoIy As ... ......,w

' My AlN . 1S

, ... -

$ec;:tion ".7 Supplement: CA S ~tioMJ Concept

• r"'NI.' .,lntal. ' T"'~y,W~

' T"'C~"' .... _ "'"

This briefing is for status only and does nnt rpnrpc::.pnt

44

Section "'.7 SUpplement: CAS ~r.tion.1 Concept

• rhl o.U" firI'I. J. • T/NUltC. IUJI" IJ' ... '_fll' • ,,.. C......,...",Autum'lIt

ScM,,,,, • • • Any Coil",""

Section ".7 Supplement: CA S {)pemtional Concept

' TNN': ••. ."." • ' TM ""~,. s.:.,~

. r"'C~A ..... ~$r." t .. utJ ' '''"yn" .. ,.

tIL'ill[.l~1 U .. A • • ••• m. nt Nol P,rformed

11/19/2009

22

CAIS Report

"

Development Progression

Mgt Decision Maker

Pilot

Diaz --+ Report

--+ f------------1 --+ f---------I

NASA / t aCE

Direction M&S Literature

>lLU.-aDl' ~"1"~ n"'IX.."lIfKAlI(»;

!<ct!.IUSllln:t!q![~!rm"t

Interim Nov 06

Final Submitted Nov 07

2 Credibility Scales 1 New Credibility Scale

~/ External Efforts

NASA-wide Fonnal Review

Something to say about models:

Hurricane Ivan Track Prediction Models

45

46

11/19/2009

23

Something to say about models:

• Model Map Display from the Mid-Atlantic WX.com (shown on previous page)

IMPORT ANT! This map does *NOT* represent the OFFICIAL FORECAST TRACK! Although the "official track" may be included, this is not a product of the Tropical Prediction Center/The National Hurricane Center.

This map is a graphic representation of computer generated projected tracks. This information is EXPERIMENTAL and subject to extreme fluctuations. It is p ro lliQ@Q for iHfeR+latiooal purposes only. Do not rely o~ information! ~

47

Jeanne, Sept 16, 2004 - Track Prediction . .

-100 ' -95' -90 ' -85 ' -BO' -75' -70' -65 ' -60' -55 ' -50 '

48

11/19/2009

24