Embed Size (px)
Citation preview
~;, university o f
Central Florida
M&S at NASA
Martin J. Steele, Ph.D.
November 20, 2009
Overview
• Constellation's Discrete Event Simulation - DES?
- Analysis
• NASA's Modeling & Simulation Standard - Analysis/Results Focused
11/19/2009
1
+
CONSTELLATION
CONSTELLATION'S DISCRETE EVENT SIMULATION
Discrete Event Simulation
• Definition: - Process & System Analysis, through time-based & resource
constrained probabilistic simulation models, providing insight into operational system performance.
• "Competing" types of Analysis - Spreadsheets
- Scheduling Software
- Probabilistic Risk Assessment
11/19/2009
2
Current End-to-End CxDES Process Flow
Prod & Rorurtl Standa!oneOps
lntogOpI
A.,oo' + Ascent Abort
CxDES Architec$ 'e
PAIAriana
PooOps
1\ A
~ ~
Pott-FlightProceaslrlg
"of'"
ISS Depsrture ISSDeorblt
~~aa~~~l~~~ser If running the model with Inputs ptior to run Manifest Data and/or
F;;;~~::::::::::::;:=i:;!:=:!!!::==:;:::;!!::;!!!;;!J!~=1 Order Lead TImes, that
RS Arena ::11~~7:eo:~c:·/:r~he DES Model
_ .. ..,.. c-_·- -- -..:...~. :"- : ".:::J'1X.----
-=;:" I "::=-~. -:' ... " - .
i]AyE~
)
Doscont& umding
~-. P05t-Flight P/'OCCInlng
"of"'"
11/19/2009
3
Inputs: • Production Rates • Process Times • Transport TImes • Event Probabilities • Policies (shifting)
CxDES Analyst
I Prod&Refufb r... __ S\.Ind-'oM0p6 _000 - --
Outputs: • Mission Rate & Distribution
• Cycle Times • Utilizations • Waiting TImes
DES Analysis
Cycle
Understanding System Performance
• Critical Path • Risk to Launch Rate • Margin
Manufacturing through Launch Duration Comparisons
---.... ........... --. 155 OOp;lr'll' ~c ........
ISS Oeor.::'t "\
001(:001 I!.
, I Lt:rod~
'" I
J ~-.
7
POS1·F I'sIIlIProcO$S)i)o~ RQ("rtl Pc, t-Ftyhl. PI;)Wuhll;l
R~"ro
J
8
11/19/2009
4
Manufacturing through Launch Duration Comparisons
765 days Asse~bly I Integration
,---_-.--JA~_~, . _~ ( __ "' A,c<' I \
1_ _ __ .J._
1 . ~ ._ . ~-~ I~-':· .i I Prod& Rofurb r... __ Stand&:oneOp$
lntogOp$ - -- Pod"" ___ J
Time
9
AlIt.( Ares I/Orion shall be able to launch every 45 days & -- Baseline (With Scrubs/Rollbacks) W
Cumulative Probability of Achieving X or More Launches 4 5~Dily Mission Request; Pild Scrubs <1nd Rollb<1cks On;
Integration: 5x3 Shifting 100% r·········-········~·····~~·· .. ········-·····-----.... ---il-- ... --.... 90% +---------.---...~
RO\l1 t-~··~·~·~·~·~~····-·-·-··-·-···~-···_
7(),;.£ +--...... -... ~---... --.-.J,------------.. --.---.-----------.
GO':;' +---.-----I--=.---~ .. --------------, (N +·· .. ········ .. ··················-··· .. ~······· .. ··I ~O% +--...... ~ ... ~ ... - .... ~--.... I-
<Oli. + ................................... .. 1 O~1. j ...... --..... ","--
0% I ---------.------......... . ................. - ......... ~~ .... ------.
3
!'lumber of l~un(hes _ r crcr nt<lgc
.... Frf'(~ . l('rr:V
o
• 22% probability of 5 launches during one year • Average of 4.01 launches per year 10
11/19/2009
5
e. Ares I/Orion shall be able to launch every 45 days & Baseline (With Scrubs/Rollbacks) W
Time Between Launches 35J - 120.00%
30J 100.00%
25J
80.00')(,
~ 20)
~ 60.00%
~ .... 15J
40.~
10J
.-., .. ~, II zO.OO%
-IJlll . IJI~. , I , .~ __ . ~ u.uu"'
• Average is 91.35 days between launches I I
e. Ares I/Orion shall be able to launch every 45 days & Baseline (No Scrubs/Rollbacks) W
Launches/yr:4 4 4 4.3 4.3 4.3 4.6 5 5 .8 5.8 5.2 7.1 8 .1 100.00 l '1.~~ · '1 ,lUg H.J~ ·T··· .....
90..00 · •. - '--l ~".ti ~· ' -C..&2
IO .. 1-.1-+-• . - + I.-!-.I-+
. .. . . ..
14.55 " . ----.
19.10
72.61
" .. t-. l-t-a-t-- +-:-,:-:0, ... :+;,""".0;-' .. " .71-+'_-j_--I -1·- -f--~-"-'-
! n .51
" .. ., ..
--I
t·••··• .~ .. I
I
+t + I
. --.. ,))C---
B.IS@Une Intetrr~Uon SC:he<Iull! X
6,] Integntkln SChedule
Orion tll"I\e av~lIabll! for Int~on
fwd Any Manufadurftll hdUty CapKtty l~aSf'
ARF-Fwd Any CaP«tty Increase
All FHE are rudy to Integate whft\ needed
• Change in Integration Shift Schedules 12
11/19/2009
6
Conclusions
• 2 & 4 Launches per Year possible with Baseline Assumptions
• ~ 90% of Cycle time is in Manufacturing & Assembly
• Dependencies to 4S-day launch-to-Iaunch cycle: - Integration & Pad Shifting Policy - FHE readiness for Integration
• Manufacturing • Assembly • Off-Line Ground Ops
- Aft Skirt quantity (of reusable FHEs)
• i-time 30-day launch-to-Iaunch cycle not possible using current model data
Future Work
• Input Data Refinement - Level 3 Projects Data
• Automate Chart Production
• Refine Analyses
• Logic for minimum launch spacing
• Adjust manufacturing start time based on system behavior (manage ETE Cycle Time)
• Shelf Life of FH Es
• Lunar SRR
13
14
11/19/2009
7
NASA'S MODELING & SIMULATION STANDARD (NASA-STD-7009)
Thoughts to Discuss
• M&S Practices
• Reporting to Decision Makers
• Credibility discussion - V&V,VV&A
• Placarding results
11/19/2009
8
Why a New Standard?
• Why Aren't Software Standards Enough? - Don't cover models developed only in hardware
• With simulations carried out as an exercise using the hardware models
- M&S use is focused towards understanding a system for the purpose of decision making
Why NASA? / Why Now?
• Feb 1,2003
• Resulting Columbia Accident Investigation Board (CAIB) developed set of Recommendations, Observations, & Findings (R-O-Fs) - Directed towards the Space
Shuttle Program - Some were related to Models
& Simulations
11/19/2009
17
18
9
Findings of Shuttle Accident Investigat· Related to Modeling & Simulation
• Operating a model • Model Management outside known - Maintenance limits - Support - Conditions are - Configuration
outside known Control limits
• Data V&V (I & 0) • Model Operator - Model Verified with
- Training Real Data - Experience - Model Data is
• Assumptions Current
Communicated - Sensitivity Analysis
- Also, Abstractions Performed
Basic Ideas
~ Documentation ofM&S Activities (Sections 4.1 - 4.6)
~ Credibility Assessment (Section 4.7 & Appendix B)
~ Reporting to Decision Makers (Section 4.8) - M&S Analysis Results
- A statement on the uncertainty in the results
- Credibility of M&S Results
- Identify • Unfavorable outcomes
• Violation of assumptions
- Unfavorable Use Assessment • Difference Between V&V & Use Assessment
OSE-SIW-076 20
'-->
19
11/19/2009
10
Accreditation Results
Depth of I Depth of I Depth of Development Analysis Su~ort
A. fi-r v- -v-
Development Operations Supporting Evidence
Ver I Val Input I Uncertainty I
Pedigree Quant. Robustness
Technical Review
'-----------------------~-----------------------~
Use Assessment
I Within
FideUty Validated Domain
Depth of Review
Use I History
V&V Foundation
Ol'tllliuutal / 'Vtlic1Atha.
,. I
" :!.xperinutatiol
I /
Comp,*ru:c4 Uo401
Verificntiln
© Sargent, R. G. (c. 1980).
GOICQtu.,J Modol
V3!i.dfltiCil
.U;1.1,:Jil: I.Di
Model
I Mgt
11/19/2009
"
People Qual.
11
Verification & Validation
Verification
• Structure
• Flow • Fidelity
• How: - Comparing to
Conceptual Model Entity (Code) Tracing Primitive Tests (All l 's)
- Min/Max Value Tests
Validation: .... . determining the degree to which a model or a simulation is an accurate representation of the real world .....
Rcru .!
EuvitullllltmL
UK
SiwiJulityof Analogous System mll~ t. h I"; C';1!I~fnlly
c:nmdlit";rt: n
Caution
Sim ilar S)utem & Environment Environment dAta. fidelity
mus t be carefully considered
I R<.1tP.
Best
Data from a p~vioUIS
flight of the same .o;pacecndt in the Sftllle
type of SpAce environ~nt
OK
Sim:l1o.rityof An41og:ms Environment
must be carefully considered
SLm tiar 8imDarity Real Ry~t.~m of Syste m. Syst.p.m
Input:
Input Pedigree Input Form:
• Source - Notional - Subject Matter Expert - Applicability to current
problem • Referent Quality
relative to current problem
- Referent System - Referent Environment
- Authoritative Data
• Quantity of Source Data
• What's the character of your analysis?
- Average
- Uniform
- Triangular ~
- Estimated PDF (from min, mOde~
95%) ) ~
- PDF from adequate real-world data
11/19/2009
12
Accuracy & Uncertainty • Accuracy:
True Value 'Modeled' Value
A ~ I } Uncertainty in
-25
Decrease
Uncertainty:
+-- 'Modeled' Value
Uncertainty in True Value
Low
~ O K
~At lCQstyou know That you have wide
lI nc~rtn i t1 ty
Worst
UO$ure of wide lIT u:t'! r1.n in l,.v
-
High Confidence: of a Narrow uncertainty
OK
Low confidence of a narrow uncertainty
may at least have some rnn.neuvering
Wide Results Narrow
Uncertalnty
• Types • Epistemic • Aleatory
• Sources • 'Size' (Le., how big)
How Confident
Reducible
Subjective Model Form Assumptions Abstractions Incomplete Information
Irreducible (Na tural) Variability
Inherent
Stochastic
Uncertainty
• 2 Types Epistemic
• Reducible • Subjective • Model Form • Lack of Knowledge • Incomplete Information
Aleatory • Variability • IrreduCible • Inherent • Stochastic
• Uncertainty Occurrences - Parameters of the model - Accuracy of the model - Sequence of possible event
• Parametric Uncertainty - Aleatoric - Stochastic Parameters
• Model Form - Epistemic - Model Structure/Selection
• Why M&S Results may not be correct - Variability
Uncertainty - Error
• Methods Representation Aggregation
- Propagation - Interpretation of Results
More Experiments
More System Knowledge
1----+1 Less Epistemic Uncertainty
11/19/2009
13
Robustness
Robustness of Results. Le .. Sensitivity of:
• The Real World System (RWS)
• The M&S
Insensitive To Changes
SenSitive To Changes
Worst Situation Best Situation
M&Sshows a RWS 1.5 robust (Insensitive Robustness not. present to Changes)
in the R\VS & - Valtdation Issue the M&s matches
- M&5 not so useful theRWS
"'~ <?f,<::'
Not a Good Situation OKSfluaUon
M&S Is not robust. but RWS Is sensitive to change RWSls
& the M&S matches - Validation Issue theRWS - Results will be overly
conservative
SenSitive To Changes
RWS Insensitive To Changes
Use History & Management
Use History:
• Similarity of Uses - Analogous Systems - Exact Systems
• Length of Time in Use Just Developed
• Just Updated
Long-Term Successful Use
M&S Management:
• Models & Data under Configuration Control
• Models are Maintained Sustained
11/19/2009
14
People Qualifications & Tech Review
People Qualifications: Technical Review:
• Education • When accomplished
• Training - DUling M&S Development
• Experience - InM&S - With the Modeled (Real
World) System
• Use of Recommended Practices
- During M&S Operations
• Qualifications & Independence of the 'Peer' Review Group: - Self - Internal Organization - External - Non-Expert to Expert
• Level of Formalism - Planning - Documentation
Sample Report Formats
BarChart Radar Plot
V.rtllcatlon
R .. ,,1\a Rabustne ••
nus briefing Is for status only and does not represent complete engineering data analysis
30
11/19/2009
15
Scope of the M&S Standard
• Standard covers the use ofM&S affecting:
n ~c. . . As defined by each Program C ·t· al {. Human Safety } DeCISIons • MISSIon Success
M&S Results Influence
-----;:-::---Sample Risk Matrix
Models / Modeling
Modeling Aspects: • Incidents (events , activities) • Lifecycle (phases) • Functions
Model Dynamics • Social • Physical • Environmental • Economic • Organizational • Infrastructure • Other (e.g., Engineering
Processes
Model Representations: • Conceptual • Mathematical • Dynamic • Programming Paradigms • Analytical Techniques
Interaction Methods: • Live • Virtual • Constructive
Uses / Objective: • Decision Support • Planning • Analysis • Systems Engineering • Training / Gaming • Performance Measure • Component / Module
11/19/2009
32
16
Questions to Ask
• Type of Analysis • Level of Detail • Type ofM&S • Application S /W • Uncertainty • Use History • Config Mgt • V&V Domain/Range • Analysis Domain/Range
Model Types
Behavior Mimicking (S imulations)
Y=X2 Mathematical, Physical,
or Chemical Formula (Algebraic Equations, ode. pde. Physical Formulas &
Chemical Reaction Equations)
2H, +0, ~ 2H,G
Behavioral
Listing & Relating Pieces oflnformatlon
(Databases. ObJectOriented
Hierarchy. Organizational.
a~~
Physical
Visual Form or Representation (Pictures. Graphs)
Physical/ Tangible (Abstract. Scaled)
Versions (Model Cars. Dolls)
t
11/19/2009
33
34
17
Sim Types
Process Analysis
F,(t)~
Computational t! Science & I H(t)
Engineering
Scenario Analysis (Autonomous
Entity Interaction in
an Environment)
(Physics-based,
Process Feasibility
(Visualizing, Form, Fit, Function)
Military View of M&S
pde)<'EM Fa(t)
Sys & S/W Validation
Missing Element Testing
Training
(from an 'Interaction Modes' perspective)
Real System
Live
This is currently not defined, but lcavl:s room fur AutOllOlDOIIII I
Robotic systems operating in a
real environment
Simulated System
Virtual
ColiltrliCt1Ye
-,,-
• This looks at M&S from an 'Interaction Mode' perspective
• Description of categorization from: - McLean, et al. - Taxonomy
paper - S1S0 2008 - Lee Lacey (DRC) - OneSAF
2008 Conference
• Pink box is from conversation with Lee Lacey (DRC)
36
11/19/2009
18
Analysis Methods
System
Experiment with Actual System
Physical Model
Experiment with Model of System
Mental Model
Analytical Model
Law & Kelton (2000), Sim.llarioD Modeling and ~ 3rd ed , McGraw-Hill, Inc.
~ Modified by Steele with added detail
Static QI
Dynamic
M&S Uses:
Analysis
Prediction
Training
Testing
Gaming
Experiencing Visualizing Analyzing
Numerical/Computational (including Simulation)
Deterministic QI Continuous Stochastic QI
/ Probabilistic Discrete
Visualization Sensory Immersion
Simple
ill Complex
Level of Detail
, .
•
'!' 0 ~ . . 0 ,
• 0
8
11/19/2009
19
Network Layered Protocol Approach
Send er
Application
Presentation
Session
Transport
08E-SIW-076
Like the Layered Network Protocol
Model
39
Layered M&S View (Influences in M&S Results)
Receiver
User Input including I
Run
Analyzing Output
M&SV&V t and
Credibility Assessment
Industry Standards { and
Broad Use
08E-SIW-076
Setup I t including Post-Processing
put Data I of Out
MtS Input I M/SOuqmt
: Model/Simulation :
I Application Software I dperating System SOftwo/e
L Com....E.ute~~dw~ J
40
Need for a Clearinghouse for Commercial & Open Source M&S Languages & Application Software
11/19/2009
20
BACKUPS
Martin's Response 'Measured'Value = M&S Result
Comparing Values that have Uncertainty
True Value
Short Definitions
Accuracy - Agreement between a measurement (M&S Result) & the True Value
Uncertainty - A range of values likely to enclose the True Value
Validation - Process of determining the accuracy of aM&S
co 75 2 .., '0 ~ 1:> ~ 25 w
0
·25
Decrease
To know how much agreement there is between a measured & true value, the uncertainty of each must be evaluated.
- 'Measured'Value
+-- Uncertainty in 'Measured'Value
Uncertainty in 42
True Value
11/19/2009
21
Expected Output Range
, , , : . ,
I I I I Va 'dated
Use Assessment
Envelope of VaUd<lt iQn PQI(\1~
: 0 tput Range I I I
, , ,
Prediction Point OutsIde-
V"id"T~IOP'
* Prediction Point
InsidE! Validation E.nvelope
·_- --------- Intended Input Domain
OSE-SIW-076 Note - this is a 2-dimensional example of a potentially multidimensional input domain & multi-dimensional output range
Information Reported to Decision-makers
Section .04 ,1 Supplement : CAS ()penJtional Concepr
_ 1". ... ,.,,*,,,. ' T"'~""'r s.:.,1mtIrIt ' l"' C~iIoIy As ... ......,w
' My AlN . 1S
, ... -
$ec;:tion ".7 Supplement: CA S ~tioMJ Concept
• r"'NI.' .,lntal. ' T"'~y,W~
' T"'C~"' .... _ "'"
This briefing is for status only and does nnt rpnrpc::.pnt
44
Section "'.7 SUpplement: CAS ~r.tion.1 Concept
• rhl o.U" firI'I. J. • T/NUltC. IUJI" IJ' ... '_fll' • ,,.. C......,...",Autum'lIt
ScM,,,,, • • • Any Coil",""
Section ".7 Supplement: CA S {)pemtional Concept
' TNN': ••. ."." • ' TM ""~,. s.:.,~
. r"'C~A ..... ~$r." t .. utJ ' '''"yn" .. ,.
tIL'ill[.l~1 U .. A • • ••• m. nt Nol P,rformed
11/19/2009
22
CAIS Report
"
Development Progression
Mgt Decision Maker
Pilot
Diaz --+ Report
--+ f------------1 --+ f---------I
NASA / t aCE
Direction M&S Literature
>lLU.-aDl' ~"1"~ n"'IX.."lIfKAlI(»;
!<ct!.IUSllln:t!q
Something to say about models:
• Model Map Display from the Mid-Atlantic WX.com (shown on previous page)
IMPORT ANT! This map does *NOT* represent the OFFICIAL FORECAST TRACK! Although the "official track" may be included, this is not a product of the Tropical Prediction Center/The National Hurricane Center.
This map is a graphic representation of computer generated projected tracks. This information is EXPERIMENTAL and subject to extreme fluctuations. It is p ro lliQ@Q for iHfeR+latiooal purposes only. Do not rely o~ information! ~
47
Jeanne, Sept 16, 2004 - Track Prediction . .
-100 ' -95' -90 ' -85 ' -BO' -75' -70' -65 ' -60' -55 ' -50 '
48
11/19/2009
24
