Moser September2011

Quantifying Uncertainty in Simulations of ComplexEngineered Systems

Robert D. Moser

Center for Predictive Engineering & Computational Science (PECOS)Institute for Computational Engineering and Sciences (ICES)

The University of Texas at Austin

September 20, 2011

Special thanks to: Todd Oliver, Onkar Sahni, Gabriel Terejanu

PECOS

Acknowledgment: This material is based on work supported by the Department of Energy [National NuclearSecurity Administration] under Award Number [DE-FC52-08NA28615].

R. D. Moser Quantifying Uncertainty in Simulations 1 / 43

Motivation

PECOS: a DoE PSAAP CenterDevelop Validation & UQ Methodologies for Reentry Vehicles Multi-physics, multi-scale physical modeling challenges Numerous uncertain parameters Models are not always reliable (e.g. turbulence)


Motivation

Prediction

Prediction is very difficult, especially if its about the future. N. Bohr

Predicting the behavior of the physical world is central to both scienceand engineering

Advances in computer simulation has led to prediction of morecomplicated physical phenomena

The complexity of recent simulations . . .I Makes reliability difficult to assessI Increases the danger of drawing false conclusions from inaccurate

predictions


Motivation

Imperfect Paths to Knowledge and Predictive Simulation

ofTHE UNIVERSE

REALITIESPHYSICAL

VALIDATION VERIFICATION

ObservationalErrors

OBSERVATIONS MATHEMATICALTHEORY /

MODELS

COMPUTATIONALMODELS

ErrorsModeling Discretization

Errors

KNOWLEDGE

DECISION

Predictive Simulation: the treatment of model and data uncertainties and theirpropagation through a computational model to produce predictions of quantitiesof interest with quantified uncertainty.


Motivation

Quantities of Interest

Simulations have a purpose: to inform a decision-making process

Quantities are predicted to inform the decision These are the Quantities of Interest (QoIs) Models are not (evaluated as) scientific theories

I Involve approximations, empiricisms, guesses . . . (modelingassumptions)

I Generally embedded in an accepted theoretical framework (e.g.conservation of mass, momentum & energy)

Acceptance of a model is conditional on: its purpose the QoIs to be predicted the required accuracy


Motivation

What are Predictions?

PredictionPurpose of predictive simulation is to predict QoIs for whichmeasurements are not available (otherwise predictions not needed)

Measurements may be unavailable because:

instruments unavailable scenarios of interest inaccessible system not yet built ethical or legal restrictions its the future

How can we have confidence in the predictions?


Motivation

Sir Karl Popper

The Principle of Falsification

A hypothesis can be accepted asa legitimate scientific theory

if it can possibly be refuted byobservational evidence

A theory can never be validated; it canonly be invalidated by (contradictory)experimental evidence.

Corroboration of a theory (survival ofmany attempts to falsify) does notmean a theory is likely to be true.


Motivation

Willard V. Quine

The falsification of an individualproposition by observations is notpossible, as such observations rely onnumerous auxiliary hypotheses.

Only the complete theory (including allauxiliary hypotheses) can be falsified byan experiment.


Motivation

Rev. Thomas Bayes

P (|D) = P (D|)P ()P (D)

Genesis of Bayesian interpretation ofprobability & Bayesian statistics

Theories have to be judged in terms of

their probabilities in light of the evidence.


Principles of Validation & Uncertainty Quantification

Posing a Validation ProcessValidation of physical models, uncertainty models & their calibration

Two types of validation question:I Are their unanticipated phenomena affecting the systemI Do modeling assumptions yield acceptable prediction of the QoIs

Challenge the model with validation data (generally not of QoIs)I Use observations that challenge modeling assumptionsI Use observations that are informative for the QoIs

Are discrepancies between model & data significant?I Can they be explained by plausible errors due to modeling

assumptions?I If so, what is their impact on the prediction of the QoIs?

Validation expectations are model dependent Interpolation models: simple fit to data Physics-based models: formulated from theory extrapolatable



Uncertainty

Need to Treat Uncertainty in these Processes

Mathematical representation of uncertainty (Bayesian probability) Uncertainty models Probabilistic calibration & validation processes (Bayesian inference)

Modeling Uncertainty Uncertainty in data

I instrument error & noiseI inadequacy of instrument models (a la Quine)

Uncertainty due to model inadequacyI Represent errors introduced by modeling assumptions.I Impact of these errors within the accepted theoretical framework



Stochastic Extension of Physical Models

Physics Mathematical representation of physical phenomena of interest At macroscale, usually deterministic (e.g., RANS)

Experimental UncertaintyModel for uncertainty introduced by imperfections in observations used toset model parameters (calibrate)

Model UncertaintyModel for uncertainty introduced by imperfections in physical model

Prior InformationAny relevant information not encoded in above models



Model Likelihood

p(D|) =

p(D|Dtrue, ) Experimental uncertainty

p(Dtrue|) Prediction model

dDtrue

p(Dtrue|) =p(Dtrue|Dphys, )

Model uncertainty

p(Dphys|) Physical model

dDphys

Physics + model uncertainty = Prediction model Prediction model + experimental uncertainty = Likelihood Different/further decomposition possible depending on available

information

Models coupled with prior form a stochastic model class



UQ Using Stochastic Model Classes

Processes Single Model Class, M

I Calibration: p(|D) p() p(D|)I Prediction: p(q|D) = p(q|,D) p(|D) dI Experimental design

Multiple Model Classes,M = {M1, . . . ,MN}I Calibration, prediction, and experimental design with each model classI Model comparison/selection: P (Mi|D,M) P (Mi|M) p(D|Mi)I Prediction averaging: p(q|D,M) = i p(q|D,Mi)P (Mi|D,M)

Software used at PECOS DAKOTA: Forward propagation QUESO: Calibration, model comparison

I Metropolis-Hastings, DRAM, Adaptive Multi-Level Sampling



Information Theoretic InterpretationRearranging Bayes formula:

P (d|M1) = P (|M1) P (d|,M1)P (|d,M1) = P (d|,M1)

/P (|d,M1)P (|M1)

Log and integrating:

ln[P (d|M1)] P (|d,M1) d = 1

= . . .

Then:

ln[P (d|M1)] log evidence

= E [ln[P (d|,M1)]] how well the modelclass fits the data

E[lnP (|d,M1)P (|M1)

]

how much the modelclass learns with the data

(expected information gain,EIG)



Summary: Four Stage Bayesian Framework

Stochastic Model DevelopmentGenerate extension of physical model to enable probabilistic analysis

Closure parameters viewed as random variables Stochastic representations of model and experimental errors

CalibrationBayesian update for parameters: p(|D) p()L(;D)

PredictionForward propagation of uncertainty using stochastic model

Model ComparisonBayesian update for plausibility: P (Mj |D,M) P (Mj |M)E(Mj ;D)


Data Reduction Modeling


Why is it needed? Very likely that quantities we wish to measure are not directly

measurable in an experiment

Have to infer the values from other measurements using amathematical model

Estimate/recover uncertainties in legacy experimental data

Impact on Validation and UQ All mathematical models must be validated Must incorporate uncertainty of both the measurements and the data

reduction model into the final uncertainty quantification of the data




Traditional calibration schematic:

ValidationProcess

Inversion Process

Model

Data




Incorporation of data reduction model:

Validation

Calibration

Model

Validation

Calibration

Data Reduction ModelInstrument

DATA



DRM Example: Surface/Wall Catalysis

Motivation Surface/wall catalysis plays a critical role in surface heat flux on a

re-entry vehicle

Reported estimates of surface reaction efficiency vary by few ordersof magnitude1 (often supercatalytic wall is assumed2)

1 Zhang et. al. AIAA-2009-42512 Wright et. al. AIAA-2004-2455



Experimental Setup1

1 Zhang et. al. AIAA-2009-4251



Plug Flow Reactor Model (1D) PFRM

d(vCN )/dx = N vthN CN /deff 2kNNC2NCN2mC = pidstMC

LsRs(x)dx

Two recombination mechanisms:

Recombination at tube surface: N = NTnN Recombination in gas-phase: kNN = ANN e(E

aNN

/(RT ))

Gas-surface mechanism:

Reaction flux: Rs(x) = CN vthN CN (x)/4R. D. Moser Quantifying Uncertainty in Simulations 22 / 43


Synthetic Data : Joint PDFs of Parameters

d(vCN )/dx = NTnN vthN CN /deff2ANN e(E

aNN

/(RT ))C2NCN2

mC = pidstMCCN (vthN/4)

Ls0

CN (x)dx



Observed Data (35 runs) : Posterior PDFs of Parameters

1 0 10

2

4

6

log10(CN/CN,0)1 0 1

2

1

0

1

2

1 0 11

0.5

0

0.5

1

2 0 20

1

2

3

log10(N/N,0)2 0 2

1

0.5

0

0.5

1

1 0 10

0.2

0.4

0.6

0.8

nNnN,0

d(vCN )/dx = NTnN vthN CN /deff2ANN e(E

aNN

/(RT ))C2NCN2

mC = pidstMCCN (vthN/4)

Ls0

CN (x)dx

Multiple model formulation: physics model & a stochastic extension for modelinadequacy (multiplicative error on m)

Physics model w/o inadequacy model has null plausibility One or more modeling assumptions invalid



A Look Back at Modeling Assumptions

Assumptions:

no radial gradients in species concentration: CN

(x)

bulk flow velocity: v(x) effective diameter: d

eff

concentration at sample surface is same as bulk: CN,ws

(x)

All these assumptions consequence of 1-D plug flow approximationneed at least 2-D (axisymmetric) model


Model Inadequacy Modeling

Model Inadequacy Models

Stochastic model of errors in physical modelI Model form uncertaintiesI Incomplete dependenciesI Poor functional forms

Important for at least two reasons:I Invalidate physics model in Bayesian multi-model formulationI Uncertainty in predictions due to model inadequacy

For prediction, extrapolate uncertainty modelI Uncertainty of unreliable modeled quantity, not model outputI Uncertainty model formulated consistent with physical characteristics

of modeled quantity



Example: RANS Turbulence ModelsMotivation Majority of engineering simulations of turbulent flows use

Reynolds-averaged Navier-Stokes (RANS) models

RANS models for RS well-known to be imperfect and unreliable RS models embedded in reliable mean momentum equation Uncertainty due to uncertain parameters

I Model parameters are not constants of nature Uncertainty due to model inadequacy

I Closure models are not physical laws

Must quantify effects of these uncertainties on model predictions

Approach Formulate stochastic models to represent uncertainty Use Bayesian probabilistic approach to calibrate and compare models



Stochastic Model Overview

Models Four competing RANS turbulence (physics) models

I Baldwin-Lomax; Spalart-Allmaras; Chien, low Re k-; Durbins v2-f Five competing model uncertainty representations

I Denial model (no model inadequacy)I Three velocity-based with different spatial correlation assumptionsI One Reynolds stress-based

Twenty total stochastic models

Calibration Data and Prediction QoI Fully-developed, incompressible channel flow Calibrate using DNS data for Re 1000, 2000 (Re u/) Predict centerline velocity at Re = 5000



Physical Modeling ApproachRANS Decompose flow into mean and fluctuating parts: u = u+ u. Average the Navier-Stokes equations. For channel flow,

dd

(1

Re

du+

d+ +

)= 1,

where u+ = u/u , + = uv/u2 , u2 = dudy , = y/.

Model Reynolds stress using eddy viscosity t dudy Make up or choose a turbulence model for t (Baldwin-Lomax,

Spalart-Allmaras, k-, k-, ...)

Key PointModel inadequacy introduced by closure modeli.e., combination of eddyviscosity assumption and model for t



Stochastic Models: Basic IdeasGoalCreate model that produces distribution over mean velocity fields

Incorporate information from RANS solution Model the fact that RANS solution represents incomplete knowledge

Simple Example

u(y; , ) = u(y; ) + (y;) u is the RANS mean velocity is a random field representing uncertainty due to RANS infidelity u is stochastic prediction of true mean velocity

Issues Where to introduce uncertainty model representing model inadequacy Details of that model (e.g., distribution for , dependence on scenario)



Velocity-Based Stochastic Models

Multiplicative Gaussian Error

u+(; , ) = (1 + (;))u+(; )where is a zero-mean Gaussian random field.

Covariance Structures Independent: cov((), ()) = 2( ) Correlated (homogeneous): cov((), ()) = 2 exp

( 12 (

)2`2

) Correlated (inhomogeneous):cov((), ()) = 2

(2`()`()`2()+`2()

)1/2exp

( ()2

`2()+`2()

)where

`() =

ìn for < inìn +

òutìnoutin ( in) for in out

òut for > out



Inhomogeneous Model Details

`() =

ìn for < inìn +

òutìnoutin ( in) for in out

òut for > out

All length scales non-dimensionalized by channel height Inner lengths scale with viscous length, not channel height Rewrite inner variables using viscous scales:

ìn = `+in/Re in =

+in/Re

Length scales (`+in, òut), blend points (+in, out), and variance

2 arecalibration parameters



Covariance Models

Homogeneous

0 0.2 0.4 0.6 0.8 10.97

0.98

0.99

1

1.01

1.02

1.03

Inhomogeneous

0 0.2 0.4 0.6 0.8 10.97

0.98

0.99

1

1.01

1.02

1.03

Inhomogeneous covariance enables better representation of two-layerstructure of channel flow



Reynolds Stress-Based Stochastic ModelsMotivation Structure of the RANS equations is not uncertain Only the closure (i.e., Reynolds stress tensor field) is uncertain Formulate to be more generally applicable

Additive Model uv+(;,) = T+(;) (;) where T+ obtained by solving

RANS+turbulence model

Find u by forward propagation through mean momentum

dd

(1

Re

du+d

+ uv+)

= 1,

chosen to be zero-mean Gaussian random field with

cov((), ()) = kin(, ) + kout(, ),R. D. Moser Quantifying Uncertainty in Simulations 34 / 43


Reynolds Stress-Based Stochastic Models

Reynolds Stress

0 0.2 0.4 0.6 0.8 10.1

0.08

0.06

0.04

0.02

0

0.02

0.04

0.06

0.08

0.1

Velocity

0 0.2 0.4 0.6 0.8 13

2

1

0

1

2

3

u

Large Reynolds stress uncertainty in inner layer



Results Overview

Calibration:I Joint posterior PDFs for model parameters for each model class

Model comparison:I Posterior plausibility for each stochastic model classI Examine joint and conditional plausibilities

QoI Prediction:I Compare predictions (PDF for QoI) of each model class



Sample Parameter Posterior PDFs

Spalart-Allmaras

0.5 1 1.50

2

4

6

p()

0.5 1 1.50.5

1

1.5

cv1

0.5 1 1.50

1

2

3

4

5

cv1

p(cv1

)

Chien k-

0.5 1 1.5 2 2.50

1

2

3

k

p(k)

0.5 1 1.5 2 2.50.5

1

1.5

2

2.5

k

0.5 1 1.5 2 2.50

1

2

3

p()

Parameter joint posterior PDFs computed using Adaptive Multi-LevelAlgorithm implemented in QUESO Library



Joint Model Plausibility

Baldwin Spalart Chien DurbinDenial 0 0 0 0

Independent 0 0 0 0Homogeneous 0 0 0 0

Inhomogeneous 0 0 0.995 3.24 103Reynolds Stress 0 1.36 103 0 0

Chien k- coupled with inhomogeneous velocity uncertainty modelstrongly preferred



Uncertainty Model Plausibility

Conditioned on turbulence model, which uncertainty model is preferred?

Uncertainty Model Baldwin Spalart Chien DurbinDenial 0 0 0 0

Independent 0 0 0 0Homogeneous 0 0 0 0

Inhomogeneous 1 6.69 103 1 0.9998Reynolds Stress 0 0.993 0 1.86 105

Observations Data prefers the inhomogeneous correlation structure for all models Makes sense given the two-layer structure of the mean velocity profile



Turbulence Model Plausibility

Conditioned on uncertainty model, which turbulence model is preferred?

Turb Model Indep Homog Inhomog Rey StressBaldwin 1 0.779 0 0Spalart 0 0 1.01 105 0.99995Chien 0 0 0.996 1.01 105Durbin 0 0.221 3.33 103 4.11 105

ObservationPreferred turbulence model depends on uncertainty model



QoI Predictions

Uncertainty Model Comparison

24 24.5 25 25.5 26 26.5 27 27.5 280

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

q = +(1)

p(q)

INDSEVLSEARSM

Turbulence Model Comparison

24 24.5 25 25.5 26 26.5 27 27.5 280

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

q = +(1)

p(q)

BLSAChienv2f

Observations Different stochastic model extensions lead to significantly different

uncertainty predictions

With same stochastic extension, turbulence models similar for this QoIR. D. Moser Quantifying Uncertainty in Simulations 41 / 43

Challenges

Some Important Challenges

Uncertainty modelingI Data & model inadequacy modelsI Correlation structure

ValidationI Posing appropriate validation questionsI Validating physics & uncertainty modelsI Validating for use in prediction

AlgorithmsI Usual problems: curse of dimensionality, expensive physics modelsI With complex inadequacy models, likelihood evaluation is a

high-dimensional probability integral


Challenges

Thank you!

Questions?


MotivationPrinciples of Validation & Uncertainty QuantificationData Reduction ModelingModel Inadequacy ModelingChallenges

Documents

Moser September2011