On the Application of Estimation Theory to Complex System ......• The ability to incorporate tools of estimation theory for fusing sources of information • A mathematical framework

On the Application of Estimation Theory to Complex System Design Under Uncertainty

Douglas Allaire, Karen Willcox, and John Deyst Department of Aeronautics and Astronautics

Massachusetts Institute of Technology

SIAM Conference on Computational Science and Engineering 2011 March 1, 2011

Reno, NV

ThisworkwaspartiallysupportedbyDARPAunderAFRLcontractFA8650‐10‐C‐7083,bytheUSAirForceunderSTTRAwardFA9550‐10‐C‐0175,andbytheSingaporeUniversityofTechnologyandDesignInternationalDesignCenter.

Bayesian Estimation View of System Design, Development, and Deployment

•  Design processrecast as a discoveryprocedure

•  Models andexperiments usedin concert to conducta series of observationsof key parameters

•  Bayesian characterizationof key design parametersrepresents the level of uncertainty (confidence) in each parameter at any point during the design process

2

Tracking Uncertainty with an Evolving System Estimate Leads to …

•  A useful quantitative definition of system complexity

•  The applicability of mathematical methods to identify sources of uncertainty for resource allocation purposes

•  Systematic guidance to managing models of varying fidelities (and possibly experiments)

•  The ability to incorporate tools of estimation theory for fusing sources of information

•  A mathematical framework for determining what quantities should be tracked throughout the design/development of a complex system

•  In combination: a mathematical foundation on which to manage risk throughout the design process

3

A Critical Need for Better Methods to Manage Uncertainty and Risk in Complex Systems

4

Development times and costs of aerospace systems have reached unsustainable levels – and are getting worse.

Source:www.boeing.com.

The Boeing 787 program has incurred significant cost and schedule overruns due to unexpected integration issues.

Source: DARPA. Figure appeared in Aviation Week & Space Tech., Nov. 1-8 2011.

System Complexity

•  There are many possible definitions of complexity.

•  We define complexity as the potential of a system to exhibit unexpected behavior in the quantities of interest (whether detrimental or not). –  Captures the notion of the emergent behavior and nonlinear

interaction phenomena. –  A system that does not necessarily exhibit complex behavior can

still be defined as complex if there is potential for such behavior.

•  Claim: Information entropy provides an appropriate means for considering complexity in the context of managing system design and development decisions.

5

Information Entropy

6

Normal

Uniform

Triangular

Consider a random variable Y with probability mass function p(y)The entropy of Y is defined as :

H (Y ) = − pi∑ (yi )log p(yi ),

where y1, y2 ,… are the values of y such that p(y) ≠ 0Consider a random variable X with probability density function fX (x)Differential entropy of X is defined as :

h(X) = − fXX∫ (x)log fX (x)dx

Examples :

h(N (µ,σ 2 )) = 12

ln(2πeσ 2 )

h(U[a,b]) = ln(b − a)

h(T (a,b,c)) = 12+ ln b − a

2⎛⎝⎜

⎞⎠⎟

System Complexity Metric

7

In general, for a continuous random variable Y , h(Y ) ∈(−∞,∞).

Definition : (System Complexity) Let Q be the quantities of interest for a given system. Then CQ = exp(h(Q)) is the complexity of the system.

Note: CQ ∈(0,∞).

y ϒ

Dens

ity

Dens

ity

System Complexity Metric Interpretation

8

Consider a uniformly distributed random variable Y with support set Y, then CY = V(Y), where V(Y) is the volume of Y.Definition : (Entropy equivalent random variables) Random variables X and Y are said to be entropy equivalent if h(X) = h(Y ).

Example : Let Y ~ N (0,σ 2 ) and ϒ ~U[0, (2πeσ 2 )]

h(Y ) = 12

ln(2πeσ 2 ) = h(ϒ).

If Y is a continuous random variable, then CY is the volume of the support of anentropy equivalent uniformly distributed random variable. That is CY = exp(h(Y )) = V(support(ϒ)) Vϒ (Y ), where h(Y ) = h(ϒ), and ϒ is uniformly distributed.

System Complexity Estimation

9

Factors Quantities of interest System Entropy

1.  Identify uncertainty information for system factors 2.  Estimate maximum entropy distributions for system factors 3.  Sample from each system factor distribution 4.  Compute estimates of the quantities of interest 5.  Estimate the joint distribution of the quantities of interest 6.  Compute the entropy of the joint distribution of the

quantities of interest


10





11





12





13





14





15




System Complexity Metric: Adder Example

We will consider 3 cases and compute the complexity of each •  Case 1. We know the component is an adder and we care about V0

•  Case 2. We know nothing about the component and we care about V0 and V1 •  Case 3. We know the component is an adder and we care about V0 and V1

16

R

R

V1 V2

V0 v105

0.2

v205

0.2

v005

0.4

R ≡ ResistanceV1,V2 ,V0 ≡ Voltages

V0 =V1 +V2

2Assume: V1 ~U[0,5] V2 ~U[0,5]Thus: V0 ~ T (0,5,2.5)

Assume: The component is an adder V1 ~U[0,5] V2 ~U[0,5]Thus, V0 ~ T (0,5,2.5)Define: Quantities of Interest ≡Q = [V0 ]Then System Complexity ≡ CQ = 4.1

System Complexity Metric: Adder Example (Cont.)

17

Case 1. We know the component is an adder and we care about V0

R

R

V1 V2

V0 Component

18

Case 2. We know nothing about the component and we care about V0 and V1

R

R

V1 V2

V0 Component

V1

Assume: V1,V2 ,V0 are independent V1 ~U[0,5] V2 ~U[0,5] V0 ~ T (0,5,2.5)Define: Quantities of Interest ≡Q= [V0 ,V1]T

Then System Complexity ≡ CQ = 20.1


19

R

R

V1 V2

Component

Case 3. We know the component is an adder and we care about V0 and V1

Assume: The component is an adder V1 ~U[0,5] V2 ~U[0,5] V0 ~ T (0,5,2.5)Define: Quantities of Interest ≡Q= [V0 ,V1]T

Then System Complexity ≡ CQ = 12.2

V0

V1


20

Summary: Case 1. We know the component is an adder and we care about V0

System Complexity = 4.1 Case 2. We know nothing about the component and we care about V0 and V1 System Complexity = 20.1 Case 3. We know the component is an adder and we care about V0 and V1

System Complexity = 12.2 Comments:

•  Choice of quantities of interest impact complexity (e.g. restricting ourselves to use of V0 only from the adder has the smallest system complexity).

•  The more we know about a component, the lower the entropy

h(X,Y ) ≤ h(X) + h(Y )









21

Uncertainty Source Identification

22

Cathode Anode Space Thruster for

Orbital Repositioning

Satellite Source:blog.cunysustainablecities.org.

Complex System Analysis Complex System Design

Source:rst.gsfc.nasa.gov

Global climate change

Ice sheet dynamics

Source:www.boeing.com.

Variance-Based Global Sensitivity Analysis!

!"#

f (x) = f0+ fi

i

! (xi ) + fij (xi , x j )i< j

! +!+ f12!n (x1, x2 ,…, xn )

High Dimensional Model Representation3!

ANOVA Constraint!

fi1 ,…,is0

1

! (xi1 ,…, xis )dxp = 0 for p = i1,…,is

Variance Decomposition!

V = Vi

i

! + Vij

i< j

! +!+V12!n

Consider a square integrable function

y = f (x1,…, xn ),

where xz ![0,1] "z

Sensitivity Indices!

Si1 ,…,is

=Vi1 ,…,is

V

$# $%#$!#

$%!#Quantity of Interest variance

Factor 1

Factor 2

Factor 1, Factor 2 Interaction

Complexity-Based Global Sensitivity Analysis

24

Consider a model Y = f (X1,…,Xn ), where Y is the quantity of interestWe have h(Y ) − h(Y | Xi ) Entropy removed once Xi is known h(Y ) − [h(Y ) − h(Y | Xi )] Entropy remaining once Xi is known exp(h(Y )) = volϒ (Y ) Initial system complexity (CY ) exp(h(Y ) − [h(Y ) − h(Y | Xi )]) Final system complexity = exp(h(Y | Xi )) exp(h(Y )) − exp(h(Y | Xi )) Change in system complexity

exp(h(Y )) − exp(h(Y | Xi ))exp(h(Y ))

Normalized by the initial complexity

= 1− exp(h(Y | Xi ) − h(Y )) Mutual Information: = 1 - exp(-I(Xi |Y )) I(Xi;Y ) = h(Y ) − h(Y | Xi )Definition : Complexity Global Sensitivity Indices ηi = 1− exp(−I(Xi |Y )).ηi ∈[0,1].


25

Consider a model Y = f (X1,…,Xn ), where Y is the quantity of interestWe have h(Y ) − h(Y | Xi ) Entropy removed once Xi is known h(Y ) − [h(Y ) − h(Y | Xi )] Entropy remaining once Xi is known exp(h(Y )) = volϒ (Y ) Initial system complexity (CY ) exp(h(Y ) − [h(Y ) − h(Y | Xi )]) Final system complexity = exp(h(Y | Xi )) exp(h(Y )) − exp(h(Y | Xi )) Change in system complexity





26

Consider a model Y = f (X1,…,Xn ), where Y is the quantity of interestWe have h(Y ) − h(Y | Xi ) Entropy removed once Xi is known h(Y ) − [h(Y ) − h(Y | Xi )] Entropy remaining once Xi is known exp(h(Y )) = Vϒ (Y ) Initial system complexity (CY ) exp(h(Y ) − [h(Y ) − h(Y | Xi )]) Final system complexity = exp(h(Y | Xi )) exp(h(Y )) − exp(h(Y | Xi )) Change in system complexity





27






28






29






30






31




= 1− exp(h(Y | Xi ) − h(Y )) Mutual Information: = 1 - exp(-I(Xi |Y )) I(Xi;Y ) = h(Y ) − h(Y | Xi )Definition : (Complexity Global Sensitivity Indices) ηi = 1− exp(−I(Xi |Y )).Note: ηi ∈[0,1].








32

Stochastic Process Model of System Design and Development •  Characterize system development as Bayesian estimation

•  Over the course of the development process, the estimation process reduces uncertainty about the system state to a sufficient level that the actual system is realized

•  Estimation theory givesa way to synthesizemultifidelity simulationand experimental datato evolve estimate of thesystem state (Kalman filter,particle filter)

33

AdaptedfromMarch2010.

System Development as a Bayesian Estimation Process

34

Multidisciplinary Development Cycle

System Development as a Bayesian Estimation Process

35

Multidisciplinary Development Cycle

Extended Kalman Filter

Maximum Entropy Distributions1

36

Consider a probability space (Ω,F ,P) and a random variable X :Ω→ withprobability density function fX (x)Differential entropy of X is defined as :

h(X) = − fXX∫ (x)log fX (x)dx

Optimization problem for maximum entropy distributions :Given all probability densities f , maximize h( f ) over all f such that 1. fX (x) ≥ 0, with equality outside the support, X, of fX (x),

2. fXX∫ (x)dx = 1,

3. fXX∫ (x)ri (x)dx = α i , for 1 ≤ i ≤ m,

Solution :

fX (x) = exp(λ0 −1+ λii=1

m

∑ ri (x)), x ∈X

Maximum Entropy Distributions1

37

The differential entropy h( f ) is a concave function over a convex set.Thus, we form the functional

J ( f ) = − f ln f∫ + λ0 f∫ - 1⎡⎣

⎤⎦ + λi

i=1

m

∑ f∫ ri - α i⎡⎣

⎤⎦

and differentiate

∂J∂f (x)

= −ln f (x) −1+ λ0 + λii=1

m

∑ ri (x).

Setting this equal to zero, we arrive at

f (x) = exp(λ0 −1+ λii=1

m

∑ ri (x)), x ∈X

where λ0 ,λ1,…,λm are chose so that f satisfies the constraints.

Entropy Estimation2

38

�

Let (Γ,d) be a metric space with a measure µ, and let ν be a probability measure with a measurable

density ρ =dνdµ

, ν(Γ) = ρΓ∫ dµ = 1.

Let {Γ j} j∈J be a partition of Γ into measureable subsets,γ j = µ(Γ j ), 0 < µ(Γ j ) < ∞, j ∈J, where J is a finite or countable set.

Set ρ j =λ j

γ j

, λ j = ρΓ j∫ dµ, λ j

j∈J∑ = 1

and define a new density ρ :Γ→ such that ρ Γ j= ρ j , j ∈J.

Define coarse grained entropy as s = − ρΓ∫ logρdµ. Then

s = − ρ jΓ j∫

j∈J∑ logρ jdµ = − γ j

j∈J∑ ρ jlogρ j = − λ jlogλ j

j∈J∑ + λ j

j∈J∑ logγ j

= − λ jj∈J∑ logλ j + logγ , if γ j = γ ∀j ∈J

Variance-Based Global Sensitivity Analysis: Monte Carlo Estimates4

39

Theorem 1.Let x = {x j} jj∈N , u = {x j} j∈M and v = {x j} j∈N \M , whereM = {k1,…,km :1 ≤ k1 < < km ≤ n} and N = {1,…,n}.

Then Vu = f∫ (x) f (u, ′v )dxd ′v − f02 .

Proof.

f∫ (x) f (u, ′v )dxd ′v = d∫ u f∫ (u,v)dv f∫ (u, ′v )d ′v = d∫ u f∫ (u,v)dv⎡⎣

⎤⎦

2.

We have f∫ (u,v)dv = f0 + fi1 ,…,is(i1 <<is )∈M∑

s=1

m

∑ (xi1 ,…, xis ).

Squaring and integrating over du yields

f∫ (x) f (u, ′v )dxd ′v = f02 + Vi1 ,…,is

(i1 ,…,is )∈M∑

s=1

m

∑ = f02 +Vu .

References 1.  T. Cover and J. Thomas, Elements of Information Theory, John Wiley & Sons,

Inc. 1991, pp. 266-268. 2.  V. Kozlov and D. Treshchev, “Fine-grained and Coarse-grained Entropy in

Problems of Statistical Mechanics”, Theoretical and Mathematical Physics 151(1), 2007, pp. 539-555.

3.  H. Rabitz and O.F. Alis, “General Foundations of High-Dimensional Model Representations”, Journal of Mathematical Chemistry 25 (1999), pp. 197-233.

4.  I.M Sobol’, “Global Sensitivity Indices for Nonlinear Mathematical Models and Their Monte Carlo Estimates”, Mathematics and Computers in Simulation 55 (2001), pp. 271-280.

5.  E. Linfoot, “An Informational Measure of Correlation”, Information and Control 1 (1957), pp. 85-89.

6.  J. Deyst, “The Application of Estimation Theory to Managing Risk in Product Developments”, Proceedings of the 21st Digital Avionics Systems Conference, Volume 1, 2002, pp. 4A3-1-4A3-12.

7.  T. Homma and A. Saltelli, “Importance Measures in Global Sensitivity Analysis of Nonlinear Models”, Reliability Engineering and System Safety 52 (1996), pp. 1-17.

8.  March, A. and Willcox, K., “A Provably Convergent Multifidelity Optimization Algorithm not Requiring High-Fidelity Derivatives,” AIAA-2010-2912, presented at 3rd MDO Specialist Conference, Orlando, FL, April 12-15, 2010.

40

Documents

On the Application of Estimation Theory to Complex System ......• The ability to incorporate tools of estimation theory for fusing sources of information • A mathematical framework