Numerical Methods for Partial Differential

Equations with Random Data

Howard Elman

University of Maryland

Outline

1

I. Problem statement and discretization

II. Solution algorithms

• Example: diffusion equation with random diffusion

coefficient

• Discretization by stochastic Galerkin method

• Discretization by stochastic collocation method

• Multigrid-style methods for various discretizations

• Comparison of solution costs for different discretizations

Forcing function f

Boundary data

Viscosity ν in Navier-Stokes equations

I. Stochastic Differential Equations with Random Data

DNDD

d

nuagu

Rfua

DDDDD

\on 0)( ,on

in )(-

Example: diffusion equation

a = a(x,ω) a random field

For each fixed x, a(x,ω) a random variable

Uncertainty / randomness:

Other possibly uncertain quantities :

Dg

2

0 div grad)grad( 2

ufpuuu

Depictions: Random Data on Unit Square

3

1. Spatial correlation of random field: For

Diffusion Equation with Random Diffusion Coefficient

Random field a(x,ω)

Mean μ(x) = E(a(x,·))

Variance

Covariance function

c(x,y) = E( (a(x,·)- μ(x)) (a(y,·)- μ(y)) )

is finite

:, Dyx

22 )),(()( xaEx

vs. white noise, where c is a δ-function

Din )(- fua

4

210 a2. Coercivity

Problem is well-posed

Assumptions:

Monte-Carlo Simulation

Sample a(x,ω) at all x , solve in usual way

Standard weak formulation: find such that

D

)(1 DEHu

)(),( vvua

),(1

0DEHv

DD

dxvfvdxvuavua )( ,),(

for all

Multiple realizations (samples) of a(x,·)

Multiple realizations of u

Statistical properties of u

Problem: convergence is slow, requires many solves5

= identically distributed uncorrelated random

variables with mean 0 and variance 1

Another Point of View

10 )()()(),(r rrr xaxaxa

Din )(- fua

Covariance function is finite

random field (diffusion coefficient) has Karhunen-Loève expansion:

)(r

)(),()( )( ),())((D

CC dyyayxcxaxaxa

rr xa ),(

mean )),(()()(0 xaExxa

= eigenfunctions/eigenvalues of covariance operator

6

Requires: m large enough so that the fluctuation of a

is well-represented, i.e. is small11 /m

Finite Noise Assumption

~ Principal components analysis

m

r rrr xaxaxa10 )()()(),(

Din )(- fua

Truncated Karhunen-Loève expansion:

7

More precisely: error from truncation is2

1

2

||

||

D

Dm

j j

Choose m to make this small

1. Stochastic Finite Element (Galerkin) Method:

Introduce a weak formulation analogous to finite elements

in space that handles the “stochastic” component of the problem

2. Stochastic Collocation Method:

Devise a special strategy for sampling ξ that converges more

quickly than Monte Carlo simulation; derived from interpolation

8

Various Ways to Use This

m

r rrr xaxaxa10 )()()(),(

Ghanem, Spanos, Babuška, Deb, Oden, Matthies, Keese, Karniadakis,

Xiu, Hesthaven, Tempone, Nobile, Webster, Schwab, Todor, Ernst,

Powell, Furnival, E., Ullmann, Rosseel, Vandewalle

Stochastic Finite Element (Stochastic Galerkin) Method

Probability space (Ω,F, P)

)(2

PL {square integrable functions wrt dP(ω)}

Inner product on : )(2

PL

Use to concoct weak formulation on product space )()( 21

PE LH D

Find such that

)(),( vvua

)()( 21

0 PE LHv Dfor all

)()( 21

PE LHu D

D

dPdxvua )(

Solution u=u(x,ω) is itself a random field

)()()()(, dPwvvwEwv

9

For Computation: Return to Finite Noise Assumption

Truncated Karhunen-Loève expansion

m

r rrr xaxaxa10 )()()())(,(

Stochastic weak formulation uses

)(

)(),()(),(DD

ddxvuxadPdxvuavua

ξ plays the role of a

Cartesian coordinate

Bilinear form entails

integral over image of

random variables ξ Require joint

density function

associated with ξ10

11

Statement of Problem Becomes

DD

ddxfvddxvuxa )()(),(

Find such that

))(( )()( 21

0 PE LHv Dfor all

)()( 21

PE LHu D

Like an ordinary Galerkin (or Petrov-Galerkin) problem on a

(d+m)-dimensional “continuous” space

d = dimension of spatial domain

m = dimension of stochastic space

12

Discretization

DD

ddxvfddxvuxa )()(),(

Finite dimensional spaces:

• spatial discretization:

for example: piecewise linear on triangles

• stochastic discretization:

for example: polynomial chaos = m-variate Hermite

polynomials (orthogonal wrt Gaussian measure)

xN

jjh HS 1

1

0 }{by spanned ),(D

N

llp LT 1

2 }{by spanned ),(

phhphphphp TSvvvua allfor )(),(

xN

j

N

l ljjlhp xuu1 1

)()(

Discrete weak formulation:

13

Basis Functions for Stochastic Space

}|{ )()v()v()( 22 dL

)(2LTp

Underlying space:

)()()()( 2211 MM

Let polynomial of degree j orthogonal wrt)()(

k

k

jq k

Examples: if ρk ~ Gaussian measure Hermite polynomials

ρk ~ uniform distribution Legendre polynomials

Any ρk can be handled computationally (Gautschi)

Rys polynomials

)}()()({ )(

2

)2(

1

)1(

21 m

m

jjj mqqq spanned by

Orthogonality of basis functions sparsity of coefficient matrix

Matrix Equation Au=f

ddxxxff

GG

dxxxxaxA

dxxxxaA

AGAGA

qk

Γ

kq

qlrlqrqllq

kjrjkr

kjjk

rr

)()()(),(][

,][ ,,][

)()()()(][

)()()(][

0

r

00

m

1r00

D

D

D

Properties of A:

• order = Nx x Nξ = (size of spatial basis) x (size of stochastic basis)

• sparsity: inherited from that of { } and { }rG rA14

m

r rrr xaxaxa10 )()()())(,(

15

Dimensions of Discrete Stochastic Space

)(2LTp)}()()({ )(

2

)2(

1

)1(

21 m

m

jjj mqqq spanned by

Full tensor product basis: ,m, ipji 1 ,0

Dimension: mp )1(

“Complete” polynomial basis: pjjj m21

Dimension: ( )m+p

p=

(m+p)!

m! p!

Too large

)(,),(),( 21 N

More

manageable

Order these in a systematic way

16

Example

)(2LTp)}()()({ )(

2

)2(

1

)1(

21 m

m

jjj mqqq spanned by

3)( ,1)( ,)( ,1)( 3

3

2

210 HHHH

Orthogonal (Hermite) polynomials in 1D:

“Complete” polynomial basis: pjjj m21

25

1

3

14

2

13

12

1

)(

3)(

1)(

)(

1)(

2

3

210

1

2

29

2

28

2

2

17

216

3)(

)1()(

)1()(

)1()(

)(

m=2, p=3 ( )m+p

p ( )52= =10

Gives basis set:

17

Example of Sparsity Pattern

Nξ=(m+p)!

m!p!

10!

6!4!=

= 210

For m-variate

polynomials of

total degree p:

using orthogonality of stochastic basis functions

18

Uses of the Computed Solution:

N

j jj

l

M

l lhp

xuxu

dxuuE

1 11

1

)()(

)()()()(

First moment of u (expected value):

Free!

Similarly for second moment / covariance

1. Moments:

)(

)()()()(11 1

xu

xuxuu

l

N

l ll

N

l

N

j ljjlhp

x

19

Uses of the Computed Solution:

)(

)()()()(11 1

xu

xuxuu

l

N

l ll

N

l

N

j ljjlhp

x

)),(( xuP hpE.g.:

2. Cumulative distribution functions

at some point x

Sample ξ

)()(),(1

N

l llhp xuxuEvaluate

PrecomputedRepeat

Not free, but no solves required

Given as above

20

Stochastic Collocation Method

r

m

r rr xaxaxa10 )()(),(

Let ξ be a specified realization (~ Monte Carlo)

Weak formulation:

DD

dxvfdxvuxaxa r

m

r rr ))()((10

Discretize in space in usual way.

Stochastic collocation: choose special set

from considerations of interpolation

)()2()1( ,,,N

Advantage: Spatial systems are decoupled

Given and v(ξ), consider an interpolant

21

Multi-Dimensional Interpolation

,,,,)()2()1( N

where Lagrange interpolating polynomial ,)()(

jk

j

kL

DD

dxvfdxvuxaxa r

m

r rr ))()((10

If solves the discrete (in space) version of

)()(),(1

)(

k

N

k

k

hhp Lxuxu

)(k

hu

),()()())((1

)(vLvIv

N

k

k

k

,)(k

with then the collocated solution is

22

To Compute Statistical Quantities

dLxuxuE k

N

k

k

hhp )()()())((1

)(

1. Moments

Not free but can be precomputed

2. Distribution functions

Obtained by sampling, cheap

)()(),(1

)(

k

N

k

k

hhp LxuxuSolution

1D interpolating polynomial

pk j

k j

0

23

Strategy for Interpolation

),()()())((1

)(vLvIv

N

k

k

k

One choice of )()()()(:}{ 21 21 mkkkkk mLL

Advantage: easy to construct

Disadvantage: “curse of dimensionality,”

dimension = (p+1)m

Detour: Sparse Grids

Given: 1D interpolation rule )()())(( )(

1

)()()( k

j

m

j

k

j

kk yyvyvUk

Multidimensional rule above is induced by fully populated

multidimensional grid .

Derived from (1D) grid },,{ )()(

1

)( k

m

kk

kyyY

)()2()1( mYYY

Alternative: multidimensional sparse grid (Smolyak)

piimp

iii

m

mYYYmpm1

21

1

)()()()(),(H

1|| )( pmY k

k

24

25

Sparse Grid Interpolation

Example of sparse grid

for m=3, p=16

For v of the form interpolating

function takes the form

)()()()())(( 22

)1()(

11

)1()( 22

1

11 vUUvUUvii

pii

ii

m

I

)()()1()(

mm

iivUU mm

),()()()( 2211 mmvvvv

Sparse Grid Interpolation

For sparse grid and a tensor product polynomial of

total degree at most p,

)(v

).())(( vvI

Theorem (Novak, Ritter, Wasilkowski, Wozniakowski)

That is: sparse grid interpolation evaluates the set of complete

m-variate polynomials exactly

Overhead: number of sparse grid points to achieve this

(= # stochastic dof) is larger than for Galerkin

( )m+p

p≈ 2p ( )m+p

pvs.

pjjjqqqv mm

m

jjj m 21

)(

2

)2(

1

)1( ),()()()(21

26

Analysis

Monte-Carlo:

))()(( ))()(()()( hphhhp uEuEuEuEuEuE

, rrc p 12 uEhc )|(| 21

uEhc )|(| 21

))()(( ))()(()()( hshhhs uEuEuEuEuEuE

s1~

(Babuška, Tempone, Zouraris, Nobile, Webster)

Convergence is slow wrt number of samples but

independent of number of random variables m

Stochastic Galerkin and Collocation:

Rule of thumb: the same p gives the same error

(for all versions of SG and collocation)

More dof for collocation than SG

Exponential in polynomial degree p

Constants (c2 , r) depend on m

27

Recapitulating

Monte-Carlo methods:

Many samples needed for statistical quantities

Many systems to solve

Systems are independent

Statistical quantities are free (once data is accumulated)

Stochastic Galerkin methods:

One large system to solve

Statistical quantities are free or (relatively) cheap

Stochastic collocation methods:

Systems are independent

Fewer systems than Monte Carlo

More degrees of freedom than Galerkin

Statistical quantities are (relatively) cheap

s

r

r

hhs xus

uE1

)( )(1)(With s realizations:

Convergence is slow but independent of m

Similar convergence

behavior

Faster than MC

Depends on m

28

II. Computing with the Stochastic Galerkin and

Collocation Methods

For both: compute a discrete solution, a random field

)()()()(),(11 1 l

N

l l

N

l

N

j ljjlhp LxuLxuxux

),(xuhp

Stochastic Galerkin:

N

l ll

N

l

N

j ljjlhp xuxuxux

11 1)()()()(),(

Stochastic Collocation:

Postprocess to get statistics

29

Computational Issues

Stochastic Galerkin: Solve one large system of order Nx x Nξ

m+p

pNξ=( )

Frequently cited as a problem for

this methodology

Stochastic Collocation: Solve Nξ “ordinary” algebraic systems

(of order Nx), one for each sparse grid point

Here: )()( 2~ Galerkinpncollocatio NN

Some savings possible 30

spatial discretization parameter h, , )()( hhrr AAAA

31

Multigrid Solution of Matrix Equation I (E. & Furnival)

, , )2()2( hhrr AAAA spatial discretization parameter 2h

Solving Au=f

Ω

rqllqr

D

kjrjkr

rr

dG

dxxxxaA

AGAGA

)()()(][

,)()()(][

r

m

1r00

Develop MG algorithm for spatial component of the problem

32

Multigrid Algorithm (Two-grid)

)(1)()(1)( )( hhhh fQuAQIu

for i=0,1,…

for j=1:k k smoothing steps

end

Restriction

Solve Coarse grid correction

Prolongation

end

)( )()()()2( hhhh uAfr R)2()2()2( hhh rcA

)2()()( hhh cuu P

Prolongation and restriction:

TT PRRI

PI

,

,

P R

P induced by natural inclusion in spatial domain

Let Q = smoothing operator,)( NQA h

0)( , )(])([ )(2

)(1)(increasesk

A

khh kyykyAQIA h

Establish approximation property

and smoothing property

yyCyAAhA

hh ]))([2

1)2(1)(

)(RP(

33

Convergence Analysis: Use “Standard” Approach

)()(1)(1)2(1)()1( ])([)]))[( ikhhhhi eAQIAAAe RP(Error propagation matrix:

)(

)()(

||||)(

||])([||

||])([)]))(||||||

)(

2

)()(1)(

)()(1)(1)2(1)()1(

h

hh

A

i

ikhh

A

ikhhhh

A

i

ekC

eAQIAC

eAQIAAAe RP(Analysis is:

34

Approximation Property

2

)(

)(

2

)(

2

2

2

2/1

222

)2()(1)2(1)(

2

)),((

]))([

2

22

)()(

)(

yC

fCh

uDhCuDCh

uuuu

uuuuauu

uuyAA

L

LL

ahah

hhhhahh

A

hh

A

hh

hh

D

DD

RP(

Approximability

Property of mass

matrix

Regularity

“Standard” MG analysis for deterministic problem:

35

For Approximation Property in Stochastic Case

Introduce semi-discrete space pTH )(1

0 D

Weak formulation

p

ppppp

u

THvvvua

Solution

)( allfor )(),( 1

0 D

)()(

2

22 LLpahpp uDChuu

D

Similarly for other steps used for deterministic analysis

Discrete stochastic

space

ahpaph

aphhpA

hh

uuuu

uuyAAh

2

,2

1)2(1)(

]))([)(

RP(

Approximation (in 2D):

Established using best approximation property of

and interpolanthpu

),(),(~jpjp xuxu

36

• Establishes convergence of multigrid with rate independent of

spatial discretization size h

Comments

• Coarse grid operator: size O(Nξ )

• No dependence on stochastic parameters m, p

• Applies to any basis of stochastic space

derives from basis of multivariate polynomials of total

degree p, orthogonal wrt probability measure ρ(ξ)dξrG

,m

1r00 rrr GaGaG

Maximum eigenvalue η = max root of orthogonal polynomial,

bounded for bounded measure

CG iteration is an option

),()()(0 x11x111x1x11 m

1r0

m

1r0 rrrr aaGaa

37

Iteration Counts / Normal Distribution

Polynomial

degree

# terms (m) in KL-expansion

m=1 m=2 m=3 m=4

p=1 8 8 8 8

p=2 8 8 8 8

p=3 9 9 9 9

p=4 9 10 10 10

h=1/16

Polynomial

degree

m=1 m=2 m=3 m=4

p=1 7 7 8 8

p=2 8 8 8 8

p=3 8 8 9 9

p=4 9 9 9 9

h=1/32

Ω

rqllqr

kjrjkr

rr

dG

dxxxxaA

AGAGA

)()()(][

,)()()(][

r

m

1r00

D

Multigrid Solution of Matrix Equation II

Solving Au=f

Preconditioner for use with CG: 00 AGQ

)()()(~

0

00

IG

dxxxxaA kj

D

(Kruger, Pellissetti,

Ghanem)

Deterministic diffusion,

from mean

38

Analysis (Powell & E.)

Theorem : For μ constant, the Rayleigh quotient satisfies

1),(

),(1

Qww

Aww

||||)()(1 r

m

r r apc

11

Consequence: dictates convergence of PCG

m

r rrr xaxaxa10 )()()(),(Recall

m

1r00 rr AGAGA

00 AGQ

39

Sketch of Proof ||||)()(1 r

m

r r apc

40

c(p) bounded by largest root of scalar orthogonal polynomial

m

1r00 rr AGAGA

DdxxxxaA rrr )()()(~),(

Ddxxxarr )()(||||

),(||||)/( 0Aarr

In spatial domain:

From stochastic component: as above

Multigrid Variant of this Idea

Replace action of with multigrid preconditioner1

0A

MGMG AGQ ,00(Le Maitre, et al.)

Analysis:),(

),(

),(

),(

),(

),(

wQw

Qww

Qww

Aww

wQw

Aww

MGMG Spectral equivalence

of MG approximation

to diffusion operator],[ 21

1

2

)1(

)1(

41

Experiment

fua )(-

Starting with a with specified covariance and small σ (=.01):

# Samples s

Max SFEM 100 1000 10,000 40,000

Mean .06311 .06361 .06330 .06313 .06313

Variance 2.360(-5) 2.161(-5) 2.407(-5) 2.258(-5) 2.316(-5)

Compare Monte-Carlo simulation with SFEM, for

N.B.: No negative samples of diffusion obtained in MC

Solve one system

of order 210x225 Solve s systems of size 225 42

Comparison of Galerkin and Collocation

Recall, for stochastic collocation

43

)()(),(1

)(

k

N

k

k

hhp LxuxuDiscrete solution

Obtained by solving

DD

dxvfdxvuxaxa r

m

r rr ))()((10

For set of samples situated in a sparse grid}{)(k

Advantage of this approach: simpler (decoupled) systems

Disadvantage: larger stochastic space for comparable accuracy

larger by factor approximately p2

Dimensions of Stochastic Space

m

(#KL)

p Galerkin Collocation

Sparse

Collocation

Tensor

4 1

2

3

4

5

15

35

70

9

41

137

401

16

81

256

625

10 1

2

3

4

11

66

286

1001

21

221

1582

8,801

1024

59,049

1,048,576

9,765,625

30 1

2

3

4

31

496

5456

46,376

61

1861

37,941

582,801

1.07(9)

2.06(14)

1.15(18)

9.31(20)

~ size of coarse grid space

for MG / Version 1# systems for collocation

MG / Version II

Experiment

• Solve the stochastic diffusion equation by both methods

• Compare the accuracy achieved for different parameter sets¹

• For parameter choices giving comparable accuracy, compare

solution costs

• Spatial discretization fixed (32x32 finite difference grid)

Solution algorithm for both discretizations:

Preconditioned conjugate gradient with

mean-based preconditioning, using AMG for the

approximate diffusion solve

¹Estimated using a high-degree (p=10) Galerkin solution.

(E., Miller, Phipps, Tuminaro)

46

Experimental Results

46

polynomial degree for SG

“level” for collocation

produces comparable errors

p=1

p=2

p=3

p=4

p=5

Accuracy:

for fixed m=4: similar p=

p=3

p=5p=4

p=2

p=1

p=4

p=5

p=6

p=3p=2

p=1

Degrees of freedom

Performance:

M=3

ErrorError

47

Experimental Results: Performance

4747

p m=5 m=10 m=15 m=5 m=10 m=15

1 .058 .147 .32- .069 .163 .286

2 .269 1.20 3.80 .532 2.13 5.08

3 1.20 13.14 51.45 2.41 16.99 57.98

4 3.50 53.79 168.11 8.31 102.60 493.04

5 6.51 117.73 24.56 515.75

CPU times for larger m = #KL terms:

Galerkin Collocation

Performed on a serial machine with C code and

CG/AMG code from Trilinos

Observation: Galerkin faster, more so as number of

stochastic variables (KL terms) grows

More General Problems

),(

min),( xceaxa

Nonlinear

48

For the problem discussed, based on a KL expansion, has

a linear dependence on the stochastic variable ξ

Other models have nonlinear dependence. For example

m

r rrr xa

xaxc

1

0

)(

)(),(

In particular: coercivity is guaranteed with this choice

For Gaussian c, called a log-normal distribution

More General Problems

49

)()( )(),(10

M

r rrr xaxaxa

For stochastic Galerkin, need a finite term expansion for a

Note: not ξr

M

1r00 rr AGAGA

matrix

][ jirijrG Less sparse

More importantly: # terms M will be larger

perhaps as large as 2Nξ

mvp will be more expensive

comes from for each

sparse grid point

In Contrast

50

Collocation is less dependent on this expansion

D

dxvuxak

),()()(kA

)(k

Many matrices to assemble, but mvp is not a difficulty

Concluding Remarks

51

• Exciting new developments models of PDEs with uncertain

coefficients

• Replace pure simulation (Monte Carlo) with finite-dimensional

models that simulate sampling at potentially lower cost

• Two techniques, the stochastic Galerkin method and the

stochastic collocation method, were presented, each with some

advantages

• Solution algorithms are available for both methods, and work

continues in this direction