fm May 18, 2010

Numerical Methods for StochasticComputations

fm May 18, 2010

fm May 18, 2010

Numerical Methods for StochasticComputations

A Spectral Method Approach

Dongbin Xiu

P R I N C E T O N U N I V E R S I T Y P R E S S

P R I N C E T O N A N D O X F O R D

fm May 18, 2010

Copyright c 2010 by Princeton University PressPublished by Princeton University Press, 41 William Street,Princeton, New Jersey 08540

In the United Kingdom: Princeton University Press, 6 Oxford Street,Woodstock, Oxfordshire OX20 1TW

press.princeton.edu

All Rights Reserved

Library of Congress Cataloging-in-Publication DataXiu, Dongbin, 1971Numerical methods for stochastic computations : a spectral methodapproach / Dongbin Xiu.

p. cm.Includes bibliographical references and index.ISBN 978-0-691-14212-8 (cloth : alk. paper)1. Stochastic differential equationsNumerical solutions.2. Stochastic processes. 3. Spectral theory (Mathematics).4. Approximation theory. 5. Probabilities. I. Title.QA274.23.X58 2010519.2dc22 2010014244

British Library Cataloging-in-Publication Data is available

This book has been composed in Times

Printed on acid-free paper. Typeset by S R Nova Pvt Ltd, Bangalore, India

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

fm May 18, 2010

To Yvette, our parents, and Isaac.

fm May 18, 2010

fm May 18, 2010

Contents

Preface xi

Chapter 1 Introduction 1

1.1 Stochastic Modeling and Uncertainty Quantification 11.1.1 Burgers Equation: An Illustrative Example 11.1.2 Overview of Techniques 31.1.3 Burgers Equation Revisited 4

1.2 Scope and Audience 51.3 A Short Review of the Literature 6

Chapter 2 Basic Concepts of Probability Theory 9

2.1 Random Variables 92.2 Probability and Distribution 10

2.2.1 Discrete Distribution 112.2.2 Continuous Distribution 122.2.3 Expectations and Moments 132.2.4 Moment-Generating Function 142.2.5 Random Number Generation 15

2.3 Random Vectors 162.4 Dependence and Conditional Expectation 182.5 Stochastic Processes 202.6 Modes of Convergence 222.7 Central Limit Theorem 23

Chapter 3 Survey of Orthogonal Polynomials and Approximation Theory 25

3.1 Orthogonal Polynomials 253.1.1 Orthogonality Relations 253.1.2 Three-Term Recurrence Relation 263.1.3 Hypergeometric Series and the Askey Scheme 273.1.4 Examples of Orthogonal Polynomials 28

3.2 Fundamental Results of Polynomial Approximation 303.3 Polynomial Projection 31

3.3.1 Orthogonal Projection 313.3.2 Spectral Convergence 333.3.3 Gibbs Phenomenon 35

fm May 18, 2010

viii CONTENTS

3.4 Polynomial Interpolation 363.4.1 Existence 373.4.2 Interpolation Error 38

3.5 Zeros of Orthogonal Polynomials and Quadrature 393.6 Discrete Projection 41

Chapter 4 Formulation of Stochastic Systems 44

4.1 Input Parameterization: Random Parameters 444.1.1 Gaussian Parameters 454.1.2 Non-Gaussian Parameters 46

4.2 Input Parameterization: Random Processes and DimensionReduction 474.2.1 Karhunen-Loeve Expansion 474.2.2 Gaussian Processes 504.2.3 Non-Gaussian Processes 50

4.3 Formulation of Stochastic Systems 514.4 Traditional Numerical Methods 52

4.4.1 Monte Carlo Sampling 534.4.2 Moment Equation Approach 544.4.3 Perturbation Method 55

Chapter 5 Generalized Polynomial Chaos 57

5.1 Definition in Single Random Variables 575.1.1 Strong Approximation 585.1.2 Weak Approximation 60

5.2 Definition in Multiple Random Variables 645.3 Statistics 67

Chapter 6 Stochastic Galerkin Method 68

6.1 General Procedure 686.2 Ordinary Differential Equations 696.3 Hyperbolic Equations 716.4 Diffusion Equations 746.5 Nonlinear Problems 76

Chapter 7 Stochastic Collocation Method 78

7.1 Definition and General Procedure 787.2 Interpolation Approach 79

7.2.1 Tensor Product Collocation 817.2.2 Sparse Grid Collocation 82

7.3 Discrete Projection: Pseudospectral Approach 837.3.1 Structured Nodes: Tensor and Sparse Tensor

Constructions 857.3.2 Nonstructured Nodes: Cubature 86

7.4 Discussion: Galerkin versus Collocation 87

fm May 18, 2010

CONTENTS ix

Chapter 8 Miscellaneous Topics and Applications 89

8.1 Random Domain Problem 898.2 Bayesian Inverse Approach for Parameter Estimation 958.3 Data Assimilation by the Ensemble Kalman Filter 99

8.3.1 The Kalman Filter and the Ensemble Kalman Filter 1008.3.2 Error Bound of the EnKF 1018.3.3 Improved EnKF via gPC Methods 102

Appendix A Some Important Orthogonal Polynomials in the Askey Scheme 105

A.1 Continuous Polynomials 106A.1.1 Hermite Polynomial Hn(x) and Gaussian Distribution 106A.1.2 Laguerre Polynomial L()n (x) and Gamma Distribution 106A.1.3 Jacobi Polynomial P (,)n (x) and Beta Distribution 107

A.2 Discrete Polynomials 108A.2.1 Charlier Polynomial Cn(x; a) and Poisson Distribution 108A.2.2 Krawtchouk Polynomial Kn(x; p, N) and Binomial

Distribution 108A.2.3 Meixner Polynomial Mn(x; , c) and Negative

Binomial Distribution 109A.2.4 Hahn Polynomial Qn(x; , , N) and Hypergeometric

Distribution 110

Appendix B The Truncated Gaussian Model G(, ) 113

References 117

Index 127

fm May 18, 2010

fm May 18, 2010

Preface

The field of stochastic computations, in the context of understanding the impactof uncertainty on simulation results, is relatively new. However, over the past fewyears, the field has undergone tremendous growth and rapid development. Thiswas driven by the pressing need to conduct verification and validation (V&V) anduncertainty quantification (UQ) for practical systems and to produce predictionsfor physical systems with high fidelity. More and more researchers with diversebackgrounds, ranging from applied engineering to computer science to computa-tional mathematics, are stepping into the field because of the relevance of stochas-tic computing to their own research. Consequently there is a growing need for anentry-level textbook focusing on the fundamental aspects of this kind of stochasticcomputation. And this is precisely what this book does.

This book is a result of several years of studying stochastic computation andthe valuable experience of teaching the topic to a group of talented graduate stu-dents with diverse backgrounds at Purdue University. The purpose of this bookis to present in a systematic and coherent way numerical strategies for uncertaintyquantification and stochastic computing, with a focus on the methods based on gen-eralized polynomial chaos (gPC) methodology. The gPC method, an extension ofthe classical polynomial chaos (PC) method developed by Roger Ghanem [45] inthe 1990s, has become one of the most widely adopted methods, and in many casesarguably the only feasible method, for stochastic simulations of complex systems.This book intends to examine thoroughly the fundamental aspects of these methodsand their connections to classical approximation theory and numerical analysis.

The goal of this book is to collect, in one volume, all the basic ingredients nec-essary for the understanding of stochastic methods based on gPC methodology. Itis intended as an entry-level graduate text, covering the basic concepts from thecomputational mathematics point of view. This book is unique in the fact that itis the first book to present, in a thorough and systematic manner, the fundamen-tals of gPC-based numerical methods and their connections to classical numericalmethods, particularly spectral methods. The book is designed as a one-semesterteaching text. Therefore, the material is self-contained, compact, and focused onlyon the fundamentals. Furthermore, the book does not utilize difficult, complicatedmathematics, such as measure theory in probability and Sobolev spaces in numer-ical analysis. The material is presented with a minimal amount of mathematicalrigor so that it is accessible to researchers and students in engineering who are

fm May 18, 2010

xii PREFACE

interested in learning and applying the methods. It is the authors hope that aftergoing through this text, readers will feel comfortable with the basics of stochas-tic computation and go on to apply the methods to their own problems and pursuemore advanced topics in this perpetually evolving field.

West Lafayette, Indiana, USA Dongbin XiuMarch 2010

fm May 18, 2010

Numerical Methods for StochasticComputations

fm May 18, 2010

chapter01 May 18, 2010

Chapter One

Introduction

The goal of this chapter is to introduce the idea behind stochastic computing in thecontext of uncertainty quantification (UQ). Without using extensive discussions (ofwhich there are many), we will use a simple example of a viscous Burgers equationto illustrate the impact of input uncertainty on the behavior of a physical system andthe need to incorporate uncertainty from the beginning of the simulation and not asan afterthought.

1.1 STOCHASTIC MODELING AND UNCERTAINTY QUANTIFICATION

Scientific computing has become the main tool in many fields for understanding thephysics of complex systems when experimental studies can be lengthy, expensive,inflexible, and difficulty to repeat. The ultimate goal of numerical simulations isto predict physical events or the behaviors of engineered systems. To this end, ex-tensive efforts have been devoted to the development of efficient algorithms whosenumerical errors are under control and understood. This has been the primary goalof numerical analysis, which remains an active research branch. What has beenconsidered much less in classical numerical analysis is understanding the impactof errors, or uncertainty, in dat