Error Analysis and Simulations of Complex Phenomenafei/samsi/Readings/DHigdon/lascience.pdfReliable Predictions of Complex Phenomena There is an increasing demand for reliable predictions

Los Alamos Science Number 29 2005

Error Analysis

Error Analysis and Simulationsof Complex PhenomenaMichael A. Christie, James Glimm, John W. Grove, David M. Higdon,

David H. Sharp, and Merri M. Wood-Schultz

6

Large-scale computer-based simulations are being used increasingly to predict the behavior ofcomplex systems. Prime examples include the weather, global climate change, the performance of nuclear weapons, the flow through an oil reservoir, and the performance of advanced aircraft.Simulations invariably involve theory, experimental data, and numerical modeling, all with theirattendant errors. It is thus natural to ask, “Are the simulations believable?” “How does one assess theaccuracy and reliability of the results?” This article lays out methodologies for analyzing and com-bining the various types of errors that can occur and then gives three concrete examples of how errormodels are constructed and used.

At the top of these two pages is a simulation of low-viscosity gas (purple) displacing higher-viscosity oil (red) in an oil recovery process. Error models can be used to improvepredictions of oil production from this process. Above, at left, is a component of such anerror model, and at right is a prediction of future oil production for a particular oil reser-voir obtained from a simple empirical model in combination with the full error model.

Reliable Predictions ofComplex Phenomena

There is an increasing demand forreliable predictions of complex phe-nomena encompassing, where possi-ble, accurate predictions of full-sys-tem behavior. This requirement isdriven by the needs of science itself,as in modeling of supernovae or pro-tein interactions, and by the need forscientifically informed assessments insupport of high-consequence decisionsaffecting the environment, nationalsecurity, and health and safety. Forexample, decisions must be madeabout the amount by which green-house gases released into the atmos-phere should be reduced, whether andfor what conditions a nuclear weaponcan be certified (Sharp and Wood-Schulz 2003), or whether develop-ment of an oil field is economicallysound. Large-scale computer-basedsimulations provide the only feasiblemethod of producing quantitative,predictive information about suchmatters, both now and for the foresee-able future. However, the cost of amistake can be very high. It is there-fore vitally important that simulationresults come with a high level ofconfidence when used to guide high-consequence decisions.

Confidence in expectations aboutthe behavior of real-world phenomenais typically based on repeated experi-ence covering a range of conditions.But for the phenomena we considerhere, sufficient data for high confi-dence is often not available for a vari-ety of reasons. Thus, obtaining the

needed data may be too hazardous orexpensive, it may be forbidden as amatter of policy, as in the case ofnuclear testing, or it just may not befeasible. Confidence must then besought through understanding of thescientific foundations on which thepredictions rest, including limitationson the experimental and calculationaldata and numerical methods used tomake the prediction. This understand-ing must be sufficient to allow quanti-tative estimates of the level of accura-cy and limits of applicability of thesimulation, including evidence thatany factors that have been ignored inmaking the predictions actually have asmall effect on the answer. If, assometimes happens, high-confidencepredictions cannot be made, this factmust also be known, and a thoroughand accurate uncertainty analysis isessential to identify measures thatcould reduce uncertainties to a tolera-ble level, or mitigate their impact.

Our goal in this paper is to providean overview of how the accuracy andreliability of large-scale simulations ofcomplex phenomena are assessed, andto highlight the role of what is knownas an error model in this process.

Why Is It Hard to MakeAccurate Predictions ofComplex Phenomena?

We begin with a couple of examplesthat illustrate some of the uncertaintiesthat can make accurate predictions dif-ficult. In the oil industry, predictions offluid flow through oil reservoirs are

difficult to make with confidencebecause, although the fluid propertiescan be determined with reasonableaccuracy, the fluid flow is controlledby the poorly known rock permeabilityand porosity. The rock properties canbe measured by taking samples atwells, but these samples represent onlya tiny fraction of the total reservoirvolume, leading to significant uncer-tainties in fluid flow predictions. As ananalogy of the difficulties faced in pre-dicting fluid flow in reservoirs, imag-ine drawing a street map of Londonand then predicting traffic flows basedon what you see from twelve streetcorners in a thick fog!

In nuclear weapons certification, adifferent problem arises. The physicalprocesses in an operating nuclearweapon are not all accessible to labo-ratory experiments (O’Nions et al.2002). Since underground testing isexcluded by the Comprehensive TestBan Treaty (CTBT), full system pre-dictions can only be compared withlimited archived test data.

The need for reliable predictions isnot confined to the two areas above.Weather forecasting, global climatemodeling, and complex engineeringprojects, such as aircraft design, allgenerate requirements for reliable,quantitative predictions—see, forexample, Palmer (2000) for a study ofpredictability in weather and climatesimulations. These often depend onfeatures that are hard to model at therequired level of detail—especially ifmany simulations are required in adesign-test-redesign cycle.

More generally, because we are

Number 29 2005 Los Alamos Science 7

Error Analysis

dealing with complex phenomena,knowledge about the state of a systemand the governing physical processesis often incomplete, inaccurate, orboth. Furthermore, the strongly non-linear character of many physicalprocesses of interest can result in thedramatic amplification of even smalluncertainties in the input so that theyproduce large uncertainties in the sys-tem behavior. The effects of this sen-sitivity will be exacerbated if experi-mental data are not available formodel selection and validation.Another factor that makes predictionof complex phenomena very difficultis the need to integrate large amountsof experimental, theoretical, and com-putational information about a com-plex problem into a coherent whole.Finally, if the important physicalprocesses couple multiple scales oflength and time, very fast and veryhigh memory capacity computers andsophisticated numerical methods arerequired to produce a high-fidelitysimulation. The examples discussed inthis article exhibit many of these diffi-culties, as well as the uncertainties inprediction to which they lead.

To account for such uncertainties,models of complex systems and theirpredictions are often formulated prob-abilistically. But the accuracy of pre-dictions of complex phenomena,whether deterministic or probabilistic,varies widely in practice. For example,estimates of the amount of oil in areservoir that is at an early stage ofdevelopment are very uncertain. Largecapital investments are made on thebasis of probabilistic estimates of oilin place, so that the oil industry is fun-damentally a risk-based business. Theestimates are usually given at threeconfidence levels: p90, p50, and p10,meaning that there is a 90 percent,50 percent, and 10 percent chance,respectively, that the amount of oil inplace will be greater than the specifiedreserve level. Figure 1 shows aschematic plot (based on a real North

Sea example) of estimated reserves asa function of time. The plot clearlyshows that, as more information about

the reservoir was acquired during thecourse of field development, estimatesof the range of reserves changed out-

8 Los Alamos Science Number 29 2005

Error Analysis

Figure 1. Oil-in-Place Uncertainty Estimate Variation with TimeThis figure shows estimates of p90, p50, and p10 probabilities that the amount of oilin a reservoir is greater than the number shown. The estimated probabilities areplotted as a function of time. The variations shown indicate the difficulties involvedin accurate probability estimations. [Photo courtesy of Terrington (York) Ltd.]

Figure 2. Calibration Curve for Weather ForecastsThis plot shows estimates of the probability of precipitation from simulation fore-casts vs the observed frequency of precipitation for a large number of observa-tions. Next to each data point is the number of observations for that forecast.

side the initial prediction. In otherwords, the initial estimates of reserves,although probabilistic, did not capturethe full range of uncertainty and werethus unreliable. This situation wasobviously a cause for concern for acompany with billions of dollars ininvestments on the line.

Probabilistic predictions are alsoused in weather forecasting. If theprobabilistic forecast “20 percentchance of rain” were correct, then onaverage it would have rained on 1 in 5days that received that forecast. Dataon whether or not it rained are easilyobtained. This rapid and repeatedfeedback on weather predictions hasresulted in significantly improved reli-ability of forecasts compared with pre-dictions of uncertainty in oil reserves.The comparison between the observedfrequency of precipitation and a proba-bilistic forecast for a locality in theUnited States shown in Figure 2 con-firms the accuracy of the forecasts.

This accuracy did not come easily,and so we next briefly describe two ofthe principal methods currently usedto improve the accuracy of predictionsof complex phenomena: calibrationand data assimilation.

Calibration is a procedure wherebya simulation is matched to a particularset of experimental data by perform-ing a number of runs in which uncer-tain model parameters are varied toobtain agreement with the selecteddata set. This procedure is sometimescalled “tuning,” and in the oil industryit is known as history matching.Calibration is useful when codes areto be used for interpolation, but it isof limited help for extrapolation out-side the data set that was used for tun-ing. One reason for this lack of pre-dictability is that calibration onlyensures that unknown errors from dif-ferent sources, say inaccurate physicsand numerics, have been adjusted tocompensate one another, so that thenet error in some observable is small.Because different physical processes

and numerical errors are unlikely toscale in the same way, a calibratedsimulation is reliable only for theregime for which it has been shown tomatch experimental data.

In one variant of calibration, multi-ple simultaneous simulations are per-formed with different models. The“best” prediction is defined as aweighted average over the resultsobtained with the different models. Asadditional observations become avail-able, the more successful models arerevealed, and their predictions areweighted more heavily. If the modelsused reflect the range of modelinguncertainty, then the range of resultswill indicate the variance of the pre-diction due to those uncertainties.

Data assimilation, while basically aform of calibration, has important dis-tinctive features. One of the mostimportant is that it enables real-timeutilization of data to improve predic-tions. The need for this capabilitycomes from the fact that, in opera-tional weather forecasting, for exam-ple, there is insufficient time to restarta run from the beginning with newdata, so that this information must beincorporated on the fly. In data assim-ilation, one makes repeated correc-tions to model parameters during asingle run, to bring the code outputinto agreement with the latest data.The corrections are typically deter-mined using a time series analysis ofthe discrepancies between the simula-tion and the current observations.Data assimilation is widely used inweather forecasting. See Kao et al.(2004) for a recent application toshock-wave dynamics.

Sources of Error and How to Analyze Them

Introducing Error Models. Therole of a thorough error analysis inestablishing confidence in predictionshas been mentioned. But evaluating

the error in a prediction is often moredifficult than making the prediction inthe first place, and when confidencein the answer is an issue, it is just asimportant.

A systematic approach for deter-mining and managing error in simula-tions is to try to represent the effectsof inaccurate models, neglected phe-nomena, and limited solution accuracyusing an error model.

Unlike the calibration and dataassimilation methods discussed above,an error model is not primarily amethod of increasing the accuracy ofa simulation. Error modeling aims toprovide an independent estimate ofthe known inadequacies in the simula-tion. An error model does not purportto provide a complete and preciseexplanation of observed discrepanciesbetween simulation and experimentor, more generally, of the differencesbetween the simulation model and thereal world. In practice, an error modelhelps one achieve a scientific under-standing of the knowable sources oferror in the simulation and put quanti-tative bounds on as much of the erroras possible.

Simulation Errors. Computercodes used for calculating complexphenomena combine models fordiverse physical processes with algo-rithms for solving the governing equa-tions. Large databases containingmaterial properties such as cross sec-tions or equations of state that tie thesimulation to a real-world systemmust be integrated into the simulationat the lowest level of aggregation.These components and, significantly,input from the user of the code mustbe linked by a sophisticated computerscience infrastructure, with the resultthat a simulation code for complexphenomena is an exceedingly elabo-rate piece of software. Such codes,while elaborate, still provide only anapproximate representation of reality.

Simulation errors come from three


Error Analysis

main sources: inaccurate input data,inaccurate physics models, and limit-ed accuracy of the solutions of thegoverning equations. Clearly, each ofthese generic sources of error ispotentially important. A perfectphysics model with perfect input datawill give wrong answers if the equa-tions are solved poorly. Likewise, aperfect solution of the wrong equa-tions will also give incorrect answers.The relative importance of errors from

each source is problem dependent, buteach source of error must be evaluat-ed. Our discussion of error modelswill reflect the above comments bycategorizing simulation inadequaciesas due to input, solution, and physicserrors.

Input errors refer to errors in dataused to specify the problem, and theyinclude errors in material properties,the description of geometrical config-urations, boundary and initial condi-

tions, and others. Solution error is thedifference between the exact mathe-matical solution of the governingequations for the model and theapproximate solution of the equationsobtained with the numerical algo-rithms used in the simulation. Physicserror includes the effects of phenome-na that are inadequately represented inthe simulation, for example, theunknown details of subscale physics,such as the microscopic details ofmaterial that is treated macroscopical-ly in the simulation. Evaluations ofthe effects of these details are typical-ly based on statistical descriptions.The physics component of an errormodel is thus based on knowledge ofaspects of the nominal model thatneed or might need correction.

Experimental Errors andSolution Errors. Much of our under-standing of how to analyze errorscomes from studies of experimentalerror. We will also see below thatexperimental and solution errors playa similar role in an uncertainty analy-sis. We therefore start by discussingexperimental errors.

Experimental errors play an impor-tant role in building error models forsimulations. First, they can bias con-clusions that are drawn when simula-tion results are compared with meas-ured data. Second, experimental errorsaffect the accuracy of simulationsindirectly through their effects ondatabases and input data used in asimulation. Experimental errors areclassified as random or systematic.Typically, both types of error are pres-ent in any particular application. Afamiliar example of a random error isthe statistical sampling error quotedalong with the results of opinion polls.Another type of random error is theresult of variations in random physicalprocesses, such as the number ofradioactive decays in a sample perunit time. The signals from measuringinstruments usually contain a compo-


Error Analysis

Figure 3. Uncertainties in Reported Measurements of the Speed ofLight (1870–1960)This figure shows measured values of the speed of light along with estimates of theuncertainties in the measured values up until 1960. The error bars correspond to theestimated 90% confidence intervals. The currently accepted value lies outside theerror bars of more measurements than would be expected, indicating the difficultyof truly assessing the uncertainty in an experimental measurement. Refer to the arti-cle by Henrion and Fischoff on pp. 666–677 in Heuristics and Biases (2002) for moredetails on this and other examples of uncertainties in physical constants.(Photo courtesy of Department of Physics, Carnegie Mellon University.)

nent that either is or appears to berandom whether the process that is thesubject of the measurement is randomor not. This component is the ubiqui-tous “noise” that arises from a widevariety of unwanted or uncharacter-ized processes occurring in the meas-urement apparatus. The way in whichnoise affects a measurement must betaken into consideration to attain validconclusions based on that data. Noiseis typically treated probabilistically,either separately or included with astatistical treatment of other randomerror. However, systematic error isoften both more important and moredifficult to deal with than randomerror. It is also frequently overlooked,or even ignored.

To see how a systematic error canoccur, imagine that an opinion poll onthe importance of education was con-ducted by questioning people on streetcorners “at random”—not knowingthat many of them were coming andgoing from a major library that hap-pened to be located nearby. It is virtu-ally certain that those questionedwould on average place a higherimportance on education than the pop-ulation in general. Even if a very largenumber of those individuals werequestioned, an activity that wouldresult in a small statistical samplingerror, conclusions about the impor-tance of higher education drawn fromthis data could be incorrect for thepopulation at large. This is why care-fully conducted polls seek to avoidsystematic errors, or biases, by ensur-ing that the population sampled is rep-resentative.

As a second example, suppose that10 measurements of the distance fromthe Earth to the Sun gave a meanvalue of 95,000,000 miles due, say, toflaws in an electric cable used in mak-ing these measurements. How wouldsomeone know that 95,000,000 milesis the wrong answer? This error couldnot be revealed by a statistical analy-sis of only those 10 measurements.

Additional, independent measure-ments made with independent measur-ing equipment would suggest thatsomething was wrong if they wereinconsistent with these results.However, the cause of the systematicerror could only be identified througha physical understanding of how theinstruments work, including an analy-sis of the experimental procedures andthe experimental environment. In thisexample, the additional measurementsshould show that the electrical charac-teristics of the cable were not asexpected. To reiterate, the point ofboth examples is that an understand-ing of the systematic error in a meas-ured quantity requires an analysis thatis independent of the instrument usedfor the original measurement.

An example of how difficult it canbe to determine uncertainties correctlyis shown in Figure 3, a plot of esti-mates of the speed of light vs the dateof the measurement. The dotted lineshows the accepted value, and thepublished experimental uncertaintiesare shown as error bars. The length ofthe error bars—1.48 times the stan-dard deviation—is the “90 percentconfidence interval” for a normallydistributed uncertainty for the experi-mental error; that is, the experimentalerror bars will include the correctvalue 90 percent of the time if theuncertainty were assessed correctly. Itis evident from the figure, however,that many of the analyses were inade-quate: The true value lies outside theerror bars far more often than 10 per-cent of the time. This situation is notuncommon, and it provides an exam-ple of the degree of caution appropri-ate when using experimental results.

The analysis of experimental erroris often quite arduous, and the rigorwith which it is done varies in prac-tice, depending on the importance ofthe result, the accuracy required,whether the measurement technique isstandard or novel, and whether theresult is controversial. Often, the best

way to judge the adequacy of ananalysis of uncertainty in a complexexperiment is to repeat the experimentwith an independent method and anindependent team.

Solution errors enter an analysis ofsimulation error in several ways. Inaddition to being a direct source oferror in predictions made with a givenmodel, solution errors can bias the con-clusions one draws from comparing amodel to data in exactly the same waythat experimental errors do. Solutionerrors also can affect a simulationalmost covertly: It is common for thedata or the code output to need furtherprocessing before the two can bedirectly compared. When this process-ing requires modeling or simulationwith a different code, then the solutionerror from that calculation can affectthe comparison. As with experimentalerrors, solution errors must be deter-mined independently of the simulationsthat are being used for prediction.

Using Data to Constrain Models

The scientific method uses a cycleof comparison of model results withdata, alternating with model modifica-tion. A thorough and accurate erroranalysis is necessary to validateimprovements. The availability ofdata is a significant issue for complexsystems, and data limitations permeateefforts to improve simulation-basedpredictions. It is therefore importantto use all relevant data in spite of dif-ferences in experiment design andmeasurement technique. This meansthat it is important to have a proce-dure to combine data from diversesources and to understand the signifi-cance of the various errors that areresponsible for limitations on pre-dictability.

The way in which the various cate-gories of error can affect comparisonwith experimental data and the steps


Error Analysis

to be taken if the errors are too largeare discussed in the next section. Thecomparison of model predictions withexperimental data is often called theforward step in this cycle and is a keycomponent in uncertainty assessmentsof a predicted result. The backwardstep of the cycle for model improve-ment, which is discussed next, is thestatistical inference of an improvedmodel from the experimental data.The Bayesian framework provides asystematic procedure for inference ofan improved model from observa-tions; lastly, we describe the use ofhierarchical Bayesian models to inte-grate data from many sources.

Some discussion of the use of theterms “uncertainty” and “error” is inorder. In general, any physical quan-tity, whether random or not, has a spe-cific value—such as the number ofradioactive decays in a sample of triti-ated paint in a given 5-minute period.The difference between that actualnumber and an estimate determinedfrom knowledge of the number of tri-tium nuclei present and the tritiumlifetime is the error in that estimate. Ifthe experiment were repeated manytimes, a distribution of errors wouldarise, and the probability density func-tion for those errors is the uncertaintyin the estimate.

Decomposition of Errors. Ourability to predict any physical phe-nomenon is determined by the accura-cy of our input data and our modelingapproach. When the modeling inputdata are obtained by analysis ofexperiments, the experimental errorand modeling error (solution errorplus physics approximations) termscontrol the accuracy of our estimationof those data, and hence our ability topredict. Because a full uncertainty-quantification study is in itself a com-plex process, it is important to ensurethat those errors whose size can becontrolled—either by experimentaltechnique or by modeling/simulation

choices—are small enough to ensurethat predictions of the phenomena ofinterest can be made with sufficientprecision for the task at hand. Thismeans that simpler techniques areoften appropriate at the start of astudy to ensure that we are operatingwith the required level of precision.

The discrepancy between simula-tion results and experimental data isillustrated in Figure 4, which showsthe way in which this discrepancy canbe related to measurement errors andsolution errors. Note that the experi-mental conditions are also subject touncertainties. This means that theobserved value may be associated

with a slightly different condition thanthe one for which the experiment wasdesigned, as shown in Figure 4.

The three steps below could serveas an initial, deterministic assessmentof the discrepancy between simulationand experiment.

Step 1. Compare Simulated andExperimental Results. The size of themeasurement error will obviouslyaffect the conclusions drawn from thecomparison. Those conclusions canalso be affected by the degree ofknowledge of the actual as opposed tothe designed experimental conditions.For example, the as-built compositionof the physical parts of the system


Error Analysis

Figure 4. Comparing Experimental Measurements with Simulations The green line shows the true, unknown value of an observable over the range ofuncertainty in the experimental conditions, and the purple cross indicates theuncertainty in the observation. The discrepancy measures the difference betweenobservation and simulation.

under investigation may differ slightlyfrom the original design. The effectsof both of these errors are typicallyreported together, but they are explic-itly separated here because error inthe experimental conditions affectsthe simulated result, as well as themeasured result, as can be seen inFigure 4.

Step 2. Evaluate Solution Errors. Ifthe error is a simple matter of numeri-cal accuracy—for example, spatial ortemporal resolution—then the error isa fixed, determinable number in prin-ciple. In other cases—for example,subgrid stochastic processes—theerror may be knowable in only a sta-tistical sense.

Step 3. Determine Impact onPredictability. If the discrepancy islarge compared with the solution errorand experimental uncertainty, then themodel must be improved. If not, themodel may be correct, but in eithercase, the data can be used to define arange of modeling parameters that isconsistent with the observations. If thatrange leads to an uncertainty in predic-tion that is too large for the decisionbeing taken, the experimental errors orsolution errors must be reduced.

A significant discrepancy in step 1indicates the presence of errors in thesimulation and/or experiment, andsteps 2 and 3 are necessary, but notsufficient, to pinpoint the source(s) oferror. However, these simple steps donot capture the true complexity ofanalyzing input or modeling errors. Inpractice, the system must be subdivid-ed into pieces for which the errors canbe isolated (see below) and independ-ently determined. The different errorsmust then be carefully recombined todetermine the uncertainties in integralquantities, such as the yield of anuclear weapon or the production ofan oil well, that are measured in fullsystem tests. A potential drawback ofthis paradigm is that experiments onsubsystems may not be able to probethe entire parameter space encoun-

tered in full system operation.Nevertheless, because the need to pre-dict integral quantities motivates thedevelopment and use of simulation, acrucial test of the “correctness” of asimulation is that it consistently andaccurately matches all available data.

Statistical Prediction

A major challenge of statisticalprediction is assessing the uncertaintyin a predicted result. Given a simula-tion model, this problem reduces tothe propagation of errors from thesimulation input to the simulatedresult. One major problem in examin-ing the impact of uncertainties ininput data on simulation results is the“curse of dimensionality.” If the prob-lem is described by a large number ofinput parameters and the response sur-face is anything other than a smoothquasilinear function of the input vari-ables, computing the shape of theresponse surface can be intractableeven with large parallel machines. Forexample, if we have identified 8 criti-cal parameters in a specific problemand can afford to run 1 million simu-lations, we can resolve the responsesurface to an accuracy of fewer than7 equally spaced points per axis.

Various methods exist to assess themost important input parameters.Sensitivities to partial derivatives canbe computed either numerically orthrough adjoint methods. Adjointmethods allow computation of sensi-tivities in a reasonable time and arewidely used.

Experimental design techniquescan be used to improve efficiency.Here, the response surface is assumedto be a simple low-order polynomialin the input variables, and then statis-tical techniques are used to extract themaximum amount of information fora given number of runs. Principalcomponent analysis can also be usedto find combinations of parameters

that capture most of the variability.The principle that underlies many

of these techniques is that, for a com-plex engineering system to be reli-able, it should not depend sensitivelyon the values of, for example, 104 ormore parameters. This is as true for aweapon system that is required tooperate reliably as it is for an oil fieldthat is developed with billions of dol-lars of investment funds.

Statistical Inference—TheBayesian Framework. The Bayesianframework for statistical inferenceprovides a systematic procedure forupdating current knowledge of a sys-tem on the basis of new information.In engineering and natural scienceapplications, we represent the systemby a simulation model m, which isintended to be a complete specifica-tion of all information needed to solvea given problem. Thus m includes thegoverning evolution equations (typi-cally, partial differential equations) forthe physical model, initial and bound-ary conditions, and various modelparameters, but it would not generallyinclude the parameters used to specifythe numerical solution procedureitself. Any or all of the information inm may be uncertain to some degree.To represent the uncertainty that maybe present in the initial specificationof the system, we introduce an ensem-ble of models M, with m ∈ M, anddefine a probability distribution onM. This is called the prior distribu-tion and is denoted by p(m).

If additional information about thesystem is supplied by an observationO, one can determine an updated esti-mate of the probability for m, calledthe posterior distribution and denotedby p(m|O), by using Bayes’ formula

(1)


Error Analysis

It is important to realize that theBayesian procedure does not deter-mine the choice of p(m). Thus, inusing Bayesian analysis, one mustsupply the prior from an independentdata source or a more fundamentaltheory, or otherwise, one must use anoninformative “flat” prior.

The factor p(O|m) in Equation (1)is called the likelihood. The likeli-hood is the (unnormalized) condi-tional probability for the observationO, given the model m. In the cases ofinterest here, model predictions aredetermined by solutions s(m) of thegoverning equations. The simulatedobservables are functionals O(s(m))of s(m). If both the experimentallymeasured observables O and the solu-tion s(m), hence O(s(m)), are exact,the likelihood p(O|m) is a delta func-tion concentrated on the hypersurfacein M defined by the equation

(2)

Real-world observations and simula-tions contain errors, of course, so thata discrepancy will invariably beobserved between O and O(s(m)).Because the likelihood is evaluatedsubject to the hypothesis that themodel m ∈ M is correct, any suchdiscrepancy can be attributed toerrors either in the solution or in the

measurements. The likelihood isdefined by assigning probabilities tosolution and/or measurement errorsof different sizes. The required prob-ability models for both types oferrors must be supplied by an inde-pendent analysis.

This discussion shows that the roleof the likelihood in simulation-basedprediction is to assign a weight to amodel m based on a probabilisticmeasure of the quality of the fit of themodel predictions to data. Probabilitymodels for solution and measurementerrors play a similar role in determin-ing the likelihood.

This point is so fundamental andsufficiently removed from commonapproaches to error analysis that werepeat it for emphasis: Numericaland observation errors are the lead-ing terms in the determination of theBayesian likelihood. They supplycritical information needed foruncertainty quantification.

Alternative approaches to infer-ence include the use of intervalanalysis, possibility theory, fuzzysets, theories of evidence, and others.We do not survey these alternativeshere, but simply mention that theyare based on different assumptionsabout what is known and what can beconcluded. For example, intervalanalysis assumes that unknown


Error Analysis

Figure 5. Bayesian Framework forPredicting System Performancewith Relevant UncertaintiesMultiple simulations are performedusing the full physical range of parame-ters. The discrepancies between theobservation and the simulated valuesare used in a statistical inference proce-dure to update estimates of modelingand input uncertainties. The updateinvolves computing the likelihood of themodel parameters by using Bayes’ theo-rem. The likelihood is computed from aprobability model for the discrepancy,taking into account the measurementerrors (shown schematically by thegreen dotted lines) and the solutionerrors (blue dotted lines). The updatedparameter values are then used to pre-dict system performance, and a deci-sion is taken on whether the accuracyof the predictions is adequate.

parameters vary within an interval(known exactly), but that the distri-bution of possible values of theparameter within the interval is notknown even in a probabilistic sense.This method yields error bars but notconfidence intervals.

An illustration of the Bayesianframework we follow to compute theimpact of solution error and experi-mental uncertainty is shown inFigure 5. Multiple simulations areperformed with the full physicalrange of parameters. The discrepan-cies (between simulation and obser-vation) are used in a statistical infer-ence procedure to update estimates ofmodeling and input uncertainties.These updated values are then usedto predict system performance, and adecision is taken on whether theaccuracy of the predictions is adequate.

Combining Information fromDiverse Sources

Bayesian inference can be extend-ed to include multiple sources ofinformation about the details of aphysical process that is being simu-lated (Gaver 1992). This informationmay come from “off-line” experi-ments on separate components of thesimulation model m, expert judg-ment, measurements of the actualphysical process being simulated,and measurements of a physicalprocess that is related, but not identi-cal, to the process being simulated.Such information can be incorporat-ed into the inference process byusing Bayesian hierarchical models,which can account for the nature andstrength of these various sources ofinformation. This capability is veryimportant since data directly bearingon the process being modeled isoften in short supply and expensiveto acquire. Therefore, it is essentialto make full use of all possible

sources of information—even those that provide only indirectinformation.

In principle, an analysis can uti-lize any experimental data that canbe compared with some part of theoutput of a simulation. To understandthis point, let us make the simple andoften useful assumption that the fam-ily of possible models M can beindexed by a set of parameters . Inthis case, the somewhat abstractspecification of the prior as a proba-bility distribution p(m) on modelscan be thought of simply as a proba-bility distribution p(θ) on the param-eters θ. Depending on the applica-tion, θ may include parameters thatdescribe the physical properties of asystem, such as its equation of state,or that specify the initial and bound-ary conditions for the system, tomention just a few examples. In anyof these cases, uncertainty in θaffects prediction uncertainty.Typically, different data sources willgive information about differentparameters.

Multiple sources of experimentaldata can be included in a Bayesiananalysis by generalizing the likeli-hood term. If, for example, theexperimental observations O decom-pose into three components (O1, O2,O3), the likelihood can be written as

if we assume that each component ofthe data gives information about anindependent parameter θ. The sub-scripts on the models are there toremind us that, although the samesimulation model is used for each ofthe likelihood components, differentsubroutines within the simulationcode are likely to be used to simulate

the different components of the out-put. This means that each of the like-lihood terms will have its own solu-tion error, as well as its own observa-tion error. The relative sizes of theseerrors greatly affect how these vari-ous data sources constrain θ. Forexample, if it is known that m2(θ )does not reliably simulate O2, thenthe likelihood should reflect this fact.Note that a danger here is that a mis-specification of a likelihood termmay give some data sources undueinfluence in constraining possiblevalues of one of the parameters θ.

In some cases, one (or more) com-ponent (components) of the observeddata is (are) not from the actualphysical system of interest, but froma related system. In such cases,Bayesian hierarchical models can beused to borrow strength from thatdata by specifying a prior model thatincorporates information from thedifferent systems. See Johnson et al.(2003) for an example.

Finally, expert judgment usuallyplays a significant role in the formu-lation and use of models of complexphenomena—whether or not themodels are probabilistic. Sometimes,expert judgment is exercised in anindirect way, through selection of alikelihood model or through thechoice of the data sources to beincluded in an analysis. Expert judg-ment is also used to help with thechoice of the probability distributionfor p(θ), or to constrain the range ofpossible outcomes in an experiment,and such information is ofteninvoked in applications for whichexperimental or observational dataare scarce or nonexistent. However,the use of expert judgment is fraughtwith its own set of difficulties. Forexample, the choice of a prior canleave a strong “imprint” on resultsinferred from subsequent experi-ments. See Heuristics and Biases(2002) for enlightening discussionsof this topic.


Error Analysis

Building Error Models—Examples

Dropping Objects from a Tower.Some of the basic ideas used in build-ing error models are illustrated inFigure 6. In this example, experimentalobservations are combined with a sim-ple physics model to predict how longit takes an object to fall to the groundwhen it is dropped from a tower. Theexperimental data are drop timesrecorded when the object is droppedfrom each of six floors of the tower.The actual drop time is measured withan observation error, which we assumefor illustrative purposes to be Gaussian(normal), with mean 0 and a standarddeviation of 0.2 second. The physicsmodel is based solely on the accelera-tion due to gravity. We observe that thepredicted drop times are too short andthat this discrepancy apparently growswith the height from which the object isdropped.

Even though this model shows asubstantial error, which is apparentfrom the discrepancy between theexperimental data and the model pre-dictions (Figure 6(b)), it can still bemade useful for predicting drop timesfrom heights that are greater than theheight of the tower. As a first step, weaccount for the discrepancy by includ-ing an additional unknown correctionin the initial specification of the model,namely, in the prior. This term repre-sents the discrepancy as an unknown,smooth function of drop height that isestimated (with uncertainty) in theanalysis. The results are applied to givepredictions of drop times for heightsthat would correspond to the sevenththrough tenth floors of the tower. Thesepredictions have a fair amount ofuncertainty because the discrepancyterm has to be extrapolated to dropheights that are beyond the range of theexperimental data. Note also that theprediction uncertainty increases withdrop height (refer to Figure 6(c)).

This strictly phenomenological


Error Analysis

Figure 6. Dropping an Objectfrom a Tower(a) The time it takes an object to dropfrom each of 6 floors of a tower isrecorded. There is an uncertainty in themeasured drop times of about ±0.2 s.Predictions for times are desired fordrops from floors 7 through 10, but theydo not yet exist.

(b) A mathematical model is developedto predict the drop times as a functionof drop height. The simulated droptimes (red line) are systematically toolow when compared with the experi-mental data (triangles). The error barsaround the observed drop times showthe observation uncertainty.

(c) This systematic deviation betweenthe mathematical model and the experi-mental data is accounted for in the like-lihood model. A fitted correction termadjusts the model-based predictions tobetter match the data. The resulting 90%prediction intervals for floors 7 through10 are shown in this figure. Note thatthe prediction intervals become wideras the drop level moves away from thefloors with experimental data. The cyantriangles corresponding to floors 7through 10 show experimental observa-tions taken later only for validation ofthe predictions.

(d) An improved simulation model wasconstructed that accounts for air resist-ance. A parameter controlling thestrength of the resistance must be esti-mated from the data, resulting in someprediction uncertainty (90% predictionintervals are shown for floors 7 through10). The improved model captures moreof the physics, giving reduced predic-tion uncertainty.

modeling of the error leads to resultsthat can be extrapolated over a verylimited range only, because predictionsof drop times from just a few floorsabove the sixth have unacceptably largeuncertainties. But an improved physicsmodel can greatly extend the rangeover which useful predictions can bemade. Thus, we next carry out ananalysis using a model that incorporatesa physically motivated term for airresistance. This model requires estima-tion of a single additional parameterappearing as a coefficient in the airresistance term. But when this parame-ter is constrained by experimental data,much better agreement with the meas-ured drop times is obtained (seeFigure 6(d)). In fact, in this case, thediscrepancy is estimated to be nearlyzero. The remaining uncertainty in thisimproved prediction results from uncer-tainties in the measured data and in thevalue of the air resistance parameter.

Using an Error Model toImprove Predictions of OilProduction. In most oil reservoirs,the oil is recovered by injecting afluid to displace the oil toward theproduction wells. The efficiency ofthe oil recovery depends, in part, onthe physical properties of the displac-ing fluid. The example in this sectionconcerns estimation of the viscosity(typically poorly known) of an inject-ed gas displacing oil in a porousmedium. We will show how an errormodel for such estimates allowsimproved estimates of the uncertaintyin future oil production using thismethod of recovery.

Because the injected gas has lowerviscosity than the oil, the displace-ment process is unstable and viscousfingers develop (see Figure 7). Thephenomenon is similar to theRayleigh-Taylor instability of a densefluid on top of a less dense fluid. Thefingers have a reasonably predictableaverage behavior, but there is somerandomness in their formation and

evolution associated with the lack ofknowledge of the initial conditionsand with unknown small-scale fluctu-ations in rock properties.

The oil industry has a simpleempirical model that accounts for theeffects of fingering. This model,called the Todd and Longstaff model,fits an expansion wave (rarefactionfan) to the average behavior. Althoughthe model is good, it is not perfect,and in particular, when applied tocases with a correlated permeabilityfield, it tends to underestimate thespeed with which the leading edge ofthe gas moves through the medium. If we compare results from the Toddand Longstaff model with observeddata in order to estimate physicalparameters such as viscosity, we willintroduce errors into the parameterestimates because of the errors in thesolution method. To compensate forthese errors, we create a statisticalmodel for the solution errors.1

For this example, we assume thatthe primary unknown in the Todd andLongstaff model is the ratio of gasviscosity to oil viscosity, which deter-mines the rate at which instabilitiesgrow. This ratio will be determined by

comparing simulation and observation(in practice, oil and gas viscositieswould be measured, although therewould still be uncertainties associatedwith amounts of gas dissolved in theoil). To construct a solution errormodel for the average gas concentra-tion in the reservoir, we run a numberof fine-grid simulations at discretevalues of the viscosity ratio, which werefer to as calibration points. Then, foreach value of the viscosity ratio, wecompute the difference between theTodd and Longstaff model and thefine-grid simulations as a function ofscaled distance along the flow (x) anddimensionless time (t) (time dividedby the time for gas to break throughin the absence of fingering). Themean error computed in this way forthe viscosity ratio 10 is shown inFigure 8 as a function of the similarityvariable x/t. We also compute thestandard deviation of the error at eachtime, as well as the correlationbetween errors at different times. Thisinformation is represented as a“covariance matrix.”

We will show that the solutionerror model (the mean error and thecovariance matrix), when used in con-junction with predictions of the Toddand Longstaff model at different vis-cosity ratios, can yield good estimatesof the viscosity ratio for a given pro-duction data set. Figure 9 shows the


Error Analysis

Figure 7. Viscous Fingering in a Realization of Porous Media FlowLow-viscosity gas (purple) is injected into a reservoir to displace higher-viscosityoil (red). The displacement is unstable and the gas fingers into the oil, reducingrecovery efficiency.

1 All the results cited in this section arefrom Alannah O’Sullivan’s Ph.D. thesison error modeling (O’Sullivan 2004). Weare grateful to her for permission to usethese unpublished results in this article.

observed production data (blackcurve) for which we wish to deter-mine the unknown viscosity ratio. Wefirst run the Todd and Longstaffmodel at different viscosity ratiosfrom a prior range of 5 to 25 and thencorrect each prediction by adding to itthe mean error at that specific viscosi-ty ratio. The mean error at each vis-cosity ratio is calculated by interpolat-ing between the mean error at theknown calibration points for eachvalue of the similarity variable x/t.The blue curve in Figure 9 gives anexample of a Todd and Longstaff pre-diction, and the red curve gives thecorrected curve obtained by addingthe mean error to the blue curve. Toapply the error model, we have con-verted from the similarity variable x/tto time using the known length of thesystem.

After calculating the corrected pre-dictions for each viscosity ratio, thenext step is to compare the correctedprediction (an example is shown inred) for each viscosity ratio with theobserved data (shown in black) andcompute the misfit M between thesimulation and the data. The misfit isgiven by

(3)

where o is the observed value, s is thesimulated value, e– is the mean error,and the covariance matrix is given byC = σ2

dI + Csem. That is, for thecovariance matrix, we assume that thedata errors are Gaussian, independent,and identically distributed and thattherefore they have a standard devia-tion of σ2

d, and we estimate the solu-tion error model covariance matrixCsem from the fine-scale simulationsperformed at the calibration points.The red curve in Figure 10 shows themisfits as a function of viscosity ratiocomputed using the full error model asin Equation (3). The other misfit statis-


Error Analysis

Figure 9. Observed Production Compared with PredictionsThe mean error from the error model is added to the coarse-grid result (blue curve)at each time to generate an improved estimate of the gas concentration produced(red curve). The black curve is observed data (actually synthetic data calculatedusing the fine-grid model with oil-gas viscosity ratio equal to 13).

Figure 8. Mean Error and Data to Compute Mean ErrorThe black curve is the mean error in the gas concentration for viscosity ratio 10. Thedata to compute the mean error (gray curves) come from the differences between asingle coarse-grid or approximate solution (in this case, the Todd and Longstaffmodel) and multiple fine-grid realizations, all computed at viscosity ratio 10. Thevariability in the fine-grid realizations reflects random fluctuations in the permeabili-ty field, which create different finger locations and growth paths. In this case, thegas concentration averaged across the flow from the fine-grid solution is subtractedfrom the coarse Todd and Longstaff prediction as a function of x (distance along theflow) divided by t (time). In the example discussed in the text, we compute the meanerror and covariance matrix at viscosity ratios 5, 10, and 15, and interpolation isused to predict the behavior in between these values.

tics in Figure 10 were computed using

for the least-squares model and

for least-squares plus mean-errormodel.

The likelihood function L for theviscosity ratio is then given byL = exp(–M). Notice that the exponen-tial is a signal that the probabilitiesare sensitive to the method used forcomputing the misfit. The likelihoodsare converted to probability distribu-tion functions by being normalized sothat they integrate to 1.

To illustrate the improvement inparameter estimation that results fromusing an error model, we computedestimates of the probability distribu-tion function for the unknown viscosi-ty ratio using the three different misfitcurves in Figure 10, which were cal-culated with the three different meth-ods: standard least squares, leastsquares modified by the addition of amean error term, and least squareswith the inclusion of the mean errorplus the full covariance matrix. Therange of possible values for the vis-cosity ratio and their posterior proba-bilities are shown in Figure 11.

The true value of the viscosityratio used to generate the “observed”(synthetic) production data inFigure 9 was 13, and one can see thatthis value has been accurately identi-fied by the full error model. The stan-dard least-squares method has notidentified this value because of theunderlying bias in the Todd andLongstaff model.

We sample from the estimatedprobability distribution for the viscos-ity ratio to generate a forecast ofuncertainty in future production.Figure 12 is a plot of the maximumlikelihood prediction from the Toddand Longstaff model, along with the


Error Analysis

Figure 10. Misfit Statistic vs Viscosity Ratio Calculated in Three WaysThis figure shows a plot of misfit as a function of the viscosity ratio. The misfit iscomputed using a standard least-squares approach (black curve), least squares withmean error added (blue curve), and the full error model. The misfit measures thequality of the fit to the observed data with low misfits indicating a good fit.

Figure 11. Posterior Probability Distribution Functions for the ViscosityRatio Calculated in Three WaysThis figure shows the estimated posterior probability (assuming a uniform priorprobability in the range 5–25) of the viscosity ratio obtained from three differentmethods for matching the Todd and Longstaff predictions to observed data. Theblack curve is obtained from the Todd and Longstaff predictions and a standardleast-squares approach. The probability density rises to a maximum at the upperend of the viscosity range specified in the prior model. The blue curve shows theeffect of adding the mean error to the predictions. The bias in the coarse model hasbeen removed, but the uncertainty is still large. The red curve shows the estimatedviscosity ratio from a full error model treatment—refer to Equation (3)—indicatingthat it is possible to use a statistical model of solution error to get a good estimateof a physical parameter. The true value of the viscosity ratio in this example was 13.

95 percent confidence limits obtainedby sampling for different values ofviscosity. In addition, 20 predictionsfrom fine-grid simulation are shown.They use the exact viscosity ratio 13.The uncertainty in the evolution ofthe fingers gives rise to the uncertain-ty in prediction shown by the multi-ple light-gray curves. It is clear fromthe figure that use of an error modelhas allowed us to produce well-cali-brated predictions.

Fluid Dynamics—Error Modelsfor Reverberating Shock Waves.Compressible flow exhibits remark-able phenomena, one of the moststriking being shock waves, which arepropagating disturbances character-ized by sudden and often large jumpsin the flow variables across the wavefront (Courant and Friedrichs 1967).In fact, for inviscid flows, these jumpsare represented as mathematical dis-

continuities. Shock waves play aprominent role in explosions, super-sonic aerodynamics, inertial confine-ment fusion, and numerous otherproblems. Most problems of practicalimportance involve two- or three-dimensional (2-D or 3-D) flows, com-plex wave interactions, and othercomplications, so that a quantitativedescription of the flow can beobtained only by solving the fluid-flow equations numerically. The abili-ty to numerically simulate complexflows is a triumph of modern science,but such simulations, like all numeri-cal solutions, are only approximate.The errors in the numerical solutioncan be significant, especially when thecomputations use moderate to coarsecomputational grids as is often neces-sary for real-world problems. In thissection, we sketch an approach to esti-mating these errors.

Our approach makes heavy use of

the fact that shock waves are persist-ent, highly localized wave distur-bances. In this case, “persistent”means that shock waves propagate aslocally steady-state wave fronts thatcan be modified only by interactionswith other waves or unsteady flows.Generally, interactions consist of col-lisions with other shock waves,boundaries, or material interfaces. Thephrase “highly localized” refers toshock fronts being sharp and theirinteractions occurring in limitedregions of space and time and possi-bly being characterized by the refrac-tion of shock fronts into multiplewave fronts of different families.These properties are illustrated inFigure 13, which shows a sequence ofwave interactions being initiated whena shock incident from the left collideswith a contact located a short distancefrom a reflecting wall at the rightboundary in the figure. Each collisionevent produces three outgoing waves:a transmitted shock, a contact discon-tinuity, and a reflected shock or rar-efaction wave. The buildup of a com-plex space-time pattern due to themultiple wave interactions is evident.

Generally, solution errors are deter-mined by comparison to a fiducialsolution, that is, a solution that isaccepted, not necessarily as perfect,but as “correct enough” for the prob-lem being studied. But producing afiducial solution may not be easy. Inprinciple, one might obtain one using avery highly resolved computation.However, in real-world problems, thisis generally not feasible. If it were, onewould just do it and forget about solu-tion errors. So, what do we do whenwe cannot compute a fiducial solution?

The development of models forerror generation and propagationoffers an approach for dealing withflows that are too complex for directcomputation of a fiducial solution. Forcompressible flows, the key point isthat the equations are hyperbolic,which implies that errors are largely


Error Analysis

Figure 12. Prediction of Future Oil Production Using Error ModelThe solid red line shows the mean (maximum likelihood) prediction from the Toddand Longstaff model and the full error model. The dashed red lines show the 95%confidence interval, and the fine gray curves show the results from 20 fine-grid sim-ulations using the exact viscosity ratio of 13.

advected through smooth-flowregions and significant errors areonly created when wave fronts col-lide. The flow shown in Figure 13consists of a sequence of binarywave interactions, each of which issimple enough to be computed on anultrafine grid. The basic idea is todetermine the solution errors for anelementary wave interaction and toconstruct “composition laws” thatgive the error at any given point interms of the error generated at eachof the elementary wave interactionsin its domain of influence.

A number of points need to bemade here. First, there are a limitednumber of types of elementary waveinteractions. One-dimensional (1-D)interactions occur as refractions ofpairs of parallel wave fronts, 2-Dinteractions are refractions of twooblique wave fronts, and 3-D inter-actions correspond to triple pointsproduced by three interacting waves.It is important to note that, in eachspatial dimension, the elementarywave interactions occur at isolatedpoints. Most of the types of waveinteractions that can occur in 1-Dflow appear in Figure 13. The coher-ent traveling wave interactions thatoccur in 2-D flows have been charac-terized (Glimm et al. 1985). However,substantial limitations are left on therefinement and thoroughness withwhich 3-D elementary wave interac-tions can be studied.

Event 1 in Figure 13 is a typicalexample of a 1-D wave interaction.Here, the “incoming waves” consist ofan incident shock and a contact dis-continuity, and the “outgoing state” isdescribed by a reflected shock, a(moving) contact, and a transmittedshock. The interaction can bedescribed as the solution to aRiemann problem with data given bythe states behind the incoming wavefronts. A Riemann problem is definedas the initial value problem for ahyperbolic system of conservation

laws with scale-invariant initial data.Riemann problems and their solutionsare basic theoretical tools in the studyof shock dynamics, the developmentof shock-capturing schemes to numer-ically compute flows, and they alsoplay a key role in our study of solu-tion errors. A key point in the use ofRiemann problem solutions in ourerror model is that the solution of a1-D Riemann problem for hydrody-namics reduces to solving a single,relatively simple algebraic equation. Itis thus possible to solve large numbers

of Riemann problems for a flowanalysis quickly and efficiently. Thisobservation is particularly importantbecause our error model requires thesolution of multiple Riemann prob-lems whose data are drawn from sta-tistical ensembles of initial data torepresent uncertainties in the incom-ing waves.

A final point here is that a realisticsolution error model must include thestudy of the size distribution of errorsover an ensemble of problems, inwhich the variability of problem char-


Error Analysis

Figure 13. The Space-Time Interaction History of a Shock-TubeRefractionThis figure shows the interaction history as reconstructed from the simulated solu-tion data from a shock-tube refraction problem. A planar shock is incident from theleft on a contact discontinuity located near the middle of the test section of theshock tube. A reflecting wall is located on the right side of the tube. Event 1 corre-sponds to the initial refraction of the shock wave into reflected and transmittedwaves, event 2 occurs when the transmitted shock produced by interaction 1reflects at the right wall, and the events numbered 3–10 correspond to subsequentwave interactions between the various waves produced by earlier refractions orreflections. Our error model is applied at each interaction location to estimate theadditional solution error produced by the interaction.(This figure was supplied courtesy of Dr. Yan Yu, Stony Brook University.)

acteristics is described probabilistical-ly. Of course, one will often want tomake as refined an error analysis aspossible within a given realizationfrom the ensemble (that is, a deter-ministic error analysis), but there arepowerful reasons for a probabilisticanalysis to be needed as well. First,you need probability to describe fea-tures of a problem that are too com-plex for feasible deterministic analy-sis. Thus, fine details of error genera-tion in complex flows are modeled asrandom, just as are some details of theoperation of measuring instruments.Second, a sensitivity analysis is need-ed to determine the robustness of theconclusions of a deterministic erroranalysis to parameter variation. To getan accurate picture, one needs to dosensitivity analysis probabilistically,to answer the question of how likelythe parameter variations are that leadto computed changes in the errors.Third, to be a useful tool, the errormodel must be applicable to a reason-able range of conditions and prob-lems. The only way we are aware offor achieving these goals is to base

the error model on a study of anensemble of problems that reflects thedegree of variability one expects toencounter in practice. Of course, thechoice of such an ensemble reflectsscientific judgment and is an ongoingpart of our effort.

Now, let us return to the analysisof solution errors in elementary waveinteractions. Our work was motivatedby a study of a shock-contact interac-tion—refer to event 1 in Figure 13.The basic setup is shown in Figure 14,which illustrates a classic shock-tubeexperiment. An ensemble of problemswas generated by sampling from uni-form probability distributions(±10 percent about nominal values)for the initial shock strength and thecontact position. The solution errorswere analyzed by computing the dif-ference between coarse to moderategrid solutions and a very fine gridsolution (1000 cells). Error statisticsare shown in Figure 15 for a 100-cellgrid (moderate grid) solution. Twofacts about these solution errors areapparent. First, the solution errors fol-low the same pattern as the solution

(the shock waves) itself; they are con-centrated along the wave fronts,where steep gradients in the solutionoccur. Second, errors are generated atthe location of wave interactions. Theerror generated by the interactionincrements the error in the outgoingwaves, which is inherited from errorsin the incoming waves.

Comparable studies have been car-ried out for each of the types of waveinteraction shown in Figure 13, as wellas corresponding wave interactionsthat occur in spherical implosions orexplosions (Dutta et al. 2004). Ananalysis of statistical ensembles ofsuch interactions has led us to suggestthe following scheme for estimatingthe solution errors. The key steps are(a) identification of the main wavefronts in a flow, (b) determination ofthe times and locations of wave inter-actions, and (c) approximate evalua-tion of the errors generated during theinteractions. Wave fronts are mostsimply identified as regions of largeflow gradients, and the distribution ofthe wave positions and velocities arefound by solving Riemann problemswhose data are taken from ensemblesof state information near the detectedwave fronts. The error generated dur-ing an interaction is fit by a linearexpression in the uncertainties of theincoming wave’s strength. The coeffi-cients are computed using a least-squares fit to the distribution of outgo-ing wave strengths. This fitting proce-dure can be thought of as defining aninput/output relation between errors inincoming and outgoing waves.

A linear relation of this kind,which amounts to treating the errorsperturbatively, holds even for strong,and hence nonlinear, wave interac-tions. But there are limitations.Linearity works if the errors in theincoming waves are not too large, butit may break down for larger errors.In the latter case, higher order (forexample, bilinear or rational) terms in the expansion may be needed. See


Error Analysis

ρ = 1

P = 10–3

ν = 0

ρ(ν) = 3.97398

P(ν) = 1.33725

ν U[.9,1.1], ν = 1

γ = 5/3

0 ≤ t ≤ 3.5

ρ = 1

P = 10–3

ν = 0

0.50 1.0 1.5 2.0 2.5

Reflecting w

all

Open b

oundary

Ms(ν) = 32.6986

νs(ν) = 1.33625

S = 0.25 C U[.9,1.1], C = 1

Figure 14. Initial Data for a 1-D Shock-Tube Refraction Problem This schematic diagram is for the initial data used to conduct an ensemble of simula-tions of a 1-D shock tube refraction. Each simulation consisted of a shock wave inci-dent from the left on a contact discontinuity between gases at the indicated pres-sures and densities. Each realization from the ensemble is obtained by selecting ashock strength consistent with a velocity v behind the incident shock taken from a10% uniform distribution about the mean value v– = 1, and an initial contact locationC chosen from a 10% uniform distribution about the mean position C

–= 1. In the dia-

gram, S is the shock position, Ms is the shock strength, and vs is the velocity of theshock. The initial state behind the shock is set by using the Rankine-Hugoniot condi-tions for the realization shock strength and the specified state ahead of the shock.

Glimm et al. (2003) for details.We can now explain how the com-

position law for solution errors actual-ly works. The basic idea is that errorsare introduced into the problem bytwo mechanisms: input errors that arepresent in waves that initiate thesequence of wave interactions—seethe incoming waves for event 1 inFigure 13—and errors generated ateach interaction site. However theyare introduced, errors advect with theflow and are transferred at each inter-action site by computable relations.

Generally, waves arrive at a givenspace-time point by more than onepath. Referring again to Figure 13,suppose you want to find the errors inthe output waves for event 3, wherethe shock reflected off the wallreshocks the contact. On path A, theerror propagates directly from the out-put of interaction 1 along the path ofthe contact, where it forms part of theinput error for event 3. On path B, theoutput error in the transmitted shockfrom event 1 follows the transmittedshock to the wall, where it is reflectedand then re-crosses the contact. In thisway, the error coming into event 3 isgiven as a sum of terms, with eachterm labeled by a sequence of wave

interactions and of waves connectingthese interactions. Moreover, eachterm can be computed on the basis ofelementary wave interactions anddoes not require the full solution ofthe numerical problem. The final stepin the process is to compute the errorsin the output waves at event 3, byusing the input/output relations devel-oped for this type of wave interaction.

This procedure represents a sub-stantial reduction in the difficulty ofthe error analysis problem, and wemust ask whether it actually works.Full validation requires use in practice,of course. As a first validation step, wecompute the error in two ways. First,we compute the error directly by com-paring very fine and coarse-grid simu-lations for an entire wave pattern.Results are shown in Figure 15.Second, we compute the error usingthe composition law procedure shownin Figure 13. Comparing the errorscomputed in these two ways providesthe basis for validation.

In Glimm et al. (2003) and Dutta etal. (2004), we carried out such valida-tion studies for planar and sphericalshock-wave reverberation problems.As an example, for events 1 to 3 inthe planar problem in Figure 13, we

considered three grid levels, the finest(5000 cells) defining the fiducial solu-tion, and the other two representing“resolved” (500 cells) and “under-resolved” (100 cells) solutions for thisproblem. We introduced a 10 percentinitial input uncertainty to define theensemble of problems to be examined.The results can be summarized brieflyas follows. For the resolved case, thecomposition law gave accurate resultsfor the errors (as determined by directfine-to-coarse grid comparisons) in allcases: wave strength, wave width, andwave position errors. This was not thecase for the under-resolved simula-tion. Although the composition lawgave good results for wave strengthand wave width errors, it gave poorresults for wave position errors. Thenature of these results can be under-stood in terms of a breakdown insome of the modeling assumptionsused in the analysis.

An interesting point of contrastemerged between the planar andspherical cases. For the planar case,the dominant source of error was frominitial uncertainty, while for the spher-ical symmetry case, the dominantsource of error arose in the simulationitself, and especially from shock


Error Analysis

Panel (a) shows the space-time 100-mesh-point density field for asingle realization from the flow ensemble.The space-time errorfield for each realization is computed from the difference betweena 100-mesh-zone calculation and a fiducial solution computed

using 1000 mesh zones. Panels (b) and (c) show the mean andvariance, respectively, over the ensemble as a function ofspace and time. Note that most errors are generated at thewave interactions and then move with the wave fronts.

Figure 15. Space-Time Error Statistics for Shock-Tube Refraction Problems

reflections off the center of symmetry.We come now to the “so what?”

question for error models. What arethey good for? Our analysis showsthat, with an error model, one candetermine the relative importance ofinput and solution errors (therebyallocating resources effectively totheir reduction), as well as the precisesource of the solution error (for thesame purpose), and, finally, one canassess the error in a far more efficientmanner than by direct comparisonwith a highly refined computation ofthe full problem.

A significant limitation in ourresults to date is that they pertainmostly to 1-D flows, namely, to flowshaving planar, cylindrical, or spheri-cal symmetry. Two-dimensional prob-lems are currently under study, whilefull 3-D problems are to be solved inthe future. Furthermore, errors insome important fluid flows lie outsidethe framework we have developed,and their analysis will require newideas. One such problem—fluid mix-ing—was discussed in the previoussubsection.

Conclusions

This paper started from the prem-ise that predictive simulations of com-plex phenomena will increasingly becalled upon to support high-conse-quence decisions, for which confi-dence in the answer is essential. Manyfactors limit the accuracy of simula-tions of complex phenomena, one ofthe most important being the sparsityof relevant, high-quality data. Otherfactors include incomplete or insuffi-ciently accurate models, inaccuratesolutions of the governing equationsin the model, and the need to integratethe diverse and numerous componentsof a complex simulation into a coher-ent whole. Error analysis by itselfdoes not circumvent these limitations.It is a way to estimate the level of

confidence that can be placed in asimulation-based prediction on thebasis of a careful analysis of thesource and size of errors affecting thisprediction. Thus, the metric of successof an error analysis is the confidenceit gives that the errors are of a specificsize—not necessarily that they aresmall (they might not be).

We have reviewed some of theideas and methods that are used in thestudy of simulation errors and havepresented three examples illustratinghow these methods can be used. Theexamples show how an improvedphysics model can dramaticallyreduce the size of errors, how animproved error model can reduceuncertainty in prediction of future oilproduction, and how an error modelfor a complex shock-wave problemcan be built up from an error analysisof its components.

Similar to models of natural phe-nomena, error models will never beperfect. Estimates of errors and uncer-tainties are always provisionalbecause the data supporting these esti-mates are derived from a limitedrange of experience. Certainty is notin the picture. Nevertheless, confi-dence in predictions can be derivedfrom the scope and power of the theo-ry and solution methods that are beingused. Scope refers to the number andvariety of cases in which a theory hasbeen tested. Scope is important inbuilding confidence that one has iden-tified the factors limiting the applica-bility of the theory. Power is judgedby comparing what is put into thesimulation with what comes out.

Error models contribute to confi-dence by clarifying what we do anddo not understand. They also guideefforts to improve our understandingby focusing on factors that are theleading sources of error. Thus, in pre-dictions of complex phenomena, anerror analysis will form an indispensa-ble part of the answer. n


Error Analysis

Further Reading

Courant, R., and K. O. Friedrichs. 1967.Supersonic Flow and Shock Waves. NewYork: Springer-Verlag.

Dutta, S., E. George, J. Glimm, J. W. Grove, H.Jin, T. Lee, et al. 2004. Shock WaveInteractions in Spherical and PerturbedSpherical Geometries, Los Alamos NationalLaboratory document LA-UR-04-2989.(Submitted to Nonlinear Anal.).

Gaver, D. P. 1992. CombiningInformation: Statistical Issues andOpportunities for Research Report by Panelon Statistical Issues and Opportunities forResearch in the Combination ofInformation, Committee on Applied andTheoretical Statistics, National ResearchCouncil. Washington, DC: NationalAcademic Press.

Gilovich, T., D. Griffin, and D. Kahneman, eds.2002. Heuristics and Biases: ThePsychology of Intuitive Judgment.Cambridge, UK: Cambridge UniversityPress.

Glimm, J., J. W. Grove, Y. Kang, T. W. Lee, X.Li, D. H. Sharp, et al. 2003. StatisticalRiemann Problems and a Composition Lawfor Errors in Numerical Solutions of ShockPhysics Problems, Los Alamos NationalLaboratory document LA-UR-03-2921.SIAM J. Sci. Comput. (in press).

Glimm, J., C. Klingenberg, O. McBryan, B.Plohr, S. Yaniv, and D. H. Sharp. 1985.Front Tracking and Two-DimensionalRiemann Problems. Adv. Appl. Math. 6:259.

Johnson, V. E., T. L. Graves, M. S. Hamada,and C. S. Reese. 2003. A HierarchicalModel for Estimating the Reliability ofComplex Systems. In Bayesian Statistics 7:Proceedings of the Seventh ValenciaInternational Meeting. p. 199. UK: OxfordUniversity Press.

Kao, J., D. Flicker, R. Henninger, S. Frey, M.Ghil, and K. Ide. 2004. Data Assimilationwith an extended Kalman Filter for Impact-Produced Shock-Wave Dynamics. J. Comp.Phys. 196 (2): 705.

O’Nions, K., R. Pitman, and C. Marsh. 2002.The Science of Nuclear Warheads. Nature415: 853.

O'Sullivan, A. E. Modelling Simulation Errorfor Improved Reservoir Prediction Ph.D.thesis, Heriot-Watt University, Edinburgh.

Palmer, T. N. 2000. Predicting Uncertainty inForecasts of Weather and Climate. Rep.Prog. Phys. 63: 71.

Sharp, D. H., and M. Wood-Schulz. 2003.QMU and Nuclear Weapons Certification.Los Alamos Science 28: 47.


Error Analysis

For further information, contact David H. Sharp (505) 667-5266([email protected]).

Documents

Error Analysis and Simulations of Complex Phenomenafei/samsi/Readings/DHigdon/lascience.pdfReliable Predictions of Complex Phenomena There is an increasing demand for reliable predictions