Embed Size (px)
Citation preview
Department of Civil, Construction, and Environmental EngineeringNorth Carolina State UniversityRaleigh, NC 27695
Prepared for:
Workshop on Climate ChangeWashington, DC
March 7, 2005
H. Christopher Frey, Ph.D.Professor
Methods and Applications of Uncertainty and Sensitivity Analysis
Outline
• Why are uncertainty and sensitivity analysis needed?• Overview of methods for uncertainty analysis
– Model inputs» Empirical data» Expert judgment
– Model uncertainty– Scenario uncertainty
• Overview of methods for sensitivity analysis• Examples
– Technology assessment– Emissions Factors and Inventories – Air Quality Modeling – Risk Assessment
• Findings• Recommendations
Why are uncertainty and sensitivity analysis needed?
• Strategies for answering this question:–what happens when we ignore uncertainty and
sensitivity?–what do decision makers want to know that
motivates doing uncertainty and sensitivity analysis?
–what constitutes best scientific practice?
• Program and research managers may not care about all three, but might find at least one to be convincing (and useful)
When is ProbabilisticAnalysis Needed or Useful?
• Consequences of poor or biased estimates are unacceptably high
• A (usually conservative) screening level analysis indicates a potential concern, but carries a level of uncertainty
• Determining the value of collecting additional information
• Uncertainty stems from multiple sources
• Significant equity issues are associated with variability
• Ranking or prioritizing significance of multiple pathways, pollutants, sites, etc.
• Cost of remediation or intervention is high
• Scientific credibility is important
• Obligation to indicate what is known and how well it is known
When is a Probabilistic Approach Not Needed?
• When a (usually conservative) screening level analysis indicates a negligible problem
• When the cost of intervention is smaller than the cost of analysis
• When safety is an urgent and/or obvious issue• When there is little variability or uncertainty
Myths: Barriers to Use of Methods
• Myth: it takes more resources to do uncertainty analysis, we have deadlines, we don’t know what to do with it, let’s just go with what we have…
• Hypothesis 1: poorly informed decisions based upon misleading deterministic/point estimates can be very costly, leading to a longer term and larger resource allocation to correct mistakes that could have been avoided or to find better solutions
• Hypothesis 2: Uncertainty analysis helps to determine when a robust decision can be made versus when more information is needed first
• Hypothesis 3: Uncertainty and sensitivity analysis help identify key weaknesses and focus limited resources to help improve estimates
• Hypothesis 4: Doing uncertainty analysis actually reduces overall resource requirements, especially if it is integrated into the process of model development and applications
Role of Modeling in Decision-Making
• Modeling should provide insight
• Modeling should help inform a decision
• Modeling should be in response to clearly defined objectives that are relevant to a decision.
Questions that Decision-Makers and Stakeholders Typically Ask
• How well do we know these numbers? – What is the precision of the estimates?– Is there a systematic error (bias) in the estimates? – Are the estimates based upon measurements, modeling,
or expert judgment?
• How significant are differences between two alternatives? • How significant are apparent trends over time? • How effective are proposed control or management
strategies? • What is the key source of uncertainty in these numbers? • How can uncertainty be reduced?
Application of Uncertainty to Decision Making
• Risk preference –Risk averse–Risk neutral–Risk seeking
• Utility theory
• Benefits of quantifying uncertainty: Expected Value of Including Uncertainty
• Benefits of reducing uncertainty: Expected Value of Perfect Information (and others)
Variability and Uncertainty
• Variability: refers to the certainty that –different members of a population will have
different values (inter-individual variability) –values will vary over time for a given member
of the population (intra-individual variability)• Uncertainty: refers to lack of knowledge
regarding–True value of a fixed but unknown quantity–True population distribution for variability
• Both depend on averaging time
Variability and Uncertainty
• Sources of Variability–Stochasticity–Periodicity, seasonality –Mixtures of subpopulations–Variation that could be explained with better
models–Variation that could be reduced through control
measures
Variability and Uncertainty
• Sources of Uncertainty:– Random sampling error for a random sample of data– Measurement errors
» Systematic error (bias, lack of accuracy)» Random error (imprecision)
– Non-representativeness» Not a random sample, leading to bias in mean (e.g., only
measured loads not typical of daily operations)» Direct monitoring versus infrequent sampling versus
estimation, averaging time» Omissions
– Surrogate data (analogies with similar sources)– Lack of relevant data– Problem and scenario specification– Modeling
Overview of “State of the Science”
• Statistical Methods Based Upon Empirical Data
• Statistical Methods Based Upon Judgment
• Other Quantitative Methods
• Qualitative Methods
• Sensitivity Analysis
• Scenario Uncertainty
• Model Uncertainty
• Communication
• Decision Analysis
Statistical MethodsBased Upon Empirical Data
• Frequentist, classical
• Statistical inference from sample data–Parametric approaches
»Parameter estimation
»Goodness-of-fit
–Nonparametric approaches–Mixture distributions–Censored data–Dependencies, correlations, deconvolution
Statistical MethodsBased Upon Empirical Data
• Variability and Uncertainty– Sampling distributions for parameters– Analytical solutions– Bootstrap simulation
Propagating Variability and Uncertainty
– Analytical techniques» Exact solutions (limited applicability)» Approximate solutions
– Numerical methods» Monte Carlo » Latin Hypercube Sampling» Other sampling methods (e.g., Hammersley,
Importance, stochastic response surface method, Fourier Amplitude Sensitivity Test, Sobol’s method, Quasi-Monte Carlo methods, etc.)
Monte Carlo Simulation
• Probabilistic approaches are widely used
• Monte Carlo (and similar types of) simulation are widely used.
• Why?–Extremely flexible
» Inputs
»Models
–Relatively straightforward to conceptualize
Tiered Approach to Analysis
• Purpose of Analyses (examples)– Screening to prioritize resources– Regulatory decision-making– Research planning
• Types of Analyses– Screening level point-estimates– Sensitivity Analysis– One-Dimensional Probabilistic Analysis– Two-Dimensional Probabilistic Analysis– Non-probabilistic approaches
MethodsBased Upon Expert Judgment
• Expert Elicitation– Heuristics and Biases
» Availability» Anchoring and Adjustment» Representativeness» Others (e.g., Motivational, Expert, etc.)
– Elicitation Protocols» Motivating the expert» Structuring» Conditioning» Encoding» Verification
– Documentation– Individuals and Groups– When Experts Diasagree
An Example of Elicitation Protocols:Stanford/SRI Protocol
Motivating (Establish Rapport)
Structuring (Identify Variables)
Conditioning (Get Expert to Think About Evidence)
Encoding (Quantify Judgment About Uncertainty)
Verify (Test the Judgment)
Key Ongoing Challenges
• Expert Judgment vs. Data– Perception that judgment is more biased than analysis
of available data– Unless data are exactly representative, they too could
be biased– Statistical methods are “objective” in that the results
can be reproduced by others, but this does not guarantee absence of bias
– A key area for moving forward is to agree on conditions under which expert judgment is an acceptable basis for subjective probability distributions, even for rulemaking situations
Appropriate Use of Expert Judgment in Regulatory Decision Making
• There are examples…e.g.,– analysis of health effects for EPA standards– Uncertainty in benefit/cost analysis (EPA, OMB)– Probabilistic risk analysis of nuclear facilities
• Key components of credible use of expert judgment:– Follow a clear and appropriate protocol for selecting experts and
for elicitation– For the conditioning step, consider obtaining input via workshop,
but for encoding, work individually with experts – preferably at their location
– Document (explain) the basis for each judgment– Compare judgments: identify key similarities and differences– Evaluate the implications of apparent differences with respect to
decision objectives – do not “combine” judgments without first doing this
– Where possible, allow for iteration
Statistical MethodsBased Upon Expert Judgment
• Bayesian methods can incorporate expert judgment– Prior distribution
– Update with data using likelihood function and Bayes’ Theorem
– Create a posterior distribution
• Bayesian methods can also deal with various complex situations:
– Conditional probabilities (dependencies)
– Combining information from multiple sources
• Appears to be very flexible• Computationally, can be very complex• Complexity is a barrier to more widespread use
Other Quantitative Methods
• Interval Methods–Simple intervals–Probability bounds–Produce “optimally” narrow bounds – cannot be
any narrower and still enclose all possible outcomes, including dependencies among inputs
–Bounds can be very wide in comparison to confidence intervals
Other Quantitative Methods
• Fuzzy methods– Representation of vagueness, rather than uncertainty– Approximate/semi-quantitative– Has been applied in many fields
• Meta-analysis– Quantitatively combine, synthesize, and summarize data and
results from different sources– Requires assessment of homogeneity among studies prior to
combining– Produces data with larger sample sizes than the constituent inputs– Can be applied to summary data– If raw data are available, other methods may be preferred
Scenario Uncertainty
• A need for formal methods• Creativity, brainstorming, imagination• Key dimensions (e.g., human exposure assessment)
– Pollutants– Transport pathways– Exposure routes– Susceptible populations– Averaging time– Geographic extent– Time Periods– Activity Patterns
• Which dimensions/combinations matter, which ones don’t?• Uncertainty associated with mis-specification of a scenario –
systematic error• Scenario definition should be considered when developing and
applying models
Model Uncertainty
• Model Boundaries (related to scenario)
• Simplifications–Aggregation–Exclusion
• Resolution• Structure• Calibration• Validation, Partial validation• Extrapolation
Model Uncertainty
• Methods for Dealing with Model Uncertainty–Compare alternative models, but do not
combine–Weight predictions of alternative models (e.g.,
probability trees)–Meta-models that degenerate into alternative
models (e.g., Y = a(|x-t|)n , where n determines linear/nonlinear and t determines threshold or not)
Weighting vs. Averaging
Model A Model B
Output of Interest
Output of Interest
Pro
babili
ty D
ensi
tyPro
babili
ty D
ensi
ty
Each Model has Equal Weight
Average ofBoth Models Neither Model
Supports ThisRange of Outcomes
Sensitivity Analysis
• Objectives of Sensitivity Analysis (examples):– Help identify key sources of variability (to aid management
strategy)» Critical control points?» Critical limits?
– Help identify key sources of uncertainty (to prioritize additional data collection to reduce uncertainty)
– What causes worst/best outcomes?– Evaluate model behavior to assist verification/validation– To assist in process of model development
• Local vs. Global Sensitivity Analysis• Model Dependent vs. Model Independent Sensitivity Analysis• Applicability of methods often depends upon characteristics of a
model (e.g., nonlinear, thresholds, categorical inputs, etc.)
Examples of Sensitivity Analysis Methods
• Mathematical Methods
Assess sensitivity of a model output to the range of variation of an input.
• Statistical Methods
Effect of variance in inputs on the output distribution.
• Graphical Methods
Representation of sensitivity in the form of graphs, charts, or surfaces.
Nominal Range Sensitivity Analysis (NRSA)
Differential Sensitivity Analysis
Regression Analysis (RA)
Analysis of Variance (ANOVA)
Classification and RegressionTrees (CART)
Scatter Plots
Conditional Sensitivity
Sensitivity Analysis Methods (Examples)
• Nominal Range Sensitivity Analysis• Differential Sensitivity Analysis• Conditional Analysis• Correlation coefficients (sample, rank)• Linear regression (sample, rank, variety of basis functions possible)• Other regression methods• Analysis of Variance (ANOVA)• Categorical and Regression Trees (CART) (a.k.a. Hierarchical Tree-
Based Regression)• Sobol’s method• Fourier Amplitude Sensitivity Test (FAST)• Mutual Information Index• Scatter Plots
Sensitivity Analysis: Displays/Summaries
• Scatter plots
• Line plots/conditional analyses
• Radar plots
• Distributions (for uncertainty or variability in sensitivity)
• Summary statistics
• Categorical and regression trees
• Apportionment of variance
Guidance on Sensitivity Analysis
Guidance for Practitioners, with a focus on food safety process risk models (Frey et al., 2004):
• When to perform sensitivity analysis• Information needed depending upon objectives• Preparation of existing or new models• Defining the case study/scenarios• Selection of sensitivity analysis methods• Procedures for application of methods• Presentation and interpretion of results
Summary of Evaluation Results for Selected Sensitivity Analysis Methods
Example of Guidance on Selection of Sensitivity Analysis Methods
Source: Frey et al., 2004, www.ce.ncsu.edu/risk/
Example of Guidance on Selection of Sensitivity Analysis Methods
Communication
• Case Studies (scenarios)
• Graphical Methods–Influence Diagrams–Decision Tree–Others
• Summary statistics/data
• Evaluation of effectiveness of methods for communication (e.g., Bloom et al., 1993; Ibrekk and Morgan, 1987)
Example Case Studies
• Technology Assessment
• Emission Factors and Inventories
• Air Quality Modeling
• Risk Assessment
Role of Technology Assessment in Regulatory Processes (Examples)
• Assessment of ability of technology to achieve desired regulatory or policy goals (emissions control, safety, efficiency, etc.)
• Evaluation of regulatory alternatives (e.g., based on model cost estimates)
• Regulatory Impact Analysis – assessment of costs
An Example of Federal Decision Making:Process Technology RD&D
Research and Development
Cost High
No Yes
Cost Reasonable
Yes
Yes
No
No
Yes
R&D
Demonstration Plant
Yes No
Commercialization
Screening R&D Cost Estimate
Rework Cost Estimate?
Unresolved Technical
Uncertainties?
Reject?
Commit $ for Scope
Def.
No
Good Results?
Project Definition
R&D or
Pilot Plant?
A Probabilistic Framework for FederalProcess Technology Decision-Making
IDENTIFY DECISION CRITERIA
INITIAL SCREENING
NO, Reject
YES
DECISION ANALYSIS • DESIGN TRADE-OFFS • TECHNOLOGY SELECTION • RESEARCH STRATEGIES
SECOND SCREENING
YES, commit further resources to R&DNO, reject
POLICY CONCERNS
SET OF TECHNOLOGIES
PROBABILISTIC ANALYSESCharacterize Perfermance,
Emissions, and Cost
Uncertainties
Comparative Risk
Assessment
Prioritization of
Uncertainties
Probabilistic Design
Analysis
ENGINEERING MODELS
Design Studies
Test Data
ExpertsCHARACTERIZE UNCERTAINTIES
Methodology for Probabilistic Technology Assessment
• Process simulation of process technologies in probabilistic frameworks
– Integrated Environmental Control Model (IECM) and derivatives
– Probabilistic capability for ASPEN chemical process simulator
• Quantification of uncertainty in model inputs– Statistical analysis– Elicitation of expert judgment
• Monte Carlo simulation• Statistical methods for sensitivity analysis• Decision tree approach to comparing technologies and
evaluating benefits of additional research
Conceptual Diagram of Probabilistic Modeling
PerformanceInputs
InputUncertainties
OutputUncertainties
Engineering Performanceand Cost Model of a
New Process Technology
Raw water
Coal
Exhaust Gas
CycloneCyclone
Blowdown
CycloneCyclone
Boiler Feedwater
Return Water
Gasifier Steam
Gas Turbine ExhaustShift &
RegenerationSteam
RawSyngas
Coal
CleanSyngas
Steam
Gasification,Gasification,Particulate &Particulate &Ash Removal,Ash Removal,Fines RecycleFines Recycle
Ash Fines Fines
Gasifier AirAsh
Tailgas Sulfuric Acid Air Air Electricity
CoalCoalHandlingHandling
BoilerBoilerFeedwaterFeedwaterTreatmentTreatment
HRSGHRSG& Steam& Steam
CycleCycle
Hot GasHot GasDesulfur-Desulfur-
izationization
GasGasTurbineTurbine
SteamSteamTurbineTurbine
SulfuricSulfuricAcidAcidPlantPlant
Cost Inputs Cost
Performance
Emissions
Comparison of Probabilistic and Point-Estimate Results for an IGCC System
0
0.2
0.4
0.6
0.8
1
1600 1700 1800 1900 2000
Total Capital Requirement (1998 $/kW)
Cum
ulat
ive
Prob
abili
ty
All Uncertainties
Point Estimate
Key Uncertainties
0
0.2
0.4
0.6
0.8
1
47 48 49 50 51 52 53 54 55 56 57 58
Levelized Cost of Electricity, (1998 mills/kWh)
Cum
ulat
ive
Pro
babi
lity
All Uncertainties
Point Estimate
Key Uncertainties
Example of a Probabilistic Comparison of Technology Options
0
0.2
0.4
0.6
0.8
1
-7.5 -7.0 -6.5 -6.0 -5.5 -5.0
COE Difference of IGCC-7H and IGCC-7FA, miils/kWh
Probabilistic
Deterministic
Cum
ulat
ive
Pro
babi
lity
Uncertainty in the difference in cost between two technologies,
taking into account correlations between them
Example: Engineering Study ofCoal-Gasification Systems
• DOE/METC Engineers
• Briefing Packets:
- Part 1: Uncertainty Analysis (9 pages)- Part 2: Process Area Technical Background
- Lurgi Gasifier: 12 p., 16 ref.- KRW Gasifier: 19 p., 25 ref.- Desulfurization: 9 p., 19 ref.- Gas Turbine: 23 p., 36 ref.
- Part 3: Questionnaire
• Follow-Up
Examples of the Judgments
of One Expert• Fines Carryover
• Air/Coal Ratio
• Carbon Retention
3025201510500.0
0.1
0.2
Expert LG-1's Judgment Regarding Fines Carryover
Fin es Carry o v er, % o f co al feed
Prob
ability
Den
sity
201510500.0
0.1
0.2
0.3
Expert LG-1's Judgment Regarding Carbon Retention
Carb o n Reten tio n in Bo tto m A sh , % o f co al feed carb o n
Prob
ability
Den
sity
432100
1
2
3
Expert LG-1's Judgment Regarding Gasifier Air/Coal Ratio
A ir-to -Co al Ratio , lb air /lb D A F co al
Probab
ility De
nsity
Examples of the Judgments
of Multiple Experts
353025201510500 .0
0 .1
0 .2
0 .3
Ex pert ZF-1 's Judg men t Reg ard in g Sorben t Su lfu r Load ingSorbent Sulfur Loading, wt-%
Proba
bility
Den
sity
3 53 02 52 01 51 0500 .0 0
0 .0 2
0 .0 4
0 .0 6
0 .0 8
0 .1 0
0 .1 2
Ex p ert ZF-2 's Ju d g men t Reg ard in g So rb en t Su lfu r Lo ad in gSorbent Sulfur Loading, wt-%
Proba
bility
Den
sity
3 53 02 52 01 51 0500 .0 0
0 .0 2
0 .0 4
0 .0 6
0 .0 8
0 .1 0
Ex p ert ZF-3 's Ju d g men t Reg ard in g So rb en t Su lfu r Lo ad in gSorbent Sulfur Loading, wt-%
Proba
bility
Den
sity
2 52 01 51 0500 .0
0 .5
1 .0
1 .5
Ju d g men t o f Ex p ert ZF-1 Reg ard in g So rb en t A ttritio nSorbent R eplacement R ate , wt-%/cycle
Proba
bility
Den
sity
2 52 01 51 0500 .0
0 .1
0 .2
0 .3
0 .4
0 .5
0 .6
Ju d g men t o f Ex p ert ZF-2 Reg ard in g So rb en t LifeSorbent R eplacement R ate , wt-%/cycle
Proba
bility
Den
sity
2 52 01 51 0500
2
4
6
8
Ju d g men t o f Ex p ert ZF-3 Reg ard in g So rb en t Attritio nSorbent R eplacement R ate , wt-%/cycle
Proba
bility
Den
sity
Z F - 1
Z F - 2
Z F - 3
S o r b e n t R e p l a c e m e n t S o r b e n t L o a d i n gE x p e r t
Do Different Judgments Really Matter?
• Specific Sources of Disagreement: - Sorbent Loading - Sorbent Attrition
• Qualitative Agreement in Several Cases
• Specific Sources of Disagreement: - Sorbent Loading - Sorbent Attrition
• Qualitative Agreement in Several Cases
1201101009080706050400.0
0.2
0.4
0.6
0.8
1.0
Judgments LG-1 and ZF-1
Judgments LG-1 and ZF-2P
Judgments LG-1 and ZF-2R
Judgments LG-1 and ZF-3
Data from "Data ALH LG-1,ZF-1 research"
Levelized Cost of Electricity, Constant 1989 Mills/kWh
Cum
ulat
ive
Pro
babi
lity
Input Uncertainty Assumptions
Technology Assessment:Findings (1)
• Interactions among uncertain inputs, and nonlinearities in model, contribute to positive skewness in model output uncertainties
– Uncertainties in inputs are often positively skewed (physical, non-negative quantities)
– The mean value of a probabilistic estimate is often “worse” (lower performance, higher cost) than the “best guess” deterministic estimate, and the probability of “worse” outcomes is typically greater than 50 percent.
– A system approach is needed to account for interactions among process areas
– Deterministic analysis leads to apparent “cost growth” and “performance shortfall” because it does not account for simultaneous interactions among positively skewed inputs
– Uncertainty analysis requires more thought pertaining to developing input assumptions, but provides more insight into potential sources of “cost growth” and “performance shortfall”
Technology Assessment:Findings (2)
• A decision model provides a framework for evaluating judgments regarding the outcomes of additional research and prioritizing additional research
– Able to quantify the probability of “pay-offs” as well as downside risks
– Able to compare competing options under uncertainty and identify robust choices
– Trade-offs when comparing technologies can be evaluated probabilistically (e.g., efficiency, emissions, and cost)
• It is possible to combine approaches for quantifying uncertainty in one assessment, consistent with objectives
Technology Assessment:Findings (3)
• Thinking about uncertainties leads to better understanding of what matters most in the assessment
– Often, only a relatively small number of inputs contribute substantially to uncertainty in a model output
– Reducing uncertainty in only a few key inputs can substantially reduce downside risk and increase the pay-offs of new technology
– Conversely, for those inputs to which the output is not sensitive, it is not critical to devote resources to refinement
• When basing inputs on expert judgments, only those disagreements that really matter to the decision need become the focus of further discussion and evaluation
• Bottom Line: Probabilistic analysis helps improve decisions and avoid unpleasant “surprises”.
Emission Factors and Inventories
• Significance to Regulatory Processes:– Assessment of capability of technology to reduce/prevent
emissions– Evaluation of regulatory alternatives– Regulatory Impact Analysis – including benefit/cost analysis– Component of air quality management at various temporal and
spatial scales– Component of human and ecological exposure and risk
assessment• Modeling Aspects
– Some emission factors are estimated using models– Emission Inventories are linear models– Specialized models for some emission factors and inventories
(e.g., Mobile6, NONROAD, MOVES)
Motivations for Probabilistic Emission Factors and Inventories
• How good are the estimates?• What are the key sources of uncertainty in the estimates that
should be targeted for improvement?• Likelihood of meeting an emissions budget?• Which emission sources are the most significant?• What is the inter-unit variability in emissions?• What is the uncertainty in mean emissions for a group/fleet of
sources?• What are the implications of uncertainty in emissions for air quality
management, risk management of human exposures?• Consideration of geographic extent and averaging time• Estimation for future scenarios versus retrospective estimates of
past emissions or assessment of current emissions
Motivations for Probabilistic Analysis
• “That a perfect assessment of uncertainty cannot be done, however, should not stop researchers from estimating the uncertainties that can be addressed quantitatively” (p. 150, NRC, 2000)
• “EPA, along with other agencies and industries, should undertake the necessary measures to conduct quantitative uncertainty analyses of the mobile-source emissions models in the modeling toolkit.” (p. 166, NRC, 2000)
Current Practice for Qualifying Uncertainty
in Emission Factors and Inventories• Qualitative ratings for emission factors (AP-42)
• Data Attribute Rating System (DARS) (not really used in practice)
• Both methods are qualitative
• No quantitative interpretation
• Some sources of uncertainty (i.e. non-representativeness) difficult to quantify
• Qualitative methods can complement quantitative methods
• Compilation and evaluation of database
• Visualization of data by developing empirical cumulative distribution functions
• Fitting, evaluation, and selection of alternative probability distribution models
• Characterization of uncertainty in the distributions for variability (e.g., uncertainty in the mean)
• Propagation of uncertainty in activity and emissions factors to estimate uncertainty in total emissions
• Calculation of importance of uncertainty
Statistical Methodological Approach
Summary of Approaches to Emission Factor and Inventory Uncertainty
• Probabilistic Methods– Empirical, Parametric– Mixture distributions– Censored distributions (non-detects)– Vector autoregressive time series (intra- and inter-unit correlation)– Bootstrap simulation– Expert Judgment– Monte Carlo simulation– Sensitivity analysis
• Software tools:– AUVEE – Analysis of Uncertainty and Variability in Emissions
Estimation– AuvTool – standalone software
Summary of Probabilistic Emissions Case Studies at NCSU
• Case Studies (examples):– Point sources
» Power Plants» Natural gas-fired engines (e.g., compressor stations)
– Mobile sources» On-Road Highway Vehicles» Non-Road Vehicles (e.g., Lawn & Garden, Construction, Farm, & Industrial)
– Area sources» Consumer/Commercial Product Use» Natural Gas-Fueled Internal Combustion Engines» Gasoline Terminal Loading Loss» Cutback Asphalt Paving» Architectural Coatings» Wood Furniture Coatings
• Pollutants– NOx
– VOC– Urban air toxics (e.g., Houston case study)
Example Results: Lawn & Garden Equipment
Engine Type
Pollutant# of
Data
Fitted
Distribution
Mean
(g/hp-hr)
Relative Uncertainty
2-Stroke
NOx 16 Lognormal 0.81 -46% to +65%
THC 16 Lognormal 222 -32% to +38%
4-Stroke
NOx 27 Lognormal 2.05 -25% to +38%
THC 22 Lognormal 21.5 -38% to +45%
Based on Frey and Bammi (2002)
Probabilistic CO Emission Factors for On-Road Light Duty Gasoline Vehicles (Mobile5)
Standard Temperature and Reid Vapor Pressure
Adjusted for Temperature and Reid Vapor Pressure
Driving Cycle
Speed (mph) Mean System-
atic Error Random ( - ) %
Error ( + ) %
Mean System-atic Error
Random ( - ) %
Error ( + ) %
LSP1 2.45 36.9 23.6 -59 62 148 75 -90 282
LSP2 3.64 38.9 0.92 -70 66 154 -5 -91 279
LSP3 4.02 50.9 -16.4 -62 65 222 -88 -91 227
NYCC 7.1 33.1 -11.8 -25 23 130 -57 -88 256
SCC12 12.1 14.3 -2.89 -28 27 56.8 -16 -89 229
FTP BAG2
16.1 8.78 -0.71 -10 10 34.7 -6 -87 224
SCC36 35.9 6.65 -0.33 -20 23 26.3 -3 -88 236
HFET 48.4 4.16 0.21 -21 22 16.5 -1 -88 246
HSP1 50.9 4.50 -38 43 18.2 -88 262
HSP2 57.6 0.26 -42 43 1.03 -88 236
HSP3 64.3 0.27 -57 57 1.04 -90 272
Based on Frey and Zheng (2002)
MOVES
• Conceptual Basis for MOVES, the successor to Mobile6 and NONROAD (www.epa.gov/otaq/ngm.htm)
– “Shootout”– NCSU report on modal/binning approach
• NCSU recommended approaches for quantification of inter-vehicle variability and fleet average uncertainty in modal emission rates and estimates of emissions for driving cycles (details in our report to EPA)
• EPA requested further assessment of an approximate analytical procedure for propagating error (report by Frey to EPA)
• At last report, EPA was considering Monte Carlo simulation
Probabilistic AP-42 Emission Factors for Natural Gas-fueled Engines (July 2000 Version)
Engine and Load Range
AP-42 Emission Factora
No. of Data /
Engines
Fitted Distri-
bution.b
Mean of Bootstrap
sample meansa
Relative 95% CI of meanc (%)
Uncontrolled NOx
2SLB, 90-105% 3.17 34 / 11 W 3.05 -24 to +24
2SLB, <90% 1.94 24 / 11 W 2.18 -41 to +46
4SLB, 90-105% 4.08 12 / 4 G 4.06 -39 to +49
4SLB, <90% 0.847 13 / 5 W 1.81 -80 to +180
Uncontrolled TOC
2SLB, all load 1.64 57 / 14 W 1.45 -16 to +18
4SLB, all load 1.47 37 / 4 G 1.12 -45 to +57aUnits are lb/106 BTU. bMLE is used for 2SLB engine, MoMM is used for 4SLB engine, W=Weibull distribution, G=Gamma distribution.cCalculated based upon bootstrap simulation results.
Based on Frey and Li (2003) (submitted)
Summary of Probabilistic Emission Inventories for Selected Air Toxics
City Pollutant 95% C.I. (%, %)
Houston
Benzene (-46, 108)
Formaldehyde (-35, 67)
Chromium (-20, 34)
Arsenic (-69, 203)
Jacksonville
1,3-butadiene (-46, 108)
Mercury (-25, 30)
Arsenic (-83, 243)
Benzene (-56, 146)
Formaldehyde (-42, 89)
Lead (-54, 175)
Key Sources of Uncertainty
City PollutantKey Sources of Uncertainty (number of dominant
sources)
Houston
Benzene Gasoline onroad mobile sources (1)
Formaldehyde Onroad mobile sources; Nonroad mobile sources (2)
Chromium
Chemical manufacturing-fuel fired heaters (1)
External utility coal combustion boilers; Hard chromium electroplating (2)
Arsenic External coal combustion utility boilers (1)
Jacksonville
1, 3-butadiene Onroad mobile sources (1)
Mercury External coal combustion boilers (1)
Arsenic External coal combustion boilers (1)
Benzene Onroad mobile source (1)
Formaldehyde Onroad mobile source; Aircraft (2)
LeadExternal coal combustion boilers; Fuel oil external
combustion; Waste oil external combustion (3)
Example of Benzene Emission Factor Category 3b: Nonwinter Storage Losses at a Bulk Terminal : Empirical
Distribution
0
0.2
0.4
0.6
0.8
1
0.001 0.01 0.1 1
Benzene Emission Factor(ton/yr/tank)
Cu
mu
lativ
e P
rob
ab
ility
Example of Benzene Emission Factor Category 3b: Fitted Lognormal Distribution
0
0.2
0.4
0.6
0.8
1
0.001 0.01 0.1 1
Benzene Emission Factor(ton/yr/tank)
Cu
mu
lativ
e P
rob
ab
ility
Example of Benzene Emission Factor Category 3b: Confidence Interval in the CDF
90 percent
0.0
0.2
0.4
0.6
0.8
1.0
Cum
ulat
ive
Pro
babi
lity
10-3
10-2
10-1
100
Benzene Emission Factor (ton/yr/tank)
50 percent90 percent95 percent
Data Set
Confidence IntervalFitted Distribution
Example of Benzene Emission Factor Category 3b: Uncertainty in the Mean
0
0.2
0.4
0.6
0.8
1
0 0.05 0.1 0.15 0.2
Benzene Emission Factor(ton/yr/tank)
Cu
mu
lativ
e P
rob
ab
ility
mean = 0.6
95% Probability Range (0.016, 0.18)
Uncertainty in mean -73% to +200%
0.06
Using AuvTool to Fit a Distribution for Variability
Using AuvTool for Bootstrap Simulation
Using AuvTool to Quantify Uncertainty in the Mean
Uncertainty in Total Emission Inventory:AUVEE Prototype Software
Summary ofProbabilistic Emission Inventory
Identification of Key Sources of Uncertaintyin an Inventory
Detection Limits and Air Toxic Emission Factor Data
• Many air toxic emission factor data contain one or more measurements below a “detection limit”
• Detection limits can be unique to each measurement because of differences in sample volumes and analytical chemistry methods among sites or contractors
• A database can contain some non-detected data with detection limits larger than detected values measured at other sites
Methodology: Conventional Methods for Censored Data
• Conventional approaches to estimate the mean: -Remove non detected values (biased)
-Replace values below DL with zero (underestimate)
-Replace values below DL with DL/2 (biased)
-Replace values below DL with DL (overestimate)
• Cause biased estimates of the mean• Does not provide adequate insights regarding
–Population distribution
–Unbiased statistics
–Uncertainty in statistics
Methodology: Quantification of the Inter-Unit Variability in Censored Data
• Maximum Likelihood Estimation (MLE) is used to fit parametric distributions to censored data
• MLE is asymptotically unbiased
• Fitted distribution is the best estimate of variability
• Can estimate mean and other statistics from the fitted distribution
• Can quantify uncertainty caused by random sampling error using Bootstrap simulation
Results: Fitted Lognormal Distribution, No Censoring
0 1 2 3 4 5 6
Value of Random Variable
0.00
0.20
0.40
0.60
0.80
1.00
Cum
ulat
ive
Pro
babi
lity
95 percent90 percent
Fitted DistributionData Set
Confidence Interval
50 percent
Results: Fitted Lognormal Distribution, 30% Censoring
95 percent90 percent
Fitted DistributionData Set
Confidence Interval50 percent
1.00
0.80
0.60
0.40
0.20
0.00
Cum
ulat
ive
Pro
babi
lity
0 1 2 3 4 5 6
Value of Random Variable
Data Set
95 percent90 percent
Fitted DistributionConfidence Interval
50 percent
Detection Limt
Results: Fitted Lognormal Distribution, 60% Censoring
0 1 2 3 4 5 6Value of Random Variable
1.00
0.80
0.60
0.40
0.20
0.00
Cum
ulat
ive
Pro
babi
lity
Data Set
95 percent90 percent
Fitted DistributionConfidence Interval
50 percent
Detection Limt
Example Case Study: Formaldehyde Emission Factor from External Coal Combustion
• 14 data points including 5 censored values• Each censored data point has a different
detection limit• Some detected data values are less than some
detection limits • There is uncertainty regarding the empirical
cumulative probability of such detected data values
Results of Example Case Study: Empirical Cumulative Probability
0
0.2
0.4
0.6
0.8
1
0.001 0.01 0.1 1 10 100
Formaldehyde Emission Factor (0.0001 lb pollutants/ton coal combusted)
Cum
ulat
ive
Pro
babi
lity
Results of Example Case Study: Lognormal Distribution Representing Inter-Unit Variability
0
0.2
0.4
0.6
0.8
1
0.001 0.01 0.1 1 10 100
Formaldehyde Emission Factor (0.0001 lb pollutants/ton coal combusted)
Cum
ulat
ive
Pro
babi
lity
Results of Example Case: Uncertainty in Inter-Unit Variability
95 percent
90 percent
50 percent
Confidence IntervalFitted DistributionData Set
Detection LimitPossible Plotting Position
10-3 10-2 10-1 101 102100
Formaldehyde Emission Factor(0.0001 lb pollutants/ton coal combusted)
0.0
0.2
0.4
0.6
0.8
1.0
Cum
ulat
ive
Pro
babi
lity
Results of Example Case: Uncertainty in the Mean (Basis to Develop Probabilistic Emission Inventory)
0
0.2
0.4
0.6
0.8
1
0 4 8 12 16
Mean of Formaldehyde Emission Factor(0.0001 lb pollutants/ton coal combusted)
Cum
ulat
ive
Pro
babi
lity
95 Percent Probability Range: (0.42, 5.67)
Mean =1.84
Uncertainty in mean -77% to +208%
Mixtures of Distributions
Percent of data in 50% CI: 92%
Percent of data in 95% CI: 100%
Data Set
Fitted Mixture LognormalDistribution
100 300 500 700 900 1100NOx Emission Factor (gram/GJ fuel input)
0.0
0.2
0.4
0.6
0.8
1.0
Cum
ulat
ive
Pro
babi
lity
95 percent90 percent50 percent
Confidence Interval
Case Study
• Charlotte modeling domain• 32 units from 9 different coal-fired
power plants• 1995 and 1998 data used• Propagation of uncertainty
investigated using July 12 – July 16 1995 meteorological data
• Data available for emission and activity factors
• Vector autoregressive time-series modeling of emissions from each unit
Time Series and Uncertainty
Different uncertainty ranges for different hours of day
00.05
0.10.15
0.20.25
0.30.35
12 36 60 84 108 132Hour
Em
issi
ons
(t/h
r) . Real.
Obs.
Emission Factors and Inventories:Findings (1)
• Visualization of data used to develop an inventory is highly informative to choices of empirical or parametric distribution models for quantification of variability or uncertainty
• A key difficulty in developing probabilistic emission factors inventories is to find the original data used by EPA and others.
– When data are found, they are typically poorly documented.– The time required to assemble databases when original data could not be found
was substantial
• Test methods used for some emission sources are not representative of real world operation, implying the need for real world data and/or expert judgment when estimating uncertainty
• Uncertainty in measurement methods is not adequately reported. There is a need for more systematic reporting of the precision and accuracy of measurement/test methods
• Emissions databases should not be arbitrarily fragmented into too many subcategories. Conversely, subcategories should be created when there is a good (empirical) basis for doing so.
Emission Factors and Inventories:Findings (2)
• Uncertainties in emission factors are typically positively skewed, unless the uncertainties are relatively small (e.g., less than about plus or minus 30 percent)
• Uncertainty estimates might be sensitive to the choice of parametric distribution models if there is variation in the goodness-of-fit among the alternatives. However, in such cases, there is typically a preferred best fit. When several alternative models provide equivalent fits, results are not sensitive to the choice of the model
• The quantifiable portion of uncertainty attributable to random sampling error can be large and should be accounted for when using emission factors and inventories
• Variability in emissions among units could be a basis for assessing the potential of emissions trading programs
Emission Factors and Inventories:Findings (3)
• Intra-unit dependence in hourly emissions is significant for some sources (e.g., power plants), including hourly and daily lag effects
• Inter-unit dependence in emissions is important for some sources, such as power plants
• Range of variability and uncertainty is typically much greater as the averaging time decreases
• Even for sources with continuous emissions monitoring data, there is uncertainty regarding predictions of future emissions that can be informed by analysis of historical data
• Prototype software demonstrates the feasibility of increasing the convenience of performing probabilistic analysis
• Uncertainties in total inventories are often attributable to just a few key emission sources
Mobile5 and Mobile6Findings
• Range of variability and uncertainty in correction factors (e.g., temperature) dominate and are large
• Uncertainties in average emissions are large in some cases (e.g., -80 to +220 percent): normality assumptions are not valid
• Asymmetry in uncertainties associated with non-negative quantities and large inter-unit variability
• Sensitivity analysis was used to identify key sources of uncertainty and recommend future data collection priorities in order to reduce uncertainty
• There is a proliferation of driving cycles. Some of them are redundant and, therefore, unnecessary
• When comparing model predictions with validation data, or when comparing models with each other, their prediction uncertainty ranges should be considered
• It is difficult to do an uncertainty analysis for a model such as Mobile5 or Mobile6 after the fact, but would be much easier if integrated into the data management and modeling approach.
Air Quality Modeling
• Widely used for:–Assessment of regulatory options–Identification and evaluation of control
strategies (which pollutants, how much control, where to control?)
–Emissions permit applications–Identification of contributing factors to local,
urban, and regional air quality problems–Human exposure assessment
PROBABILISTIC MODELING
Emissions
InputUncertainties
OutputUncertainties
Local NOx
Peak Ozone
Local Ozone
Chemistry
Meteorology
Initial & BoundaryConditions
Variable-GridUrban Airshed Model
(UAM-V)
Local VOC
Case Study of Hanna et al. (2001)
• OTAG Modeling Domain (eastern U.S.)• UAM-V model with Carbon Bond-IV mechanism• Uncertainty in peak ozone concentrations• Assessment of effects of 50% reductions in each of
NOx and VOC emissions• Quantification of uncertainty in 128 inputs
– Literature review for chemical kinetic rate constants– Expert elicitation for emissions, meteorological, and
initial and boundary condition inputs• Monte Carlo simulation with n=100• Correlations used to identify key sources of
uncertainty
Key Findings of Case Study of Hanna et al. (2001)
• It was feasible to perform Monte Carlo simulation on UAM-V for the OTAG domain and a seven day episode
• Simulation results include base case uncertainty estimates for ozone concentrations and estimates of differences in ozone concentrations because of emissions reduction strategies
• There was less uncertainty in estimates of differences in concentration than in absolute estimates of total concentration
• Reductions in NOx emissions led to higher estimated reductions in O3 than did reductions in VOC emissions. This is consistent with the expectation that most of the domain is NOx-limited.
• Key uncertainties include NOx photolysis rate, several meteorological variables, and biogenic VOC emissions
• Compared to Hanna et al. (1998), there was more disaggregation of uncertainty estimates for emission sources, and this may tend to weaken the sensitivity to any one source. It is possible, however, that model outputs would be more sensitive to an aggregated collection of emissions sources.
• There is a need for improved methods of uncertainty estimation, particularly for the chemical mechanism and the meteorological fields, and for better accounting of correlations and dependencies (e.g., temperature dependence of biogenic emissions).
Case Study of Abdel-Aziz and Frey (2004)
• Focus was on evaluating implications of uncertainties in hourly coal-fired power plant NOx emissions with respect to ozone for the Charlotte, NC domain
• Key questions:– (1) what is the uncertainty in ozone predictions solely
attributable to uncertainty in coal-fired utility NOx emissions?;
– (2) can uncertainties in maximum ozone levels be attributed to specific power plant units?;
– (3) how likely is it that National Ambient Air Quality Standards (NAAQS) will be exceeded?; and
– (4) how important is it to account for inter-unit correlation in emission uncertainties
Probability of Exceeding NAAQS: Comparison of 1-hour and 8-hour Standards
Location of Power Plant Impact
Correlation Coeff = 0.87
0.115
0.12
0.125
0.13
0.135
0 20 40 60 80 100
Emissions (t/4 hrs)
Ozo
ne C
onc.
(pp
m)
Analysis of Correlation in Emissions versus Ozone Levels in a Specific Grid Cell Can Detect Influence of a Specific Plant
Key Findings from Abdel-Aziz and Frey (2004) study
• The uncertainty in maximum 1-hour ozone predictions is potentially large enough to create ambiguity regarding compliance with the NAAQS for any given emissions management strategy.
• Control strategies can be developed to achieve attainment with an acceptable degree of confidence, such as 90 or 95 percent.
• There was a substantial difference in results when comparing “independent” versus “dependent” units – thus, it can be important to a decision to account for dependencies between units
• Probabilistic air quality modeling results provide insight regarding where to site monitoring stations and regarding the number of stations needed
• Under the old 1-hour standard, uncertainties in the maximum domain-wide ozone levels could be traced to an individual power plant, thereby implying that control strategies must include that plant.
• Under the new 8-hour standard, uncertainties in maximum ozone levels are attributable to many plants, implying the need for a more widespread control strategy
Risk Assessment Modeling
• Risk assessment is growing in importance as a basis for regulatory decision making–E.g., Phase 2 of MACT standards–Urban air toxics–Food safety and international trade–(etc.)
Human Exposure and Risk Analysis
• Over the last 10-15 years, there has been growing acceptance and incorporation of probabilistic approaches to dealing with inter-individual variability and uncertainty
• EPA has issued various guidance
• International guidance: e.g., FAO/WHO
Example of Probabilistic Techniques in an Exposure and Risk Model: SHEDS
Listeria Monocytogenes Model
Population
•Perinatal •Elderly•Intermediate
Hazard Identification
•Listeriosis•Listeria monocytogen•Food Categories:Seafood, Produce, Dairy, Meats, Combination foods
Risk Characterization
•Estimated likely hood of adverse health effects
Exposure Assessment
•Food Consumption•Food Contamination•Post-Retail Growth of Listeria
Hazard Characterization
•Human Susceptibility•Virulence•Food Composition •Dose-Response Model with Adjustment
E. Coli O:157 in Ground Beef Risk Assessment Model
SlaughterProductionPreparationDose-response
Dose-Response Assessment
MorbidityMortality
ProductionOn-FarmTransportMarketing
of LiveAnimals
SlaughterDehiding
EviscerationSplittingChilling
Fabrication
PreparationGrindingGrowthCooking
Consumption
Exposure Assessment
ScopeHazard Identification
Findings Based Upon Risk Assessment
• Risk assessment applications often differ from others because of distinction between inter-individual variability and uncertainty
• A two-dimensional probabilistic simulation framework is used• Expert judgment is inherent in the process of fitting distributions to
data for variability and is often used to estimate uncertainty in the parameters of such distributions
• Sensitivity analysis is critical to interpretation of risk assessment results:
– Assists in risk management decision-making– Prioritize future work to improve the assessment
• There was a lack of practical guidance regarding sensitivity analysis of risk models that has been addressed by recent work
General Recommendations (1)
• Uncertainty and sensitivity analysis should be used to answer key decision maker and stakeholder questions, e.g.,:– prioritize scarce resources toward additional research or data
collection– make choices among alternatives in the face of uncertainty,– evaluate trends over time, etc.
• Where relevant to decision making, uncertainty and sensitivity analysis should be included as functional requirements from the beginning and incorporated into model and input data development
• There should be minimum reporting requirements for uncertainty in data (e.g., summary statistics such as mean, standard deviation, sample size)
• Federal agencies should continue to improve documentation and accessibility of models and data for public peer review
General Recommendations (2)
• Foster greater acceptance of appropriate methods for including, documenting, and reviewing expert judgment in regulatory-motivated modeling and analysis
• There is a need for flexibility since there are many possible approaches to analysis of uncertainty and sensitivity. Specific choices should be appropriate to assessment objectives, which are typically context-specific
• Human resources for modeling, including uncertainty and sensitivity analysis, should be appropriately committed.
– Adequate time and budget to do the job right the first time (could save time and money in the long run)
– Adequate training and peer review– Promote workshops and other training opportunities, and
periodic refinement of authoritative compilations of techniques and recommended practice
General Recommendations (3)
• Software tools substantially facilitate both uncertainty and sensitivity analysis (e.g., Crystal Ball) but in some ways also limit what is done in practice – there is a long-term need for software tools appropriate to specific types of applications
• Some areas need more research – e.g., best techniques for communication, real-world information needs for decision makers
• The relevance of analyses to decision making needs to be emphasized and considered by analysts
• Decision makers need or should have access to information on why/how they should use probabilistic results
• A multi-disciplinary compilation of relevant case studies and insights from them is a useful way to help convince others of the value of doing uncertainty and sensitivity analysis
• Uncertainty and sensitivity analysis should be an open and transparent process that can be subject to scrutiny and peer review
