View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Copyright © 2004 David M. Hassenzahl
Understanding Uncertainty
Definitions and Tools for Risk Analysts
Copyright © 2004 David M. Hassenzahl
Goals for this Lecture
• Understand sources and types of uncertainty
• Evaluate quantitative and qualitative representations of uncertainty– Distributional forms
• Explore some typologies• Consider how people interpret
uncertainty
Copyright © 2004 David M. Hassenzahl
Overview
• Uncertainty is unavoidable• Uncertainty is information!
– Where to do further research– Worst case scenarios
• Uncertainty is chronically understated– Generally satisfied with statistical analysis– Henrion and Fischhoff (1986), estimates of speed
of light (see Kammen and Hassenzahl 1999 page 125)
Copyright © 2004 David M. HassenzahlFrom Henrion and Fischhoff
(1986)
Expected valuewith standard error
Mea
sure
d sp
eed
of li
ght (
km/se
c)
Year of experiment
1870
299600
299650
299700
299750
299800
299850
299900
299950
300000
1880 1890 1900
Year of experiment
299750
299760
299770
299780
299790
299800
299810
299820
299830
1900 196019501940193019201910
1984value
1970
Recommended valuewith reporteduncertainty
Copyright © 2004 David M. Hassenzahl
Some thoughts
• “Research efforts in risk analysis should be viewed as tools for understanding uncertainties, not necessarily for reducing them.” (Finkel, 1990)
• “Probability does not exist” (Morgan and Henrion, 1990)– Either you die from cause X or you don’t
Copyright © 2004 David M. Hassenzahl
Uncertainty Analysis Should Be Thorough
• It should include quantitative measures of uncertainty
• It should include qualitative discussion of uncertainty
• Doing this well takes time and effort (and is not always rewarded…)
Copyright © 2004 David M. Hassenzahl
Quantitative Measures
• Display the data set
• Use descriptive statistics– Moments, ranges, etc
• Apply sensitivity analysis
• Eschew spurious precision!
Copyright © 2004 David M. Hassenzahl
Use of Point Estimates?
• “introducing confidence intervals…to deal with uncertainty may be pointless…unless the policy analyst eliminates the point estimate itself.”
• “A specific number has a vividness and simplicity which makes it an inevitable focus of policy debate.”
• Camerer and Kunreuther, 1989
Copyright © 2004 David M. Hassenzahl
Point Estimate or not?
-100%
0
1
Option
Changein risk
+100%
increase
reduction
2 3 4 5donothing
Figure 10-6 from SWRI page 258
Copyright © 2004 David M. Hassenzahl
Qualitative discussion
• Discuss known sources of error
• Consider plausible sources of error
• Evaluate the importance of uncertainty
Copyright © 2004 David M. Hassenzahl
Challenges to ignoring uncertainty
• Regarding a report on ozone depletion “…no attempt was made to estimate the systematic errors in evaluating rates or omission of chemical processes. Without such estimates, decision makers are free to make their own judgments ranging from uncritical acceptance of the current models to complete skepticism as to their having any likelihood of being correct.” Morgan and Henrion 1990.
Copyright © 2004 David M. Hassenzahl
Historical Mistakes
• Rasmussen Report (WASH 1400) on reactor safety– Significant portions retracted by US government– Still referenced
• Inhaber report on nuclear “safer” than other energy technologies– Major source (Holdren) responded “not so”
• Chauncy Starr– Voluntary/involuntary– Risk as f(benefit)– True…but not precisely known
• Tengs et al?
Copyright © 2004 David M. Hassenzahl
Briggs and Sculpher (1995)
• Cost-effectiveness analyses from medical literature– Public Health origins of CEA
• Incomplete or inadequate attention to uncertainty analysis in 86%
Copyright © 2004 David M. Hassenzahl
Starr (1969) interpretation
-11
-4
-5
-6
-7
-8
-9
-10
-3
100 500020001000500200 10000
Average annual benefit/person involved [dollars]
Pf [F
atalit
ies/p
erso
n hr
. exp
osur
e]
R~B 3
R~B 3
Voluntary
Involuntary
Hunting, skiing,smoking
Railroads
Naturaldisasters
Commercialaviation Motor
vehicles
General aviation
Average P fdue to disease
Electricpower
R = RiskB = Benefit
1010
1010
1010
1010
1010
1010
1010
1010
1010
1010
Copyright © 2004 David M. Hassenzahl
Otway and Cohen (1975) Interpretation
R~B 1.8
R~B 6.3
Voluntary
Involuntary
Hunting, skiing,smoking
Railroads
Naturaldisasters
Commercialaviation Motor
vehicles
General aviation
Average P f dueto disease
-11
-4
-5
-6
-7
-8
-9
-10
-3
100 500020001000500200 10000
Average annual benefit/person involved [dollars]
Pf [F
atalit
ies/p
erso
n hr
. exp
osur
e]
Electricpower
R = RiskB = Benefit
1010
1010
1010
1010
1010
1010
1010
1010
1010
Copyright © 2004 David M. Hassenzahl
Example: Amitraz on Pear Orchards
• EPA decided to ban Amitraz for use on pear orchards (US EPA 1979)
• Point estimate generated for Cost effectiveness– $2.6 million per life-year saved (Tengs et al
1995)
Copyright © 2004 David M. Hassenzahl
Expected Value of Ban
• Does Amitraz control pests?– If not, ban has no economic implications– cpyls ≤ $0
• Is Amitraz a carcinogen?– If not, ban has major economic implications– cpyls $ ∞
• E(cpyls) ≠ $2.6 million• E(cpyls) = uniform($0, $∞)• Hassenzahl (2004)
Copyright © 2004 David M. Hassenzahl
Estimators
• Because we can’t always get the data we want, we need to estimate data
• We can use– Frequencies– Distributions– Curve-fitting
Copyright © 2004 David M. Hassenzahl
Frequency example: 500 people
• 495 have 10 toes
• 2 have 12 toes
• 1 has 9 toes
• 1 has 5 toes
• 1 has 0 toes
• In next 1000: how many will have 6 toes?
Copyright © 2004 David M. Hassenzahl
Distributions as estimators
• We often use DISTRIBUTIONAL FORMS to approximate data sets
• We then estimate missing or future values using the distribution– Extrapolate beyond data set– Interpolate within data set
Copyright © 2004 David M. Hassenzahl
Example: 500 people
• feet number• 0 0• 0.5 0• 1 0• 1.5 0• 2 0• 2.5 0• 3 1• 3.5 1
• feet number• 4 12• 4.5 48• 5 145• 5.5 203• 6 78• 6.5 10• 7 2• 7.5 0
Copyright © 2004 David M. Hassenzahl
Triangular? Normal?
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0 1 2 3 4 5 6 7 8 9
Height (feet)
Fre
qu
ency
Copyright © 2004 David M. Hassenzahl
Normal: Characteristics: Mean = 5.6 feet, Standard Deviation = 1.3 feet
Frequency Chart
.000
.005
.011
.016
.022
0
54.5
109
163.5
218
2.21 3.89 5.58 7.27 8.96
10,000 Trials 9,921 Displayed
Forecast: Normal
Copyright © 2004 David M. Hassenzahl
Triangular? Normal?
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0 1 2 3 4 5 6 7 8 9
Height (feet)
Fre
qu
ency
Copyright © 2004 David M. Hassenzahl
Triangular? Normal?
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0 1 2 3 4 5 6 7 8 9
Height (feet)
Fre
qu
ency
Copyright © 2004 David M. Hassenzahl
A Good Estimator Is
• Consistent
• Unbiased
• Efficient
• Sufficient
• Robust
• Practical
Copyright © 2004 David M. Hassenzahl
Depicting Uncertainty: Distributions
• No consensus on how to depict ignorance!– More later under Monte Carlo Analysis
• Key forms– Uniform / rectangular– Triangular– Normal, lognormal– Many others!
Copyright © 2004 David M. Hassenzahl
Decision
• When should we hold the Founder’s Day parade?
• Assume we want to avoid tornadoes
Copyright © 2004 David M. Hassenzahl
Uniform
0.001
0.002
Prob
abili
ty o
f tor
nado
Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec
Month
Oct
0.003
From SWRI page 106. Figure 3.5 Uniform distribution for daily p(tornado) as described in problem 3-7a.
Kammen and Hassenzahl 1999
Copyright © 2004 David M. Hassenzahl
Rectangular
From SWRI page 106. Figure 3-6 Rectangular distribution for daily p(tornado) as described in problem 3-7a.
Prob
abili
ty o
f tor
nado
Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec
Month
Oct
0.004
0.002
0.006
0.008
0.010
0.012
Kammen and Hassenzahl 1999
Copyright © 2004 David M. Hassenzahl
Triangular
From SWRI page 107. Figure 3-7 Triangular distribution for daily p(tornado) as described in problem 3-7a.
Prob
abili
ty o
f tor
nado
Jun Jul Aug Sep
MonthOct
0
0.005
0.010
0.015
0.025
0.020
Kammen and Hassenzahl 1999
Copyright © 2004 David M. Hassenzahl
Policy Decision
• Radon is found in homes across the country
• We might worry at 100 pCi/liter• We can’t measure all the homes in the
country, but we have a decent sized sample
• It matters how we model that distribution
Copyright © 2004 David M. Hassenzahl
Normal Distribution
10 7432 5 6
2
10
8
6
4
0
14
12
8
Num
ber o
f hom
es
Radon level (pCi/liter)
Kammen and Hassenzahl 1999
Copyright © 2004 David M. Hassenzahl
Lognormal
0
5
10
15
20
Hous
es (%
)
222 Rn (pCi/liter)
>8
0 42 6 8
Nero et al 1987
Copyright © 2004 David M. Hassenzahl
Exponential
0.0001
0.001
0.01
0.1
1
10
100
1 10 100 1000
USAless6
pwr(1.75) %
pwr(1.25) %
LnNrm3.5%
Rn Concentration
% o
f hou
ses w
ith co
ncen
tratio
n >
x
Goble and Socolow 1990
Copyright © 2004 David M. Hassenzahl
Fitting distributions to data sets
• Compare data to predictions– Least squares– Maximum Likelihood
• We will explore these in context– Binomial for animal toxicology data
Copyright © 2004 David M. Hassenzahl
Breaking Down Uncertainty
• Useful typologies for thinking about uncertainty
• Can’t always reduce uncertainty
• Typologies have– Internal overlaps– Missing pieces (no perfect typology)
Copyright © 2004 David M. Hassenzahl
Four typologies
• Finkel– Parameter Uncertainty– Model Uncertainty– Decision-rule Uncertainty
• Smithson– non-quantifiable/holistic
aspects – uncertainty as one
component of ignorance.”
• Boholm– Situates Uncertainty
as the non-calculable part of risk
– appropriate coping strategies:
• faith• precaution and• avoidance
Copyright © 2004 David M. Hassenzahl
Typology (after Morgan and Henrion 1990)
1. Random error and statistical variation
2. Systematic error and subjective judgment
3. Linguistic imprecision
4. Variability
5. Randomness and unpredictability
6. Expert Uncertainty
7. Approximation
8. Model uncertainty
Normative Uncertainty
Copyright © 2004 David M. Hassenzahl
Random error/statistical variation
• We have a well defined set of tools
• These can be misleading!– Often the ONLY thing that is done– Often done…and ignored
• Z-scores, Chi-squared, p-values
• Meaning of 95% confidence interval?
Copyright © 2004 David M. Hassenzahl
Random Error: “Clusters”
Copyright © 2004 David M. Hassenzahl
Systematic error and subjective judgment
• Example: speed of light, or energy predictions
• Chronically understated
• Useful approach: bounding
Copyright © 2004 David M. Hassenzahl
Systematic Error: Predicted Year 2000 Energy Use
1972 19821980197819761974
Year of Publication
0
125
100
50
25
150
75
200
175
Qua
drill
ion
Btu
Per Y
ear
Goldemberg et al 1987
Copyright © 2004 David M. Hassenzahl
Linguistic imprecision
• Inconsistencies in language and usage can lead to problems
• Is “beyond a reasonable doubt” 95%? What does that mean?
• “Rain is likely”…are you from Las Vegas or Bangladesh?
• “A few thousand deaths”
Copyright © 2004 David M. Hassenzahl
Variability
• Also called “dispersion”
• Get the right population!
• Describing variability can be a challenge
• Monte Carlo analysis is a useful tool
Copyright © 2004 David M. Hassenzahl
Variability
• Height of individuals– Deterministic
element– Random element
• Susceptibility to disease– Predisposition
• Known• Unknown
– Life-history and habit
• Theoretical models• Empirical data• Who are we worried
about?– “Average” person– Most susceptible
subset
Copyright © 2004 David M. Hassenzahl
Randomness and unpredictability
• Inherent randomness is irreducible!
• Practical limitations and chaos
Copyright © 2004 David M. Hassenzahl
Expert uncertainty
• Multiple interpretations of a single data set
• Norms of analysis
• Motivational bias (decision stakes, reputation)
Copyright © 2004 David M. Hassenzahl
Expert Uncertainty
• Economists’ Conception– Limited resources– Resource substitution– Adaptability
• Ecologists’ Conception– Stable systems– Long term impact of disruption
Copyright © 2004 David M. Hassenzahl
Climate Change: Economists “versus” Ecologists
Individual respondents' answers
Loss
of g
ross
wor
ld p
rodu
ct
14 17 3 16 1 2 9 1812156114 510137 8-5
0
10
5
25
20
15
90th percentile
10th percentile
50th percentile
Nordhaus 1994
Copyright © 2004 David M. Hassenzahl
EMF’s and Expertise
• Biomechanists / physicists– “Impossibility” theorems
• Epidemiologists– Correlation and proposed causation
• Toxicologists– Extrapolation
Copyright © 2004 David M. Hassenzahl
Approximation
• Never have complete data
• There is a tradeoff between efficient computation and resolution or precision– Sensitivity analysis
• Significant figures are important
Copyright © 2004 David M. Hassenzahl
Model uncertainty
• Getting the right model– Does the model explain the data?– Is the model consistent with theory?
• Getting the model right– Is the model properly stated?– Is the math done correctly
• Other uncertainties (drawn from the typology above)
Copyright © 2004 David M. Hassenzahl
Model Uncertainty
0.0001
0.001
0.01
0.1
1
10
100
1 10 100 1000
USAless6
pwr(1.75) %
pwr(1.25) %
LnNrm3.5%
Rn Concentration
% o
f hou
ses w
ith co
ncen
tratio
n >
x
Goble and Socolow 1990
Copyright © 2004 David M. Hassenzahl
Normative Uncertainty
• Often not asked: what is important to us?
• Arguments about technical information mask the true issues
• Leads to vitriol and claim of “ignorance” and “antiscientific attitudes”
Copyright © 2004 David M. Hassenzahl
Normative UC and YMP
• Gore, Bush: “let the science decide”
• Secretary Abraham “technically suitable site”
• LV residents “technically unsuitable site”
• Have we defined “suitability?”
Copyright © 2004 David M. Hassenzahl
Interpreting Uncertainty
• Very limited information on how people interpret uncertainty
• Possible links– Uncertainty and credibility– Uncertainty and trust
Copyright © 2004 David M. Hassenzahl
References
• Briggs, A. and M. Sculpher (1995) Sensitivity analysis in economic evaluation: a review of published studies. Health Economics 4: 355-371.
• Camerer, C. F. and Kunreuther, H. (1989) “Decision Processes for Low Probability Events: Policy Implications,” J. Policy Analysis and Management 8 (4), 565 - 592.
• Goble, R. and Socolow, R. (1990), “High Radon Houses: Implications for Epidemiology and Risk Assessment,” Cented Research Report No. 5 (Clark University, Worcester, MA).
Copyright © 2004 David M. Hassenzahl
References
• Goldemberg, J,. Johansson, T., Reddy, A and Williams, R (1987). Energy for a Sustainable World. Washington DC: World Resources Institute.
• Hassenzahl, D. M. (2004). “The effect of uncertainty on ‘risk rationalizing’ decisions.” Journal of Risk Research.
• Henrion, Max and Fischhoff, Baruch (1986) “Assessing uncertainty in physical constants”, American J. of Physics,54, 791 - 798.
Copyright © 2004 David M. Hassenzahl
References
• Nero, A. V., Schwehr, M. B., Nazaroff, W.W. and Revzan, K.L. (1986) “Distribution of Airborne Radon-222 Concentrations in U.S. Homes,” Science (234) 992 - 997.
• Nordhaus, W. D. (1994) “Expert opinion on climate change”, American Scientist, 82, 45 - 51.
• Otway, H. and J. J. Cohen (1975). Revealed Preferences: Comments on the Starr Benefit-Risk Relationships. Laxenburg, Austria, International Institute for Applied Systems Analysis.
Copyright © 2004 David M. Hassenzahl
References
• Starr, C. (1969). "Social benefit versus techological risk." Science 165: 1232-1238.
• Tengs, T. O., M. E. Adams, et al. (1995). "Five-Hundred Life-Saving Interventions and Their Cost-Effectiveness." Risk Analysis 15(3): 369 - 390.
• US EPA (1979) Determination pursuant to 40CFR162.11(a)(5) concluding the rebuttable presumption against registration of pesticide products containing amitraz. Federal Register 48: 2678-83.
Copyright © 2004 David M. Hassenzahl
Additional readings
• Kammen and Hassenzahl, 1999. Should We Risk It? Exploring Environmental, Health and Technology Problem Solving, Princeton University Press, Princeton NJ.
• Finkel, Adam, 1990. Confronting Uncertainty in Risk Management (Resources for the Future, Washington DC)
• Morgan, M. Granger and Max Henrion (1990). Uncertainty Cambridge University Press, NY NY.