Embed Size (px)
Citation preview
Misleading estimates of forecast quality: quantifying skill with
sequential forecastsAlex Jarman and Leonard A. Smith
EGU 2012 Conference – Session NP4.1
23rd April 2012
The support of Munich Re is gratefully acknowledged
Centre for the Analysis of Time Series, London School of Economics
www2.lse.ac.uk/CATS
Serial Dependence in Data and Statistical Inference
Background
Geophysical forecasting applications are numerous and have potentially significant beneficial economic impacts...
Sources: NASA, interestingengineering.com, SOHO/EIT, Torsten Blackwood/AFP/Getty
...hence, the potentially adverse impacts of misleading forecast skill estimation are also significant!
Serial Dependence in Data and Statistical Inference
Background
Serial dependence is common in geophysical data time series – even onlong-term timescales; a well documented and well understood problem forstatistical inference
Source: Wilks, Statistical Methods for the Atmospheric Sciences, 2011
However, the effect on forecast skill has not been so well documented...
Serial Dependence in Data and Statistical Inference
Statistical Inference of Forecast Skill
Source: Wilks, Sampling Distributions of the Brier Score and Brier Skill Score under serial dependence, QJRMS, 2010
Serial correlation can lead to sample variance inflation and overconfidence in skill, BUT not always…
Overview of Research
3 Cases of Serial Correlation Effect/Non-effect on Forecast Skill Inference
1. Linear serial correlation in data
2. Linear serial correlation in data
3. Non-linear serial correlation in data
misleading estimate of forecast skill
non-misleading estimate of forecast skill
misleading estimate of forecast skill
Linear Serial Correlation in Data Misleading Estimate of Skill
Lorenz63 System: Univariate Time Series
Exhibits chaotic behaviour but has strong degree of serial correlation: r1 ~ 0.95
Lorenz63 System: Skill Score Statistics
Skill score serial correlation…
r
1 ~ 0.55
Linear Serial Correlation in Data Misleading Estimate of Skill
Linear Serial Correlation in Data Misleading Estimate of Skill
Lorenz63 System: Skill Score Statistics
Estimates of forecast skill are unreliable…
…but is dependent on location on the attractor and lead time
Overview of Research
3 Cases of Serial Correlation Effect/Non-effect on Forecast Skill Inference
1. Linear serial correlation in data
2. Linear serial correlation in data
3. Non-linear serial correlation in data
misleading estimate of forecast skill (e.g. Lorenz63)
non-misleading estimate of forecast skill
misleading estimate of forecast skill
Linear Serial Correlation in Data Non-misleading Estimate of Skill
AR(1) Process: Time Series
Highly autocorrelated time series (φ=0.9) ...
… but is the serial correlation propagated into the skill score statistics??
Linear Serial Correlation in Data Non-misleading estimate of skill
AR(1) Process: Skill Score Statistics
No serial correlation in skill score: r1 ~ 0
…so forecast skill estimates are reliable, even though the data is serially correlated
Linear Serial Correlation in Data Non-misleading Estimate of Skill
AR(1) Process: Skill Score Statistics
…but the skill estimate is unreliable with a perfect climatological forecast
Overview of Research
3 Cases of Serial Correlation Effect/Non-effect on Forecast Skill Inference
1. Linear serial correlation in data
2. Linear serial correlation in data
3. Non-linear serial correlation in data
misleading estimate of forecast skill (e.g. Lorenz63)
non-misleading estimate of forecast skill (e.g. AR(1) Process)
misleading estimate of forecast skill
Non-linear Serial Correlation in Data Misleading Estimate of Skill
Chaotic Map: Delay Plots
CAUTION FOR DECISION-MAKERS: sample size may be important for accurate skill inference
Truth (Data) Skill Score
Overview of Research
3 Cases of Serial Correlation Effect/Non-effect on Forecast Skill Inference
1. Linear serial correlation in data
2. Linear serial correlation in data
3. Non-linear serial correlation in data
misleading estimate of forecast skill (e.g. Lorenz63)
non-misleading estimate of forecast skill (e.g. AR(1) Process)
misleading estimate of forecast skill (e.g. Chaotic Map)
Misleading Estimates of Forecast Quality: Quantifying Skill with Sequential Forecasts
Summary
1. Serial dependence in verification data can lead to erroneous estimates of forecast skill (in some cases)
• Effect of serial correlation on forecast skill inference outlined by Wilks (2010), who showed:1. the effect is increased for skilful forecasts and rarer events2. sample size becomes important
2. Relationship between serial dependence in data and serial dependence in forecast skill is varied
• 3 identified cases of serial correlation in data, and effect or non-effect on statistical inference of skill e.g. a linearly correlated AR(1) process can result in un-correlated forecast skill
• The degree of serial correlation can be dependent on phase and forecast lead time in dynamical systems e.g. Lorenz63 attractor
• Wilks’s findings are not applicable in all cases e.g an unskilful forecast can lead to skill score sample variance inflation
3. Decision-makers should be aware of the complexities of the relationship, and recognise when the effects of serial dependence may lead to unrealistic expectations of forecast skill!
Additional Information
Contact:
Alex Jarman London School of Economics
Email: [email protected]
EGU Poster:
“Distinguishing between Skill and Value in Hurricane Forecasting”Poster Programme: HS4.3Location: Hall A, #A252
