Upload
kylee
View
38
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Hidden Error Variances and the optimal combination of static and flow dependent variances. Craig H. Bishop Elizabeth A Satterfield Kevin T. Shanley , David Kuhl, Tom Rosmond, Justin McLay and Nancy Baker Naval Research Laboratory Monterey CA November 2, 2012. Introduction: Definitions. - PowerPoint PPT Presentation
Citation preview
Craig H. Bishop Elizabeth A SatterfieldKevin T. Shanley, David Kuhl, Tom Rosmond, Justin
McLay and Nancy BakerNaval Research Laboratory
Monterey CA
November 2, 2012
Hidden Error Variancesand the optimal combination of
static and flow dependent variances
1
Introduction: Definitions• Error Variance: Mean of a large number of squared forecast
errors.• Flow Dependent Error Variance: Mean of a large number of
squared forecast errors given a particular flow. (In order to obtain a large number of errors the “flow” or “condition” must repeat itself).
• Hidden Error Variance: A flow dependent error variance that is formally unobservable because the particular flow does not repeat itself.
“A conundrum of predictability research is that while the prediction of flow dependent error distributions is one of its main foci, chaos hides flow dependent
forecast error distributions from empirical observation.”
Bishop and Satterfield (2012a,b, MWR, in press), Satterfield and Bishop (2012ab, to be submitted)
Binned Ensemble variance (mm^2)
Binned Ensemble variance0 2 4 5 6
45o
0
1
2
3
4
5
6
7
ET-ctlET-ens
731
a) 850-hpa T (oC)
Binned Ensemble variance0 20 40 50 60
45o
0
10
20
30
40
50
60
3010
b) 850-hpa Td (oC)
Binned Ensemble variance0 20 40 50
45o
0
10
20
30
40
50
3010
c) 850-hpa u (m s-1)
Binned Ensemble variance0 20 40
45o
0
10
20
30
40
3010
d) 850-hpa v (m s-1)
Spread-skill plot for COAMPS simulations relative to the control (solid line) and Mean (dashed line)
0 200 400 500 600
45o
0
100
200
300
400
500
600ET-ctl
ET-ens
300100
48-h total precipitation (mm)
Binned squared
Error (mm)^2
Mean ET-ctl = 119.41
Mean ET-ens = 100.82
Spread-skill plot for COAMPS simulations similar to for 48-h total accumulated precipitation (mm)^2.
Significant spread-skill relationships were found for all variables – including precipitation
Previous work: spread-skill diagramsCollect innovations (ob – fcst) corresponding to similar ensemble variances into a bin. Compute bin averaged squared innovation. It should increase with ensemble variance.
These diagrams do not reveal(a) the climatological range of true error variances, nor (b) the degree of variation of ensemble variance given a true error variance.
3
Overview
• Observations of hidden error variances using replicate systems.• Empirical determination of key pdfs.• Analytic model of statistical relationships of ensemble variances and
true error variances. • Use of (innovation, ensemble-variance) pairs to estimate
parameters of analytic model.• Estimation of optimal weights for Hybrid.• Comparison of performance of Hybrid DA with weights from brute
force tuning and weights from hidden error variance theory.• Other Hybrid results• Conclusions
4
What is the true flow dependent error variance ?
• Imagine an unimaginably large number of quasi-identical Earths.
Each Earth has one true state and one prediction but these differ from one Earth to another.
Collect all Earths having the same true state but differing forecasts of this state to
defin . T|e
t f
f t x x
x x
he error distribution is the distribution of differences
between individual forecasts and single truth within this set.
Collect all Earths having the same historical observati
fixed-tr
ons b
u
u
th
f
t diy
1 2
fering true atmospheric
states to define . The differences between individual truths and the
mean truth within this
| , ,...
fixed-obs set define the forecast error distribution. i
t ti x x y y
(Slartibartfast – Magrathean designer of planets, D. Adams, Hitchhikers …)
5
25000 Lorenz Model ReplicatesReveal Hidden Error Variance
• Using a 10 variable Lorenz ’96 model with additive model error and a 20 member Ensemble Transform Kalman Filter (ETKF) data assimilation scheme, we created 25,000 independent time series of analyses and forecasts, each having the same true state but differing random draws of observation error.
• True Error Variances were then obtained for each spatio-temporal point by averaging the squared forecast error for this point across the 25,000 replicates.
2( | ) ~ ( , )L s k 2 1~ ,prior prior
First demonstration of ETKF accurately predicting true flow dependent error variance in non-linear system.
Scatter plot of ETKF ensemble variance from a single replicate system as a function of true error variance. The true error variance is estimated from all 25,000 replicate systems. The linear fit to the points on the scatter plot is governed by the equation .
6
Controlled accuracy of ensemble variances by degrading ETKF variances
• A primary objective is to show how pdf of true error variances given an imperfect ensemble variance changes as the accuracy of the ensemble variance changes.
• To do this, we created degraded ensemble variances by sampling a Gamma
distribution with mean equal to the ETKF variance and relative variance determined by an “effective ensemble size” M.
2( | ) ~ ( , )L s k 2 1~ ,prior prior
2 2 1| ~ ,s
(b) M=4 (a) M=8
Examples of assumed likelihood gamma pdfs of ensemble variances with a mean of unity. Panel (a) is for an effective ensemble size of M=8, or equivalently, a relative variance of 2/7. Panel (b) is for an effective ensemble size of 4, or equivalently, a relative variance of 2/3. 7
Histograms of true error variance given an imperfect ensemble variance
2( | ) ~ ( , )L s k 2 1~ ,prior prior
2 2 1| ~ ,s The histograms give an empirical estimate of the pdf of true error variances given a constrained range of sample variances for an 8 member ensemble. The ranges are given on each figure; they correspond to the 2nd and 34th bins, respectively, of 35 bins of true error variance. The solid lines give the fit of an inverse-gamma function to the distribution of true error variances in each bin.
Inverse-gamma distribution is a very good fit to empirically derived histogram of true error variances given an ensemble variance for all ensemble variance categories.
M=8
M=8
8
Climatological pdf of true error variances
2( | ) ~ ( , )L s k 2 1~ ,prior prior
2 2 1| ~ ,s
Prior climatological distribution of true error variances. Bars show the probability density histogram of forecast error variances. Solid line shows the fit of the pdf (eq 4) to the data. The thick dashed line marks the mean of both the pdf and the data.
M=8
M=8
Inverse-gamma distribution gives a reasonable fit to empirically derived prior climatological pdf of true error variances.
9
Empirical estimation of pdf of true error variance given ensemble variance from 25000 trials
2 1prior ( ) ~ ( , )
2( | ) ~ ( , )L s k
(a) M=8, empirical (b) M=2, empirical
Red lines depict empirical estimate of pdf of true error variance (ordinate axis) given fixed values of ensemble variance (abscissa axis). Thin green and blue lines give
the mode and mean of the empirical estimates of the mode and mean of these estimates. Panels (a) and (b) show the empirical estimates for random sample
ensembles of sizes M=2 and M=8, respectively. The grey shading gives an inverse-gamma pdf fit to the climatological pdf of true error variances. 10
An analytic model of hidden error variance
Assumption 1: The error of the deterministic forecast is a random draw from a Gaussian distribution, whose true variance i
2 is a random draw from a prior climatological inverse gamma pdf of error variances.
2
2 2 1 2 2 2min
, ~ 0,
~ ( ) , var ,
f t f fi
i prior
x x N
An analytic model of hidden error variance
Assumption 2: Ensemble variances are drawn from a likelihood gamma pdf of ensemble variances with mean a(i
2 - min2)+s2
min
2 2 2 2min
2 2 2 2 2min
2 2 2 2 2 2 2 2 2
( )
| ( )
| ~ | | ,var | ,
i i min
i i i min
i i i i i i i i min
s a s
s a s
s L s s s s
stochastic
(b) M=4 (a) M=8
An analytic model of hidden error variance
Bayes’ Theorem defines the posterior inverse gamma pdf of error variances given an imperfect ensemble variance si
2
2 2 2prior2 2
post2 2 2
prior0
( | ) ( )( | )
( | ) ( )
ii
i
L ss
L s d
Climatological Prior
Distribution
Likelihood distribution of s2
given a particular 2
13
It can be shown that using assumptions 1 and 2 in Bayes’ theorem gives a that is
itself an inverse gamma distribution.
(a) M=8, empirical
(c) M=8, analytic (d) M=2, analytic
(b) M=2, empirical
2 2
2 2
Analytic model of | gives good match
to empirically estimated |
s
s
Green lines give modeBlue lines give mean
Given an ensemble variance, there are a broad range of possible true error variances.
Current DA schemes require a single value.
For the minimum error variance estimate, use the posterior mean.
For the maximal likelihood estimate, …
For QC, … 14
Posterior mean error variance is a Hybrid combination of static and ensemble variances
2 22 2 2 2min
min
1 1
| 1 1 1 1R R
R R R R
s sR Psa
P R P R
Flow dependent ensemble variance
2
2
: true error variance
: ensemble variances
Static climatological mean error variance
2 2
: Relative variance of prior climatological pdf of error variances
: Relative variance of likelihood pdf of ensemble variances |
R
R
P
R L s
i. As the stochastic variation of ensemble variance about the true variance goes to zero, the weight on the ensemble variance goes to 1.
ii. If there is any imperfection in the flow-dependent ensemble variance, the optimal error variance estimate gives weight to the climatological covariance.
iii. If there is no variance of the true error variance, the weight on the static variance goes to 1.
Purely flow dependent error variance models are sub-optimal
Implications for Ensemble DA?Implications for 4DVAR?
15
Problem
How can the climatological relative variance of true variance and/or the
relative variance of ensemble variances given a true variance be determinedwhen the true variance is hidden from direct e
R
R
P
Rmpirical observation?
16
Solution: Equations that define hidden parameters from data assimilation output
17
2
2 2
New equations have been derived that estimate hidden error variance parameters
from a long time series , , 1, 2,..., of (innovation, ensemble-variance) pairs
, where is the observation err
i iv s i n
v R R
4222 2 2 2
min
2 22 2 2 2 2
min min2
2min
or vriance (1)
var var (2)3
covar ,, recall that | = (3)
var
-
Rv
P R R
v sa s a s
s
2 2min
2 2min2 2
min
2 2
2
assigned by designer or min over all if sample is large (4)
(5)
var |
i
R
s s i
s s
as
Rs
2 2 2
2 22 2 2 22 2minmin
var var 2 (6)1var|
s a
Mas
• Retrieved hidden parameters, var(s2), s2, a and M are shown in plots (a), (b), (c) and (d), respectively
• “Light grey bars: M=2, Dark grey bars: M=8• The “given” ensemble sizes in (d) are the random sample ensemble sizes used to
degrade the quality of the ETKF ensemble variance.
Equations recover hidden parameters “observed” by replicate systems
• “Observed” values are obtained from 175 DA cycles of the 25,000 “replicate systems”
• Minimum, mean and maximum of the values retrieved from 21 single system independent time series of
• with n=2,000,000. 2, , 1,2,...,i iv s i n
Each plot summarizes information from 60 independent retrievals. The values marked as min, mean, max and std are the minimum, mean, maximum and standard-deviation of the values retrieved from 60 completely independent synthetically generated data sets.
Recovery of min(sigma^2) is inaccurate when min(sigma^2) is small
These tests pertain to synthetic data generated using the analytical model of hidden error variance
The “specified” were set equal to values previously retrieved from Lorenz model experiments with a “given” M=8 and differing values of the model error q.
Each retrieval is from 2,000,000 (innovation, ensemble-variance) pairs synthetically generated from specified distributions.
Variation of weights for mean of posterior distribution of true error variances with model error q and given effective ensemble size M. Black bars give the weights for the de-biased flow-dependent ensemble variance while grey bars give the corresponding
weights for the static mean of the climatological error variances.
Variation of optimal weights withmodel error and ensemble size, M
Ensemble variance weight in dark grey.Static variance weight in light grey.q gives model error variance parameterM gives an “effective ensemble size” corresponding to the relative variance of a random normal ensemble of size M.
The weight on the ensemble variance increases with ensemble size
The weight on the static variance increases as model error variance increases20
Use of recoveries in Hybrid DA
2 22 2 2 2min
min
22 minEnsemble minmin climatology climatol
1
| 1 , where 1 1
This is a Hybrid of the form
1
R
R R
ff
Hybrid
s s Rs Ra
P R
sa a
PP Q Pogy
f
Flow-dependent ensemble prediction
Covariance matrix of unavoidable errors(?)
Static covariance
21
As a start, let’s guess that the optimal weights for error variance prediction are “useful” weights for error covariance prediction.
{In the following 3 examples, we assume that =0}
Application to Hybrid DA: Lorenz model 1, perturbed observations.
Analysis
0.08
0.085
0.09
0.095
0.1
0.105
0.11
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1Alpha
RM
S
best practice hybrid
standard hybridoptimal weights hybrid - mean
• A suboptimal M=32 member ensemble is generated using a perturbed observations update.
• A climatological error covariance matrix (Pf
climatology) is formed by collecting forecast errors for 100,000 time steps (using an 100% ensemble based error covariance matrix)
• Pfhybrid is computed at each time step and
used in the ETKF DA scheme to obtain an analysis, which is cycled.
• We compute the “best practice” hybrid and the “standard” hybrid for all alpha values for comparison.
Forecast: 2 time steps
0.095
0.1
0.105
0.11
0.115
0.12
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Alpha
MSE
“Best Practice” hybrid: The ensemble based Pf is corrected by a factor of
2
2s
Hybrid based on weights from theory performs as well as that obtained from brute force tuning of the weights.
• The eq’s include a kurtosis term which is likely to be sensitive to data QC decisions based on the size of innovations.• Fortunately, it may be shown that the weight for the ensemble
variances is entirely independent of this term .
• The weight for the static term can then be obtained by insisting that the average of Hybrid variance be consistent with innovation variance.
23
Possible approaches to concerns in application of theory to Hybrid DA
2 2
2
covar ,
varens
v sw
a s
2 2 2
2 2
2.
ens c guess
ensc
guess
w s w
w sw
NAVDAS-AR-Hybrid ResultsLow resolution results
Hybrid weights computed for 6 distinct regions using new theory
Alpha=0.5
Hybrid based on weights from theory performs as well as that obtained from brute force tuning of the weights.
NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0
RMS wind error radiosonde verification results. Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011.(From Kuhl et al. 2012, in review)
NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0
RMS wind error self-analysis verification results. Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011.(From Kuhl et al. 2012, in review)
NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0
RMS error global radiosonde verification results. Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011.(From Kuhl et al. 2012, in review)
NAVDAS-AR-Hybrid ResultsHigh Resolution Results alpha=.5 vs alpha=0
Geopotential Height Anomaly Correlation (verification against self-analysis). Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011.(From Kuhl et al. 2012, in review)
Conclusions1. A simple theory of the relationships between ensemble variances and true
error variances has been developed.2. This theory provides a new method for estimating from an archive of
(innovation, ensemble-variance) pairs a. the prior climatological pdf of true error variances, b. the likelihood pdf of ensemble variances given a true error variance,c. the posterior pdf of true error variances given an ensemble variance, andd. the mean of (c) as a weighted sum of a static and an ensemble variance.
3. The pdfs 2a and 2c are well approximated by inverse-gamma distributions for the Lorenz 96 system.
4. Result (2d) provides a theoretical justification for Hybrid error covariance models that linearly combine static and flow-dependent covariances.
5. The ansatz that the optimal weights (2d) for error variances are useful for error covariances is true for the Lorenz ’96 system with perturbed obs Hybrid DA and a low resolution Hybrid-4DVAR version of the Navy’s operational DA scheme.
6. Enables Hybrid weights to be defined regionally at a fraction of the cost of weights obtained via trial and error.
7. QC and ensemble post-processing applications are also possible.29