Optimized localization and hybridization to lter ensemble ......Introduction Linear ltering Joint...

Optimized localization and hybridizationto �lter ensemble-based covariances

Benjamin Ménétrier and Tom AulignéNCAR - Boulder - Colorado

Roanoke - 06/04/2015

Acknowledgement: AFWA

Introduction Linear �ltering Joint optimization Results Conclusions

Introduction

Context:

• DA often relies on forecast error covariances.

• This matrix can be sampled from an ensemble of forecasts.

• Sampling noise arises because of the limited ensemble size.

• Question: how to �lter this sampling noise?

Usual methods:

• Covariance localization→ tapering with a localization matrix

• Covariance hybridization→ linear combination with a static covariance matrix

Introduction

Context:

Usual methods:

Introduction

Context:

Usual methods:

Introduction

Context:

Usual methods:

Introduction

Context:

Usual methods:

Introduction

Context:

Usual methods:

Introduction

Context:

Usual methods:

Introduction

Context:

Usual methods:

Introduction

Questions:

1. Can localization and hybridization be considered together?

2. Is it possible to optimize localization and hybridizationcoe�cients objectively and simultaneously?

The method should:

• use data from the ensemble only.

• be a�ordable for high-dimensional systems.

3. Is hybridization always improving the accuracy of forecasterror covariances?

Introduction

Questions:

The method should:

Introduction

Questions:

The method should:

Introduction

Questions:

The method should:

Introduction

Questions:

The method should:

Introduction

Questions:

The method should:

Introduction

Questions:

The method should:

Introduction

Questions:

The method should:

Outline

Introduction

Linear �ltering of sample covariances

Joint optimization of localization and hybridization

Results

Conclusions

Outline

Introduction

Results

Conclusions

An ensemble of N forecasts {x̃bp} is used to sample B̃:

B̃ =1

∑p=1

δ x̃b(δ x̃b

where: δ x̃bp

= x̃bp−〈x̃b〉 and 〈x̃b〉=

∑p=1

Asymptotic behavior: if N → ∞ , then B̃→ B̃?

In practice, N < ∞ ⇒ sampling noise B̃e = B̃− B̃?

Theory of sampling error:

E[B̃2ij

N(N−3)

(N−1)2E[B̃?2ij

]− 1

(N−1)(N−2)E[B̃iiB̃jj

(N−1)2(N−2)E[Ξ̃ijij

B̃ =1

∑p=1

δ x̃b(δ x̃b

where: δ x̃bp

∑p=1

E[B̃2ij

N(N−3)

(N−1)2E[B̃?2ij

]− 1

B̃ =1

∑p=1

δ x̃b(δ x̃b

where: δ x̃bp

∑p=1

E[B̃2ij

N(N−3)

(N−1)2E[B̃?2ij

]− 1

B̃ =1

∑p=1

δ x̃b(δ x̃b

where: δ x̃bp

∑p=1

E[B̃2ij

N(N−3)

(N−1)2E[B̃?2ij

]− 1

B̃ =1

∑p=1

δ x̃b(δ x̃b

where: δ x̃bp

∑p=1

E[B̃2ij

N(N−3)

(N−1)2E[B̃?2ij

]− 1

Localization by L (Schur product)

Covariance matrix

B̂ = L◦ B̃

Increment

δxe =1√N−1

∑p=1

δ x̃bp◦(L1/2vα

)Localization by L + hybridization with B

Covariance matrix

Increment

δx = βe

δxe + βc B1/2vc

Localization + hybridization = linear �ltering of B̃

Lh and βc have to be optimized together

Covariance matrix

B̂ = L◦ B̃

Increment

δxe =1√N−1

∑p=1

δ x̃bp◦(L1/2vα

Localization by L + hybridization with B

Covariance matrix

Increment

δx = βe

δxe + βc B1/2vc

Covariance matrix

B̂ = L◦ B̃

Increment

δxe =1√N−1

∑p=1

δ x̃bp◦(L1/2vα

Covariance matrix

Increment

δx = βe

δxe + βc B1/2vc

Covariance matrix

B̂ = L◦ B̃

Increment

δxe =1√N−1

∑p=1

δ x̃bp◦(L1/2vα

Covariance matrix

B̂h = (βe)2 L◦ B̃+ (β

Increment

δx = βe

δxe + βc B1/2vc

Covariance matrix

B̂ = L◦ B̃

Increment

δxe =1√N−1

∑p=1

δ x̃bp◦(L1/2vα

Covariance matrix

B̂h = (βe)2 L︸︷︷︸

Gain Lh

◦ B̃+ (βc)2 B︸︷︷︸

Offset

Increment

δx = βe

δxe + βc B1/2vc

Covariance matrix

B̂ = L◦ B̃

Increment

δxe =1√N−1

∑p=1

δ x̃bp◦(L1/2vα

Covariance matrix

B̂h = (βe)2 L︸︷︷︸

Gain Lh

◦ B̃+ (βc)2 B︸︷︷︸

Offset

Increment

δx = βe

δxe + βc B1/2vc

Outline

Introduction

Results

Conclusions

Joint optimization: step 1

Step 1: optimizing the localization only, without hybridization

Goal: to minimize the expected quadratic error:

e = E[‖ L◦ B̃︸︷︷︸

Localized B̃

− B̃?︸︷︷︸Asymptotic B̃

Light assumptions:

• The unbiased sampling noise B̃e = B̃− B̃? is not correlatedwith the asymptotic sample covariance matrix B̃?.

• The two random processes generating the asymptotic B̃? andthe sample distribution are independent.

An explicit formula for the optimal localization L is given inMénétrier et al. 2015 (Montly Weather Review).

e = E[‖ L◦ B̃︸︷︷︸

Localized B̃

Light assumptions:

e = E[‖ L◦ B̃︸︷︷︸

Localized B̃

Light assumptions:

e = E[‖ L◦ B̃︸︷︷︸

Localized B̃

Light assumptions:

e = E[‖ L◦ B̃︸︷︷︸

Localized B̃

Light assumptions:

e = E[‖ L◦ B̃︸︷︷︸

Localized B̃

Light assumptions:

This formula of optimal localization L involves:

• the ensemble size N

• the sample covariance B̃

• the sample fourth-order centered moment Ξ̃

=(N−1)2

N(N−3)

(N−2)(N−3)

E[Ξ̃ijij

]E[B̃2ij

N(N−2)(N−3)

E[B̃iiB̃jj

]E[B̃2ij

This formula of optimal localization L involves:

• the ensemble size N

• the sample covariance B̃

• the sample fourth-order centered moment Ξ̃

=(N−1)2

N(N−3)

(N−2)(N−3)

E[Ξ̃ijij

]E[B̃2ij

N(N−2)(N−3)

E[B̃iiB̃jj

]E[B̃2ij

Step 2: optimizing localization and hybridization together

Goal: to minimize the expected quadratic error

eh = E[‖ Lh ◦ B̃+ (β

c)2B︸︷︷︸Localized / hybridized B̃

Same assumptions as before.

Result of the minimization: a linear system in Lh and (βc)2

= Lij−

E[B̃ij

]E[B̃2ij

(βc)2 (2a)

(βc)2 =

∑ijB

(1−Lh

)E[B̃ij

eh = E[‖ Lh ◦ B̃+ (β

= Lij−

E[B̃ij

]E[B̃2ij

(βc)2 (2a)

(βc)2 =

∑ijB

(1−Lh

)E[B̃ij

eh = E[‖ Lh ◦ B̃+ (β

= Lij−

E[B̃ij

]E[B̃2ij

(βc)2 (2a)

(βc)2 =

∑ijB

(1−Lh

)E[B̃ij

eh = E[‖ Lh ◦ B̃+ (β

= Lij−

E[B̃ij

]E[B̃2ij

(βc)2 (2a)

(βc)2 =

∑ijB

(1−Lh

)E[B̃ij

Hybridization bene�ts

Comparison of:

• B̂ = L◦ B̃, with an optimal L minimizing e

• B̂h = Lh ◦ B̃+ (βc)2 B, with optimal Lh and β

c minimizing eh

We can show that:

eh− e =−(βc)2∑

B2ijVar(B̃ij

)E[B̃2ij

]︸︷︷︸≤0

With optimal parameters, whatever the static B:Localization + hybridization is better than localization alone

Comparison of:

c minimizing eh

We can show that:

eh− e =−(βc)2∑

B2ijVar(B̃ij

)E[B̃2ij

]︸︷︷︸≤0

Comparison of:

c minimizing eh

We can show that:

eh− e =−(βc)2∑

B2ijVar(B̃ij

)E[B̃2ij

]︸︷︷︸≤0

Outline

Introduction

Results

Conclusions

Practical implementation

An ergodicity assumption is required to estimate the statisticalexpectations E in practice:

• whole domain average,

• local average,

• scale dependent average,

• etc.

→ This assumption is independent from earlier theory.

Localization Lh and hybridization coe�cient βc can be computed:

• from the ensemble at each assimilation window,

• climatologically from an archive of ensembles.

• local average,

• etc.

• local average,

• etc.

• local average,

• etc.

• local average,

• etc.

Experimental setup

• WRF-ARW model, large domain, 25 km-resolution, 40 levels

• Initial conditions randomized from a homogeneous static B

• Reference and test ensembles (1000 / 100 members)

• Forecast ranges: 12, 24, 36 and 48 h

Temperature at level 7 (∼ 1 km above ground), 48 h-range forecasts

Standard-deviation (K) Correlations functions

Experimental setup

Localization and hybridization

• Optimization of the horizontal localization Lh

horand of the

hybridization coe�cient βc at each vertical level.

• Static B = horizontal average of B̃

horand of the

• Localization length-scale:

horand of the

• Hybridization coe�cients for zonal wind:

horand of the

• Impact of the hybridization:

• B̃?is estimated with the reference ensemble

• Expected quadratic errors e and ehare computed

Error reduction from e to eh for 25 members

Zonal wind Meridian wind Temperature Speci�c humidity

4.5 % 4.2 % 3.9 % 1.7 %

→ Hybridization with B improves the accuracy of theforecast error covariance matrix

horand of the

4.5 % 4.2 % 3.9 % 1.7 %

horand of the

4.5 % 4.2 % 3.9 % 1.7 %

horand of the

4.5 % 4.2 % 3.9 % 1.7 %

horand of the

4.5 % 4.2 % 3.9 % 1.7 %

Outline

Introduction

Results

Conclusions

1. Localization and hybridization are two joint aspects of thelinear �ltering of sample covariances.

2. We have developed a new objective method to optimizelocalization and hybridization coe�cients together:

• Based on properties of the ensemble only

• A�ordable for high-dimensional systems

• Tackling the sampling noise issue only

3. If done optimally, hybridization always improves the accuracyof forecast error covariances.

Ménétrier, B. and T. Auligné: Optimized Localization andHybridization to Filter Ensemble-Based CovariancesMonthly Weather Review, 2015, accepted

Conclusions

Perspectives

Already done in the paper:

• Extension to vectorial hybridization weights:

δx = βe ◦δxe +βc ◦δxc

→ Requires the solution of a nonlinear system A(Lh,βc) = 0,performed by a bound-constrained minimization.

• Heterogeneous optimization: local averages over subdomains

• 3D optimization: joint computation of horizontal and verticallocalizations, and hybridization coe�cients

To be done:

• Tests in a cycled quasi-operational con�guration

• Extension of the theory to account for systematic errors in B̃?

(theory is ready, tests are underway...)

Perspectives

To be done:

Perspectives

To be done:

Perspectives

To be done:

Perspectives

To be done:

Perspectives

To be done:

Perspectives

To be done:

Perspectives

To be done:

Perspectives

To be done:

Thank you for your attention!

Any question?

Optimized localization and hybridization to lter ensemble ......Introduction Linear ltering Joint...

Documents

Behaviour of Granger causality under ltering: Theoretical

TIME OF ARRIVAL BASED LOCALIZATION IN WIRELESS SENSOR NETWORKS: A NON-LINEAR APPROACH

University of Southampton Research Repository ePrints Soton · grammable pulse shaper based on the backward ... the linear ltering stage is incorporated into ... 3.4 Categories of

Ensemble Kalman ltering without a model

Mobile robot localization using a non-linear evolutionary

Stochastic ltering via L2 projection on mixture manifolds with … · 2019-11-20 · Stochastic ltering via L2 projection on mixture manifolds with computer algorithms and numerical

Localization of buckling modes in plates and laminates79-89).pdf · Localization of buckling modes in plates and laminates ... clamped rectangular plates made of linear elastic,

Structure-oriented bilateral ï¬ltering of seismic images Dave

The Power of Convex Relaxation: Near-Optimal Matrix Completioncandes/publications/... · collaborative ltering [12]. In a nutshell, collaborative ltering is the task of making automatic

VLSI Implementation of Motion Centroid Localization for ......The centroid localization circuit uses time-windowed asynchronously triggered row and column address events and two linear

Research Article Video Denoising Based on a Spatiotemporal ...Finally, by weighting the denoised frames from Kalman ltering and bilateral ltering, we can obtain a satisfactory result

Structural Damage Localization by Linear Technique of ...file.scirp.org/pdf/OJFD_2014123113180306.pdf · M. T. I. Khan et al. 427 Figure 1. Modeling of linear source location technique

Anderson Localization in Quantum Chaos: Scaling and ...przyrbwn.icm.edu.pl/APP/PDF/112/a112z406.pdf · where » is the localization length and L is the system linear size. The change

Optimal sensor and path selection for target tracking in ... ltering by jointly selecting the optimal candidate sensor that participates in target localization and its best communication

Solving high-dimensional nonlinear ltering problems using a … · 2019. 8. 2. · Solving high-dimensional nonlinear ltering problems using a tensor train decomposition method Dedicated

SSA ltering and Fourier inversion: an e cient and fast way

Sharpening through spatial ltering

Recommender Systems - King's College London common approach to designing recommender systems is content-based ltering. Content-based ltering methods are based on a description of the

GATEWALL ANTISPAM · 2016. 4. 6. · (Cloud Antispam); Statistical ﬁ ltering (Bayesian ﬁ ltering method designed by Entensys). ANTIVIRUS FILTERING OTHER FEATURES In addition to

Multitarget Identification and Localization Using Bistatic ... · doi:10.1155/2008/283483 ResearchArticle MultitargetIdentiﬁcationandLocalizationUsing ... form linear arrays (ULA)