The correction of forecast errorsbased on MOS: The impact of

initial condition and model errors

S. Vannitsem

Institut Royal Météorologique de Belgique

Dresden, July 30, 2009

Outline

1. Introduction2. The MOS technique3. The role of initial and model errors: a dynamical view4. Analysis of operational forecasts5. A unified MOS scheme for deterministic AND ensemble forecasts:

EVMOS6. Conclusions and perspectives

The quality of atmospheric forecasts decreases as a function of time

NCEP ensemble forecasts

Model error dynamics

Eta model runs withTwo different convectiveschemes

0.0001

0.001

0.01

0.1

1

10

0 2 4 6 8 10 12

Mea

n sq

uare

err

or

Time (hours)

Kain-Fritch vs Betts-Miller-Janjic convective scheme

U 500 hPaV 500 hPaU 850 hPaV 850 hPalinearquadratic

Two sources of errors affecting the forecasts and the data assimilation process:

-initial condition errors-model errors

See C. Nicolis, Rui A. P. Perdigao, and S. Vannitsem Dynamics of Prediction Errors under the Combined Effect of Initial Condition and Model Errors, Journal of the Atmospheric Sciences , 66, 766–778, 2009.

How can we improve the forecast quality (Besides the improvement of the modeland the observational system)?

Use of techniques to post-process the forecasts in order to improve their quality.

What is corrected by the post-processing method?

The Model Output Statistics technique

• ‘Deterministic’ approach: Linear techniques (Classical MOS, perfectprog), adaptative Kalman filtering. Ref: Glahn and Lowry, 1972, JAM, 11, 1203; Wilks, 2006, Academic Press;….

• ‘Deterministic’ approach: Nonlinear techniques (Neural networks, nonlinear fits). Ref: Marzban, 2003, MWR, 131, 1103; Casaioli et al, 2003, NPG, 10, 373;….

• Probabilistic approaches: correction of the (deterministic) precipitationforecasts. Ref: Lemcke and Kruizinga, 1988, MWR, 116, 1077;….

• (True) Probabilistic approaches: Calibration of the ensemble forecasts. Ref: Roulston and Smith, 2003, Tellus, 55A, 16; Gneiting et al, 2005, MWR, 133, 1098; Wilks, 2006, MA, 13, 243; Hamill and Wilks, 2007, MWR, 135, 2379;…….

Post-processing of forecasts in order to improve their qualitybased on information gathered from past forecasts

The Linear MOS technique

Linear regression between a set of observables of forecasts and observations

∑=

+=n

iiic tVtttX

1

)()()()( βα

∑=

−=M

kkkc tXtXtJ

1

2, ))()(()( M past forecasts

n=1

22 )t(V)t(V)t(V)t(X)t(V)t(X)t(

)t(V)t()t(X)t(

><>>=

Example of fit:

Training:

2 Summer seasons2002-2003

Verification:

2004-2005280

285

290

295

300

280 285 290 295 300

Lead time: 6 hours

Obs

erva

tion,

Cor

rect

ed fo

reca

sts

Forecasts

280

285

290

295

300

305

310

280 285 290 295 300 305

lead time: 60 hours

Obs

erva

tins,

Cor

rect

ed fo

reca

sts

Forecasts

280

285

290

295

300

275 280 285 290 295 300

Lead time: 120 hours

Obs

erva

tions

, Cor

rect

ed fo

reca

sts

Forecasts

2-m Temperature, Uccle

The mean square error

Properties of the MOS forecast

2)(

2222

)(

222 )())(),(())()(()(

)()(

tVXXtV

cc

c

ttVtXCtXtXt

tXtX

σβσσσ

σ =>=>>=−>=

Short term dynamics of MOS forecasts (Formalism based on Nicolis, 2004, JAS)

The real system{ }

{ }),,(

),,(

λ

μ

YXSdtYd

YXFdtXd

rrrr

rrrr

=

=

Xr

Vr

and span the same phase space

Variablesnot describedby the model

{ })',V(GdtVd

μ=rr

r

The model

{ }

{ })),(),(()0()(

)'),(()0()(

0

0

∫

∫

+=

+=

t

t

YXFdXtX

VGdVtV

μτττ

μττ

rrrrr

rrrrFormal sols

)(21))()(( 32210

2 tOtStSStt VC +++=−σσ

Typical initial errors

)0()0(X)0(V ε+= rrr Gaus random noise

+Systematic error

22

2)0(1

2)0(

2)0(

4)0(

0

))0(())0((),0((2

)))0(())0((),0((2

XFXGVCS

XFXGVCS

SX

rr

rr

−∝

−∝

+=

ε

ε

ε

σ

σσσ

>−>

))))0(())0((()()0()0((2)))0(())0(((2

)))0(())0(()()0()0((2

))0()0((

0

2'2

'1

22'0

>⎥⎦⎤−−−=

Summary

A. Correction of initial condition errors

B. Correction of model errors

- The systematic part is corrected.

- The time-dependent part can be corrected provided thatthe covariance of the model error with the predictors is high.

- Systematic initial error is corrected.

- Random initial errors are only partially corrected.

2)0(

2)0(

4)0(

0X

Sσσ

σ

ε

ε

+=

The Lorenz system

zxzbxydtdz

Gybxzxydtdy

aFaxzydtdx

−+−=

+−−=

+−−−= 22

- Initial condition errors: N(0,s), s small- Model errors (parametric error on a, b, F or G)

Chaotic for a=0.25, b=6, F=16, G=3

Results obtained with 100,000 realizations starting from differentinitial conditions

The Lorenz system (continued)

10-7

10-6

10-5

10-4

10-3

10-2

10-1

100

101

2 4 6 8 10

Mea

n S

quar

e E

rror

Time

(a)

10-7

10-6

10-5

10-4

10-3

10-2

10-1

100

0.01 0.1 1

10-1610-1410-1210-1010-810-610-410-2100

0.01 0.1 1 10

Mea

n s

quar

e er

ror

Time

(a)

Initial cond

ition error o

nly

Model error

only

Error in parameter a

Correction of variable x

The Lorenz system (continued)

10-12

10-10

10-8

10-6

10-4

10-2

100

0.01 0.1 1 10

( Ym- Yt)2

( Yc- Yt)2, MOS X, Y

( Yc- Yt)2, MOS Y, XZ

Mea

n S

quar

e E

rror

Time

Partial conclusions

- IC not well corrected by MOS- MOS can correct model errors when model predictors well chosen

Model errors only in the parameter b

Correction of variable y

Different MOS schemes

- Predictors: X, Y- Predictors: Y, XZ

Application to the ECMWF forecasting system: temperature

Data: temperature for the period December 1 2001 to November 30 2005

- Training period: December 1 2001 to November 30 2003

- Independent evaluation period: December 1 2003 to November 30 2005

-Temperature 500 hPa, 850 hPa-Temperature at 2 meters

Evaluation on a grid covering Belgium (verification using the set of analyses)

(52 N, 2 E) to (49 N, 7 E)

Evaluation at some synoptic stations: model forecast given by the closestgridpoint

0

1

2

3

4

5

6

0 20 40 60 80 100 120

Raw forecastsMOS, T 850 hPa

MS

E

Time (hours)

Temperature at 850 hPa

Results averaged over the wholegrid

0

1

2

3

4

5

6

7

0 20 40 60 80 100 120

Corrected forecastsRaw forecasts

RM

SE

Time (hours)

(a)

2002-2003

Temperature at 2 meters

0

1

2

3

4

5

6

0 20 40 60 80 100 120

Corrected forecasts

Raw forecasts

RM

SE

Time (hours)

(c)

2004-2005

1

2

3

4

5

6

7

0 20 40 60 80 100 120

Raw forecastsMOS, T2mMOS, T2m, T850hPaMOS, T2m, SH

MS

E

Time (hours)

(a)

Uccle

Uccle-Ukkel

1

2

3

4

5

6

7

8

9

0 20 40 60 80 100 120

Raw forecastsMOS, T2m

MS

E

Time (hours)

(a)

Uccle

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 20 40 60 80 100 120

V CD C

Cor

rect

ion

s

Time (hours)

(b)Uccle

Training + verif on 2002-2003

Training: 2002-2003Analysis on 2004-2005

Other stations

1

2

3

4

5

6

7

8

0 20 40 60 80 100 120

Raw forecastsMOS, T2m

MS

E

Time (hours)

(a)

Middelkerke

0

2

4

6

8

10

12

0 20 40 60 80 100 120

Raw forecastsMOS, T2m

MS

E

Time (hours)

(a)

Elsenborn

Training + verif on 2002-2003

A second predictor?

Sensible heat flux

A second predictor?

TMP 850 hPa

New Linear MOS

)()(

)()()()()(

)()()()(

2

2

22

22

tt

tVtVtXtXt

tVttXt

V

X

σσβ

βα

=><><

=

>=<

)(, tX kc

Intermediate cost function:

Minimization

∑= +

−+=

M

k

kk

ttttXtVtttJ

1222

2

)()()())())()()((()(

δκ σβσβα

κβςα ++=X δς +=V

One ‘free’ parameter: 2

2

2

2

tofixed X

V

σσ

σσλ

κ

δ=

∑∑==

+−+

−=

M

k

kkM

k

kk

tttX

tttVtJ

12

2

12

2

)()))(()((

)())()(()(

κδ σβςα

σς

Needs some knowledge aboutthe sources of errors

2)(

2)(

222 )())()(()(

)()(

tXtVCC

C

ttXtXt

tXtX

σσβσ =>=>>=>

268

270

272

274

276

278

280

282

284

286

288

268 270 272 274 276 278 280 282 284 286 288

Observations

Forecasts

Lead time: 24 hours, Initial time: 12

RAWEVMOSLMOS

268

270

272

274

276

278

280

282

284

286

288

268 270 272 274 276 278 280 282 284 286 288

Observations

Forecasts

Lead time: 120 hours, Initial time: 12

RAWEVMOSLMOS

265

270

275

280

285

290

266 268 270 272 274 276 278 280 282 284 286

Observations

Forecasts

Lead time: 240 hours, Initial time: 12

RAWEVMOSLMOS

Control Forecast is used.

Training: DJF 2002-2004

Verif: DJF 2002

MOS at Station ‘Uccle’.

0

0.5

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250

RMSE, SPREAD

Time (hours)

(a)

DMO, RMSEEVMOS-1, RMSE

DMO, SPREADEVMOS-1, SPREAD

0

0.5

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250

RMSE, SPREAD

Time (hours)

(b)

DMO, RMSENGR, RMSE

DMO, SPREADNGR, SPREAD

0

0.5

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250

RMSE, SPREAD

Time (hours)

(c)

DMO, RMSEEVMOS-2, RMSE

DMO, SPREADEVMOS-2, SPREAD

0

0.5

1

1.5

2

2.5

3

3.5

4

0 50 100 150 200 250

RMSE, SPREAD

Time (hours)

(d)

DMO, RMSELMOS-2, RMSEDMO, SPREAD

LMOS-2, SPREAD

Application to an operational ensemble: ECMWF, Winter, Uccle

2 meter Temperature

Conclusions on MOS

A dynamical analysis of the MOS correction has been undertaken with emphasison the correction of

-initial condition errors-Model errors

Both partly corrected butcorrection more sensitiveto model errors.

)))0(())0((),0(( XFXGVCrr

−One central quantity

An additional model observable can improveconsiderably the correction if proportional to G-F

For the ECMWF deterministic forecast:

TMP 2m

- Substantial corrections with MOS with 1 observable - MOS with 2 observables:

In the central part of Belgium: Tmp 850 hPaOn the coast: Sensible Heat Flux

TMP 850 hPa

- No correction. Mainly initial condition errors

More infos:

Vannitsem S. and C. Nicolis, Dynamical properties of Model Output Statistics forecasts. Mon. Wea. Rev., 136, 405-419, 2008.

Vannitsem S., Dynamical properties of MOS forecasts. Analysis of the ECMWF operational forecasting system. Weather and Forecasting, 23, 1032-1043, 2008.

Correction of Ensemble forecasts based on EVMOS

A unified Scheme for both deterministic and ensemble forecasts is proposed,based on the assumption that the forecast displays errors.

-Do not damp the ensemble spread anymore.

-The correction provides a mean and a variabilityin agreement with the observations

-The scheme can be trained on the control forecast only.

References:

Vannitsem, S., A unified Linear Model Output Statistics scheme for both deterministic and ensemble forecasts. Accepted inQuart. J. Roy. Met. Soc., 2009

The correction of forecast errors based on MOS: The impact of initial condition and model errors The Model Output Statistics techniqueSummaryThe Lorenz systemThe Lorenz system (continued)New Linear MOS