Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss...

Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss

Extended range forecasts at MeteoSwiss: User experience and

probabilistic verification

ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel,

Mark Liniger, Paul Della-Marta,

Christof Appenzeller

2 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Overview

Monthly forecasts

WWW: Seasonal forecasts

Verification: The RPSSD

Comparison:

ECMWF vs. other prediction strategies

3 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Overview

Monthly forecasts

WWW: Seasonal forecasts

Verification: The RPSSD

Comparison:

ECMWF vs. other prediction strategies

4 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Monthly forecasts

100 %0

Probability of T2m to be in lowest tercile

Forecast of week 1Start: 20-04-2006

5 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Monthly forecasts

100 %0

Probability of T2m to be in lowest tercile

Forecast of week 1Start: 04-05-2006

6 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Monthly forecasts

Probability of T2m to be in lowest tercile

100 %0

Forecast of week 1Start: 11-05-2006

7 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Monthly forecasts

Observed anomalies for May

What is wrong?Problems to deal with enhanced snow cover?...

8 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Overview

Monthly forecasts

WWW: Seasonal forecasts

Verification: The RPSSD

Comparison:

ECMWF vs. other prediction strategies

9 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

WWW: Seasonal forecasts

Since winter 2005/06 MeteoSwiss issues an internet bulletin

(»Climate Outlook«) on the upcoming season for Switzerland.

Designed to... provide seasonal forecast give background information on methodology point out uncertainties provide climatologic background information

Provide common reference for public and media, and

avoid dissemination of semi-true information

Use seasonal forecasts to promote public interest in other aspects of climate analysis

10 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Facts and figures on the summer in Switzerland What do the records show?

WWW: Seasonal forecasts

11 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Maximum temperaturein °C

Precipitationin mm

Average sunshine durationin %

WWW: Seasonal forecasts

12 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Seasonal forecast current model run.

WWW: Seasonal forecasts

13 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

WWW: Seasonal forecasts

Terciles from station data

14 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

What is a seasonal forecast? Methodology

WWW: Seasonal forecasts

15 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Past seasonal forecasts for Switzerland Verification

WWW: Seasonal forecasts

16 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

WWW: Seasonal forecasts

observation

17 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Overview

Monthly forecasts

WWW: Seasonal forecasts

Verification: The RPSSD

Comparison:

ECMWF vs. other prediction strategies

18 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Verification of probabilistic forecasts

Real-valued observations

Probabilisticforecasts

Common approach: Convert observation into probability distribution

Ranked Probability Score (RPS)

19 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Verification of probabilistic forecasts

Real-valued observations

Probabilisticforecasts

Ensemble predictions

But: Ensemble predictions are not truly probabilistic !!

Ranked Probability Score (RPS)

20 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

0

100%CDF

C N W

Example:

Three equiprobable categories (e.g. “cold”, “normal”, “warm”)

Let the verifying observation fall into the second category

Convert real-valued observation into CDF

The Ranked Probability Score (RPS)

21 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

C N Wd1,EPS

d2,EPS Ensemble prediction system:RPS = d2

1,EPS + d 22,EPS

Example:

Three equiprobable categories (e.g. “cold”, “normal”, “warm”)

Let the verifying observation fall into the second category

Compare with CDF of ensemble forecast

The Ranked Probability Score (RPS)

22 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

C N W

1/32/3

1

d1,Cl

d2,Cl Ensemble prediction system:RPS = d2

1,EPS + d 22,EPS

Climatologic forecast:RPSCl = d2

1,Cl + d 22,Cl

Example:

Three equiprobable categories (e.g. “cold”, “normal”, “warm”)

Let the verifying observation fall into the second category

... or with CDF of climatologic forecast

The Ranked Probability Score (RPS)

23 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

The Ranked Probability Skill Score (RPSS) is defined by relating the RPS of a forecast system with the corresponding RPS of the climatologic reference:

The RPSS is negatively biased for small ensemble size !

The RPSS

24 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Synthetic random white noise forecasts, verified against random white noise observations.

Skill of this forecast system should be zero by definition !

Three equiprobable categories

The RPSS

25 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Negative bias consequence of inconsistent definition of climatologic reference forecast.

Müller et al. 2005, J.Clim.Weigel et al. 2006,

Mon. Wea. Rev.

The RPSS

1/3

26 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Solution

K: Number of forecast categoriespi: Prob. of i-th forecast categoryM: Ensemble size

General case

Weigel et al. 2006,Mon. Wea. Rev.

The RPSSD

27 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Special case 1:

K equiprobableforecast categories

M: Ensemble size

Solution

The RPSSD

Weigel et al. 2006,Mon. Wea. Rev.

28 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Special case 2:

Brier score, i.e. twocategories with probp and (1-p)

M: Ensemble size

Solution Weigel et al. 2006,Mon. Wea. Rev.

The RPSSD

29 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

The RPSSD

Synthetic random white noise forecasts, verified against random white noise observations.

Skill of this forecast system should be zero by definition !

Three equiprobable categories

30 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

ECMWF System 2 forecasts (1988-2002), verified against ERA40 T2m predictions for March, lead time 4 months 2 equiprobable forecast categories (i.e. Brier Score situation)

Southern Africa

The RPSSD

31 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Large ensembles still useful!

RPSSD determines the “true skill” of the EPS

It measures model quality, not forecast quality

Particularly useful for model assessment studies:

multi-model studies, when models of different ensemble size are to be compared

comparison of deterministic and probabilistic forecasts

The RPSSD

32 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Overview

Monthly forecasts

WWW: Seasonal forecasts

Verification: The RPSSD

Comparison:

ECMWF vs. other prediction strategies

33 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Example 1: Statistical model

Model: CCA statistical model

Training: 1880-1960

Verification: 1960-2001

Predictors: • DJF North Atlantic SST • JFMA total precipitation (north. Mediterranian)

Predictand: JJA daily homogenized Tmax station series

Reference: Della-Marta et al. (2006), Clim. Dyn.

34 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

ECMWF (DEMETER) CCA model

Example 1: Statistical model

• Three equiprobable forecast categories• JJA forecasts of T2m, initialized in May• Verification period: 1960-2001

RPSSD

35 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Example 2: The Böögg

Böögg: RPSSD = -0.15 ECMWF: RPSSD = 0.19

36 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Böögg’s Prognosis forsummer 2006:

Time until head exploded:10 minutes 28 seconds

warm summer

37 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

The Ranked Probability Score (RPS)

Example:

Three equiprobable categories (e.g. “cold”, “normal”, “warm”)

Let the verifying observation fall into the second category

C N W

Observation

38 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

The Ranked Probability Score (RPS)

Example:

Three equiprobable categories (e.g. “cold”, “normal”, “warm”)

Let the verifying observation fall into the second category

0

100%PDF

C N W

Convert real-valued observation into PDF

39 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Böögg

Time until head explodes (min)

Mea

n JJ

A t

empe

ratu

re

R2 = 0.0071p = 0.599

RPSSD = -0.15

ECMWF (DEMETER)

Ensemble mean for JJA T2m

R2 = 0.2497p = 0.0016

RPSSD = 0.19

heat summer 2003

Example 2: The Böögg

40 Extended range forecasts at MeteoSwiss – user experience and probabilistic verification

ECMWF Forecast Products User Meeting, ECMWF, Reading, UK, 14-16 June 2006

Andreas Weigel

Central Europe

The RPSSD

ECMWF System 2 forecasts (1988-2002), verified against ERA40 T2m predictions for March, lead time 4 months 2 equiprobable forecast categories (i.e. Brier Score situation)