[email protected] 16.6.2005 ECMWF User Meeting / 1 Pertti Nurmi Juha Kilpinen Annakaisa Sarkanen ( Finnish Meteorological Institute ) Probabilistic Forecasts

16.6.2005ECMWF User Meeting / [email protected]

Pertti NurmiJuha Kilpinen

Annakaisa Sarkanen( Finnish Meteorological Institute )

Probabilistic Forecasts of Near-Gale Force Winds in the Baltic Applying ECMWF, EPS and Other Methods

ECMWF Forecast Products User Meeting15 – 17 June 2005


A study with 2 frameworks:

i. Develop warning criteria / Guidance methods to forecast probability of near-gale force winds in the Baltic Joint Scandinavian research undertaking e.g. Finland and Sweden issue near-gale & storm force wind

warnings for same areas using different criteria => homogenise !

ii. Evaluation of ECMWF products• Deterministic and probabilistic forecasts• Two (maybe three) calibration methods• Here, only ECMWF data applied Later, HIRLAM, too• Here, 6 Finnish coastal stations Later, c. 15-20 stations

from Sweden, Denmark, Norway• Goal: Common Scandinavian operational practice (?)

Introduction:


• ECMWF MARS

• u & v components at 10 m => wind speed at 10m

• Forecast lead times: +12 hr to +144 hr

• Data retrieval: 0.5 * 0.5 degree resolution

• Operational, Control, EPS data (interpolated to 0.5o * 0.5o)

• Nearest grid point used

• Forecasts / observations valid: 00, 06, 12, 18 utc

• Observations: 10 minute mean wind speed

• Data coverage: 1/10/04 – 30/4/05 212 days

Data:


We may have problems:

• with height of instrumentation ?

• with observing site surroundings and obstacles ?

– with the coast ?

– with nearby islands ?

– with barriers ?

– with installations ?

• with low-level stability ?

NE


02_873 - Hailuoto

02_910 - Valassaaret

02_980 - Nyhamn

02_979 - Bogskär

02_981 - Utö

02_987 - Kalbådagrund

Observing stations( 6 out of 39 )


(m)

55

50

45

40

35

30

25

20

15

10

5

Heights of the instrumentation ( in red, the 6 out of 39 )


46 / 8 m

873 - Hailuoto


22 / 18 m

910 - Valassaaret


32 / 4 m

979 - Bogskär


25 / 8 m

980 - Nyhamn


31 / 9 m

981 - Utö


32 / 23 m

987 - Kalbådagrund


32 m

15,5 m/s

10 m

15

Wind speed dependence:Logarithmic wind profile

14 m/s

979 - Bogskär Unstable

Neutral

Stable

threshold


-20 -10 0 10 20ERROR

0

30

60

90

Coun

t

0.0

0.1

0.2

0.3

0.4

Proportion per Bar

FITTED DISTRIBUTION

Methods for producing probabilistic forecasts:

Deterministic forecasts:

• Error distribution of original sample (212 cases)

• Approximation of the error distribution with a Gaussian fit (, ):”sample error” method

1. EPS (51 members): Probability of wind speed > 14 m/s

2. Kalman filtering– Various approaches No details given here

3. Deterministic forecasts, adjusted by “a posteriori” estimate of the observed error distribution of the dependent sample Probability distribution of near-gale

– Gives an estimate of the upper limit of the probabilistic predictability.


Methods for producing probabilistic forecasts:4. Deterministic forecasts, adjusted with a Gaussian

distribution fitted to model forecasted stability (temperature forecasts at 2 adjacent model levels)

Probability distribution of near-gale, “stability” method- Scheme used at SMHI (H. Hultberg)

5. “Neighbourhood” method- Both spatial (right)

and temporal “neighbours”- c. 25-75 “members”- Applicable primarily for

hi-res models ?


• Traditionally, calibration of the EPS is done by re-labeling the probabilities using the information of the reliability diagram (large sample of past forecasts and observations is needed)

• Here, Kalman filtering is used to calibrate the EPS mean (as well as operational and control forecasts). Then each EPS member is transformed with the same relationship (state vector).– This will calibrate the “mean” of the distribution,

hopefully also the “spread”.

• Kalman filtering is also used in the traditional way to correct the deterministic forecasts and then to estimate the probabilities using the observed error distribution.

Calibration of EPS forecasts:


Sample climatologic characteristics

Sample mean and variance of wind speed (n=212)

0

10

20

30

40

50

60

70

80

90

100

873_Hailuoto 910_Valassaaret 979_Bogskär 980_Nyhamn 981_Utö 987_Kalbåda

Obs_variance (Unit: (m/s)**2) Obs_mean (Unit: m/s)

46 m 22 m 32 m 25 m 31 m 32 m


Sample climatologic characteristics, ref. ECMWF

Sample mean at 12 utc (n=212)

0

2

4

6

8

10


Obs_mean (Unit: m/s) T511_mean (+24h) EPS_mean (+24h)

46 m 22 m 32 m 25 m 31 m 32 m


Sample variance at 12 utc (n=212)

0

20

40

60

80

100


Obs_variance (Unit: (m/s)**2) T511_variance (+24h) EPS_variance (+24h)

46 m 22 m 32 m 25 m 31 m 32 m



Obesrvations at 12 utc; T511 & Control forecasts, +48h) Station: 873_Hailuoto12 UTC

0

5

10

15

20

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171 181 191 201 211

Observed

T511

Control



Deterministic FCs: Bias - RMSE - 981_Utö

-1

-0,8

-0,6

-0,4

-0,2

0

1 2 3 4 5 6

ME_T511

ME_Control

ME_EPS Mean

ME_Kalman0

1

2

3

4

5

1 2 3 4 5 6

RMSE_T511

RMSE_Control

RMSE_EPS Mean

RMSE_Kalman

ME (Bias) RMSE

“Ensemble spread”

w.r.t to FC lead time


Deterministic FCs: Bias - RMSE - 987_Kalbåda

-2

-1,5

-1

-0,5

0

0,5

1 2 3 4 5 6

ME_T511

ME_Control

ME_EPS Mean

ME_Kalman

0

1

2

3

4

5

1 2 3 4 5 6

RMSE_T511

RMSE_Control

RMSE_EPS Mean

RMSE_Kalman

RMSEME (Bias)

w.r.t to FC lead time


Probabilistic FCs: Brier Skill w.r.t to FC lead time

-0,1

0

0,1

0,2

0,3

0,4

0,5

1 2 3 4 5 6

BSS_Kalman (EPS)

BBS_Kalman (T511)

987_Kalbåda Brier Score:

BS = ( 1/n ) Σ ( p i – o i ) 2

– Common accuracy measure of prob fcs

– o i is binary (0 or 1)

– Analogous to MSE in probability space

– A quadratic scoring rule Sensitive to large forecast errors ! Careful with limited datasets !

– Influenced by climatologic frequency

of the sample Different samples not to be compared

Brier Skill Score:

BSS = [ 1 – BS / BS ref ] *100

Range: 0 to 1Perfect score = 0

Range: - to 100Perfect score = 100


Relative Operating Characteristic

• To determine the ability of a forecasting system to discriminate between situations when a signal is present (here, occurrence of gale) from no-signal cases (“noise”)

• To test model performance relative to a specific threshold

• Applicable for probability forecasts and also for categorical deterministic forecasts Allows for their comparison

• Gained popularity in forecast verification in recent years

Probabilistic FCs: ROC


• Graphical representation in a square box of the Hit rate (H) (y-axis) against the False Alarm Rate (F) (x-axis) for different potential decision thresholds

• Curve is plotted from a “binned” set of probability forecasts by stepping (or sliding) a decision threshold (e.g. 10% probability intervals) through the forecasts, each probability decision threshold generating a separate 2*2 contingency table

The probability forecast is transformed into a set of categorical “yes/no” forecasts

A set of value pairs of H and F is obtained, forming the curve

• It is desirable that H be high and F be low, i.e. the closer the point is to the upper left-hand corner, the better the forecast

• A perfect forecast system, with only correct forecasts & no false alarms, (regardless of the threshold chosen) has a “curve” that rises from (0,0) (H=F=0) along the y-axis to (0,1) (upper left-hand corner; H=1, F=0) and then straight to (1,1) (H=F=1)

EventEvent observed

forecastYes No Marginal total

Yes a b a + b

No c d c + d

Marginal total a + c b + d a + b + c + d =n

H = a / ( a + c )

F = b / ( b + d )

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

H

F

10%

20%

30%

40%

50%

60%

90%

80%

70%

Probabilistic FCs: ROC Curve


EventEvent observed

forecastYes No Marginal total

Yes a b a + b

No c d c + d

Marginal total a + c b + d a + b + c + d =n

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

H

F

10%

20%

30%

40%

50%

60%

90%

80%

70% To learn more about ROC and Signal Detection Theory, check:

http://wise.cgu.edu/

H = a / ( a + c )

F = b / ( b + d )

a+c =1920 b+d =5351

Example

Probability # of Cumulative # of Non- Cumulative Non- H FThreshold Occurences Occurences Occurences Occurencies (%) (%)

a b

0 - 9 43 1920 613 5351 100 10010 - 19 172 1877 1389 4738 98 8920 - 29 283 1705 1183 3349 89 6330 - 39 350 1422 936 2166 74 4040 - 49 323 1072 602 1230 56 2350 - 59 287 749 327 628 39 1260 - 69 169 462 151 301 24 670 - 79 163 293 88 150 15 380 - 89 89 130 40 62 7 190 - 99 41 41 22 22 2 0

( a ) ( b )

Probabilistic FCs: ROC Curve generation


• Area under the ROC curve• Decreases from 1 when curve moves downward from the ideal top-left corner

• A useless forecast system is along the diagonal, when H=F and the area is = 0.5;

Such system cannot discriminate between occurrences and non-occurrences of the event

ROCA based skill score:

ROC_SS = 2 * ROCA - 1

• Negative below the diagonal

• At it’s minimum: ROC_SS = - 1, when ROCA = 0

• ROC is applicable for deterministic categorical forecasts– ROC_SS translates into KSS TSS (= H – F )

– Only one single decision threshold - only a single ROC point results

Typically, this is “inside“ the ROC area, i.e. indicating worse quality

• ROC, ROCA and ROC_SS are directly related to a decision-theoretic approach

– Can be related to the economic value of probability forecasts to end users

– Allowing for the assessment of the costs of false alarms

Range: -1 to 1Perfect score = 1

Range: 0 to 1Perfect system = 1

Probabilistic FCs: ROCA Area


Probabilistic FCs: ROC curve/area; T + 48 hr

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

F

H

ROC_EPS

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

F

H

ROCA = 0.73 ROCA = 0.85

ROC_Kalman (EPS)

987_Kalbåda


Probabilistic FCs: ROC curve/area; T + 24 hr981_Utö

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

F

H

ROCA = 0.96

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

ROCA = 0.88

ROC_”stability” ROC_”neighbour”


0,5

0,6

0,7

0,8

0,9

1

1 2 3 4 5 6

ROC_"sample error"

ROC_EPS

ROC_Kalman (EPS)

ROC_Kalman (T511)

0,5

0,6

0,7

0,8

0,9

1

1 2 3 4 5 6

ROC_"sample error"

ROC_EPS

ROC_Kalman (EPS)

873_Hailuoto 910_Valassaaret

EPS

Probabilistic FCs: ROC Area w.r.t to FC lead time


0,5

0,6

0,7

0,8

0,9

1

1 2 3 4 5 6

ROC_"sample error"

ROC_EPS

ROC_Kalman (EPS)

981_Utö 987_Kalbåda

0,5

0,6

0,7

0,8

0,9

1

1 2 3 4 5 6

ROC_"sample error"

ROC_EPS

ROC_Kalman (EPS)

ROC_Kalman (T511)

Probabilistic FCs: ROC Area w.r.t to FC lead time


• So far we’ve just scratched the (sea) surface Need much more experimentation with various methods Different methods for different time/space scales

e.g. very-short vs. medium-range ?

• Biases and other scores depend on station (e.g. observation height)

(Statistical) adjustment of original observations required ? Finland has an operational scheme for this !

• EPS forecasts are slightly under dispersive

• Kalman filtering reduces the biases and produces better prob. forecasts for most stations in terms of the ROC curve/area

• Apply to data from other counterparts

Reach the goal… !!!

Conclusions Future:


Forecast Quality Project 2005The Royal Meteorological Society at the behest of the UK

weather forecasting industry and their customers, has

undertaken a project to establish methodologies and metrics

by which the quality of weather forecast services can be

assessed from a user perspective on a basis that is clear,

scientifically well founded, relevant to the users’ needs

and easily applied and understood.

UK forecast user and provider input is NOW needed!

www.rmets.org/survey

Almost finnish(ed), but one advertisement…

Documents

[email protected] 16.6.2005 ECMWF User Meeting / 1 Pertti Nurmi Juha Kilpinen Annakaisa Sarkanen ( Finnish Meteorological Institute ) Probabilistic Forecasts