Diagnostics at ECMWF

M.J. Rodwell1

ECMWF Annual Seminar 2009

Diagnostics at ECMWF

Mark Rodwell

7 September 2009

Work with Thomas Jung

M.J. Rodwell2

Outline

● Scores, Metrics and Diagnostics● Diagnosis: The Changing Task● The “Diagnostics Explorer” ● Using Analysis Increments & Initial Tendencies● Scale-Dependent Verification● Diagnostic Verification: Precipitation

M.J. Rodwell3

Scores: Verification of 500 hPa Geopotential

% MA = 12 Month Moving Average

55

60

65

70

75

80

85

90

95

100

Based on ECMWF operational forecasts from 12 UTC analysis. MA=12 month moving average

T+ 24 T+ 24 MA

T+120 T+120 MA

1990 1995 20001980 1985 2005

SPATIAL ANOMALY CORRELATION (N.HEM.)

David Richardson

Not a diagnostic but used to identify errors, busts etc

Presenter

Presentation Notes

Not really a diagnostic but can be used to (e.g.) highlight bad forecasts (busts)

M.J. Rodwell4

Metrics: Blocking Frequency. DJF 1963-2006

32R232R3ERA40

29R230R1ERA40

35R235R3ERA40

32R333R1ERA40

0

30

20

10

0

30

20

10

-90 0 90 180 -90 0 90 180

Freq

uenc

y (%

)Fr

eque

ncy

(%)

Longitude Thomas Jung

(a)

(d)(c)

(b)

Highlights blocking problem, but not the reasons

Presenter

Presentation Notes

Again, not really a diagnostic(?) but can be used to monitor and assess aspects of model variability (model climate) Is recent fall in blocking frequency significant? What are the underlying reasons?

M.J. Rodwell5

Diagnosis: The Changing Task

M.J. Rodwell6

Temporal Variance of D+1 Error of Z500

0 5 10 15 20 25 30 35 40

50403020

1086432

10.80.60.40.30.2

1980/11983/4

1987/8

1991/2

1995/6

1999/2000

2003/4

2007/8

Adrian Simmons (2006) - updated→ 1279!

Growing need to optimise signal-to-noise ratio, statistical significance etc.

Presenter

Presentation Notes

Law of diminishing returns (even though a squared quantity) Peak error also indicates dominant spatial scale of Z500 …Deterministic forecast about to be run at T1279! More assessment of scale-dependent error or concentrate on smaller scale parameters … precipitation Need to carefully check the significance of results Optimise signal-to-noise ratio: E-suites, careful construction of diagnostics

M.J. Rodwell7

Observational Data Volumes

0

5

10

15

20

25

30

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

TOTAL

CONV+AMV

Peter Bauer and Jean-Noel Thepaut

SatelliteConventional & AMV

x106 24hr-1

Unit is millions of data values assimilated per 24 hour period

Growing monitoring task. Increasingly difficult to attribute errors: Observation or model? (VAR-BC)

Presenter

Presentation Notes

Ever-increasing number of observations (probably exponential prior to 1996 too!) Growing task of monitoring impacts But more information to check against Better estimation of the truth With aspects like VAR-BC, which can attribute model error to observation bias, diagnostics need to be a lot more ‘seamless’ across the traditional assimilation / forecast divide.

M.J. Rodwell8

Diagnostics of the Diagnostics group

M.J. Rodwell9

On-line Diagnostics: A 5D view of the IFSIFS Component Diagnostics

Data Assimilation

Observation space – observation usage● Many data sources including radiosonde and satellite● Data count, first-guess departures (mean, rms), bias corrections

Model space – analysis increments● Prognostic and other parameters● Mean, standard deviation, rms● 21 pressure levels and zonal means

Weather forecast

Forecast error● Prognostic and other parameters● Mean, standard deviation, rms● 21 pressure levels and zonal means

Scale-dependent error and activity● Several parameters, levels and regions● All spatial scales and selected spatial scales

Climate of atmospheric model and coupled model

Seasonal-means of error● Several diagnostics including geopotential height, winds, velocity

potential, Hadley and Walker circulations, ocean waves

Seasonal-means of variability● Blocking● ENSO teleconnections● Empirical Orthogonal Functions● Planetary and synoptic activity● Power spectra● Tropical waves (including Madden-Julian Oscillation)

● All diagnostics are produced for operational forecasts (seasonal means) and “E-suites”.

● Some diagnostics are produced for research experiments.

● “Initial Tendency” diagnostics will be added.

● Aim: Seamless and efficient diagnosis of entire forecasting and data assimilation system.

● Other sections produce more detailed diagnostics for their particular IFS component.

M.J. Rodwell10

Analysis Increments and Initial Tendencies

M.J. Rodwell11

-55 -25 -15 -5 5 15 25 65 -55 -25 -15 -5 5 15 25 65 -9 -5 -3 -1 1 3 5 9 -9 -5 -3 -1 1 3 5 9

-21 -5 -3 -1 1 3 5 15 -21 -5 -3 -1 1 3 5 15

T500 Forecast Error as function of lead-time

D+1

D+5

D+2

Based on DJF 2007/8 operational analyses and forecasts. Significant values (5% level) in deep colours.

x0.01K

-34 -10 -6 -2 2 6 10 26 -34 -10 -6 -2 2 6 10 26

x0.1K

x0.1K x0.1K

D+10

Shorter lead-times localise error and increase signal-to-noise ratio

Presenter

Presentation Notes

Shorter ranges focus on local error and enhance statistical significance. Both factors are important

M.J. Rodwell12

Data Assimilation Cycle: Perfect Model

(Imperfect, unbiased observations)

ObservationsAnalysis

Analysis increment

First guess forecast

0 1 2 3 4Time (cycles)

T

Mean Analysis Increment = 0

For a perfect model, positive and negative increments tend to cancel

Presenter

Presentation Notes

In the data assimilation process, we try to produce an ‘analysis’ that is as close to the observations as possible but also being (approximately) a valid model state. The data assimilation starts with a ‘first guess’ forecast initiated from a previous analysis. As shown above, this forecast will diverge from the subsequent observations. Chaos ensures that this divergence occurs even if the model is ‘perfect’. The data assimilation at ECMWF then uses the tangent-linear model to iteratively find a new model state that is closer to the new observations. The ‘analysis increment’ (as denoted by the dotted arrows) is the difference between the new analysis and the first guess forecast. To first-order, a perfect model will produce as many erroneously cold first-guess forecasts as it will produce erroneously warm first-guess forecasts. Hence, the analysis increments for a perfect model will average to zero over sufficiently many data assimilation cycles. Note that this is true (to first order) even if the observations are not perfect as long as they are unbiased.

M.J. Rodwell13

Data Assimilation Cycle: Imperfect Model

ObservationsAnalysis

Analysis increment

First guess forecast

0 1 2 3 4Time (cycles)

T

Mean Analysis Increment ≠ 0

−Mean Analysis Increment = Mean Net Initial Tendency (in appropriate units)= Convective + Radiative + … + Dynamical mean tendencies

(summed over all processes in the model)The use of “Initial Tendencies” was first proposed by Klinker and Sardeshmukh (1992)

Presenter

Presentation Notes

The mean analysis increment is equivalent to (minus) the mean initial tendency (in units of, e.g., K/cycle). If a model has a systematic error (we will assume it has a cooling tendency as seen above) then, on average, the first guess will be colder than the observations. This will be reflected in a positive mean temperature analysis increment (as denoted by the dotted arrows). How might such a systematic error arise? The concept of ‘radiative-convective’ equilibrium embodies the idea that radiative processes act to destabilise the atmosphere (heat the surface and cool the mid-to-upper troposphere) and the convection induced by this destabilisation acts to restore balance by cooling the surface and heating the mid-to-upper troposphere. With this idealised concept in mind, either a convection scheme that is too weak (given the observed temperature and humidity profiles) or a radiation scheme that is too strong (given the observed conditions; as embodied by the analysis) would lead to a systematic initial net cooling of the mid-troposphere. With the mean initial tendencies (or analysis increments), we therefore have a diagnostic that can quantify local model physics error before significant interactions have taken place with the resolved dynamics. The advantage of initial tendencies over analysis increments is that initial tendencies can be broken-down into the component tendencies from each physical and dynamical process within the model. We can, for example, diagnose convective and radiative tendencies separately.

M.J. Rodwell14

-86 -10 -6 -2 2 6 10 74

AIRS CH 215 OBS-FIRST GUESS

0.6m/s0.6m/s

Unit 0.01K

-52 -20 -12 -4 4 12 20 76 -52 -20 -12 -4 4 12 20 76

T500 ANALYSIS INCREMENT-55 -25 -15 -5 5 15 25 65 -55 -25 -15 -5 5 15 25 65

D+1 T500 FORECAST ERROR

Confronting Models with ObservationsUNIT=0.01K

Based on DJF 2007/8 operational analyses and forecasts. Significant values (5% level) in deep colours.AIRS CH 215 BRIGHTNESS TEMPERATURE ~T500

● Every 1o square has data every cycle● ~6 Million data values

● Independent vertical modes of information:● IASI / AIRS: ~ 15● HIRS / AMSUA: ~ 5 (~ 2 IN TROP)

● Anchors (not bias corrected):● Radiosonde● AMSUA-14● Radio Occultation

Presenter

Presentation Notes

Shows some of the plots available on the “Diagnostics Explorer” Reasonable agreement between AIRS FG-DEP and analysis increments. However, the main point here is not that the AIRS looks like the analysis increments but that there is a lot of data that constrains the analysis

M.J. Rodwell15

Zonal Mean Errors

M.J. Rodwell16

-11 -5 -3 -1 1 3 5 41 -11 -5 -3 -1 1 3 5 41 -57 -15 -9 -3 3 9 15 63 -57 -15 -9 -3 3 9 15 63 -57 -15 -9 -3 3 9 15 63 -57 -15 -9 -3 3 9 15 63

O O O O O O O O O

-57 -15 -9 -3 3 9 15 63 -57 -15 -9 -3 3 9 15 63-57 -15 -9 -3 3 9 15 63 -57 -15 -9 -3 3 9 15 63

0

0

0

0

0

0

0

0

0

0

-57 -15 -9 -3 3 9 15 63 -57 -15 -9 -3 3 9 15 63

Mean Temperature TendenciesTOTAL CI = 0.2Kd-1

V.DIFF & GWDLARGE-SCALE PRECIPRADIATIVE

CONVECTIVE CI = 0.6Kd-1 DYNAMICAL

0ON 40OS 80OS80ON 40ON

200

400

600

800

1000

hPa

200

400

600

800

1000

hPa

0ON 40OS 80OS80ON 40ON 0ON 40OS 80OS80ON 40ON

Mean tendencies are deduced on model levels. Y-axis shows approximate pressure value. Average is over December 2008, 4 forecasts per day, tendencies accumulated from T+1 to T+7. Model cycle 33R1, TL159, L91

Total tendency should be zero for a perfect model. Why isn’t it? Extratropics too!

Presenter

Presentation Notes

Emphasise that total should be zero. What is the reason of the (erroneous) total tendency? Too much convection? Wrong balance between compensating dynamics and radiation? Interesting features in the extratropics too

M.J. Rodwell17

Impact of New Radiation Scheme

O O O O O O O O O

MEAN AN INC

RMS AN INCMEAN AN INC

TOTAL TENDENCYCI=0.2Kd-1 NEW RADIATION (McICA) – OLDCONTROL

200

400

600

800

1000

hPa

200

400

600

800


New radiation scheme reduced upper-tropospheric increments

Presenter

Presentation Notes

Some recent improvements in mid-to-upper tropical troposphere actually associated with radiation changes

M.J. Rodwell18

-14 -10 -6 -2 2 6 10 18 -14 -10 -6 -2 2 6 10 18

-351 -15 -9 -3 3 9 15 339 -351 -15 -9 -3 3 9 15 339

80ON 60ON 40ON 20ON 0O 20OS 40OS 60OS 80OS

-351 -15 -9 -3 3 9 15 339 -351 -15 -9 -3 3 9 15 339-15 -5 -3 -1 1 3 5 33 -15 -5 -3 -1 1 3 5 33

-351 -15 -9 -3 3 9 15 339 -351 -15 -9 -3 3 9 15 339

Mean Meridional Wind TendenciesTOTAL CI=0.2ms-1d-1

V.DIFF & GWDAN INC CI=0.4ms-1d-1

CONVECTIVE CI=0.6ms-1d-1 DYNAMICAL


200

400

600

800

1000

hPa

200

400

600

800

1000

hPa



Either the dynamics in the upper troposphere are wrong, or there is a missing process.

Hypothesis: need to increase tropical ascent and decrease radiative cooling and convective heating.

Presenter

Presentation Notes

Erroneous tendency is a weakening of the Hadley Circulation. Dynamical tendencies at 300 hPa are not compensated and are directly reflected in total tendencies and an incs. Either the dynamics are wrong or there is a missing process. (My hunch is that the dynamics is “wrong” in that there is not enough ascent balancing the tropical convection. This may imply that the radiation should cool a little less in the upper troposphere, which may lead to weaker convection but (hopefully!) more ascent. This would improve both the tropical T tendencies and the subtropical v tendencies.

M.J. Rodwell19

-70 -10 -6 -2 2 6 10 74 -70 -10 -6 -2 2 6 10 74

-70 -10 -6 -2 2 6 10 74 -70 -10 -6 -2 2 6 10 74 -70 -10 -6 -2 2 6 10 74 -70 -10 -6 -2 2 6 10 74

-70 -10 -6 -2 2 6 10 74 -70 -10 -6 -2 2 6 10 74

-36 -20 -12 -4 4 12 20 84 -36 -20 -12 -4 4 12 20 84

-84 -20 -12 -4 4 12 20 28 -84 -20 -12 -4 4 12 20 28

Mean Specific Humidity TendenciesTOTAL CI=0.08gkg-1d-1

V.DIFF & GWDLSPANALYSIS INCREMENT

CONVECTIVE CI=0.4gkg-1d-1 DYNAMICAL


200

400

600

800

1000

hPa

200

400

600

800

1000

hPa



Presenter

Presentation Notes

An incs dry pbl: partly a consequence of rectification of noise at saturation point. Observations actually want to moisten profile (Alan Geer)

M.J. Rodwell20

Specific Humidity at ~850 hPa

0.5m/s0.5m/s

Unit = 0.01g/kg

-63 -15 -9 -3 3 9 15 39 -63 -15 -9 -3 3 9 15 39

SSMI channel 3Observation – First GuessCI=0.4K

Analysis Increment CI=0.06gkg-1

SSMI channel 3 all-sky microwave brightness temperature “first guess departures” and analysis increments are based on all 0 and 12 UTC data assimilation cycles 20090401—20090815 for IFS cycle 35R3 (E-suite), TL799, L91. SSMI brightness temperature has a positive correlation with humidity.

-62 -10 -6 -2 2 6 10 82

Drying increments in lower troposphere despite observations ‘wanting’ to moisten: rectification of noise at saturation point(?)

Presenter

Presentation Notes

Some agreement where there are observations but drying increments elsewhere (these dominate at lower levels)

M.J. Rodwell21

-129 -15 -9 -3 3 9 15 93 -129 -15 -9 -3 3 9 15 93

-205 -25 -15 -5 5 15 25 225 -205 -25 -15 -5 5 15 25 225 -205 -25 -15 -5 5 15 25 225 -205 -25 -15 -5 5 15 25 225-105 -15 -9 -3 3 9 15 153 -105 -15 -9 -3 3 9 15 153

-205 -25 -15 -5 5 15 25 225 -205 -25 -15 -5 5 15 25 225

Mean Zonal Wind Tendencies (Trop & Strat)

TOTAL CI=0.6ms-1d-1

V.DIFF & GWDAN INC CI=0.6ms-1d-1

CONVECTIVE CI=1.0ms-1d-1 DYNAMICAL



1020

100200

1000

hPa12

1020

100200

1000

hPa12 Balance between

dynamical and gravity-wave drag tendencies is not perfect

Presenter

Presentation Notes

Main balance is between dynamical and GWD tendencies. Still not perfectly balanced. AN INCs imply that low-frequency variability is not an issue (but a mean drift back from months mean anomaly towards climate could be an issue?)

M.J. Rodwell22

The Asian Monsoon

M.J. Rodwell23

1

3

5

7

9

11

13

15

Precipitation 33R1-GPCP (6-8 1963-2006)

-10

-4

-2

-0.5

0.5

2

4

10

Precipitation JJA 1963-2006

Too much rainfall in the Asian monsoon has been a long-standing problem

OBSERVED PRECIPITATIONGPCP

PRECIPITATION ERRORMODEL CLIMATE-GPCP

Model is cycle 33R1, TL159 L91

mm d-1

M.J. Rodwell24

JJA 2008 u and v 925hPa Analysis Increments

0.8m/s0.8m/s

Unit = 0.1m/s

-29 -5 -3 -1 1 3 5 15 -29 -5 -3 -1 1 3 5 15

Analysis Increments indicate that the model wants to transport too much moisture into the monsoon: The root-cause of the monsoon error?

x0.1ms-1

M.J. Rodwell25

-158 -10 -6 -2 2 6 10 154 -158 -10 -6 -2 2 6 10 154

-158 -10 -6 -2 2 6 10 154 -158 -10 -6 -2 2 6 10 154

-158 -10 -6 -2 2 6 10 154 -158 -10 -6 -2 2 6 10 154

-45 -5 -3 -1 1 3 5 103 -45 -5 -3 -1 1 3 5 103

Initial Tendencies JJA 2008: u at 925 hPa

(c) Convection (d) Total

(b) Vertical diffusion & Gravity Wave Drag(a) Dynamics

Unit = ms-1 over first 24h of forecast

Arabian Sea region is unusual in having strong compensating tendencies

M.J. Rodwell26

-183 -15 -9 -3 3 9 15 141 -183 -15 -9 -3 3 9 15 141

-183 -15 -9 -3 3 9 15 141 -183 -15 -9 -3 3 9 15 141

-43 -5 -3 -1 1 3 5 41 -43 -5 -3 -1 1 3 5 41

-183 -15 -9 -3 3 9 15 141 -183 -15 -9 -3 3 9 15 141

Initial Tendencies JJA 2008: u at 925 hPa

(c) Convection (d) Total

(b) Vertical diffusion & Gravity Wave Drag(a) Dynamics

Unit = ms-1 over first 24h of forecast

Further work required to understand which tendency is at fault (other parameters, CRMs etc)

M.J. Rodwell27

The Madden-Julian Oscillation (MJO)

M.J. Rodwell28

EOFs of Vel.Pot. ERA-Interim Re-analysis

(a) EOF 1

-0.035 -0.025 -0.015 -0.005 0.005 0.015 0.025 0.035

(b) EOF 2

EOFs derived from ERA-Interim re-analyses for the period 19890101—20071231

Will focus on the physics associated with EOF1 in the box indicated.

M.J. Rodwell29

PC1&2 of Vel.Pot. Operational Analyses

50 strongest PC1(used in composite)

EOF2 leads EOF1 by a quarter period: indicating eastward propagation

M.J. Rodwell30

MJO Phase Propagation In ECMWF Model

Phase Shift 1→2 2→3 3→4 4→5 5→6 6→7 7→8 8→1

OBS 71% 81% 81% 80% 86% 79% 68% 55%

Mod 71% 81% 80% 71% 72% 78% 65% 87%

Convection over Maritime Continent

Percentage of times that the MJO moves from one phase to the next within 10 days. Model cycle 32R3.

Frederic VitartModel finds it difficult to propagate the MJO through the “Warm Pool”

M.J. Rodwell31

Unit 0.1K

-108 -20 -12 -4 4 12 20 84 -108 -20 -12 -4 4 12 20 84

Dyn

Con

Rad

Initial Tendencies (First 24hr): T500, δMJOUnit=0.1K

~4Kday-1

Main balance between convective heating and dynamic cooling (due to ascent).Radiation stabilises atmosphere behind MJO

M.J. Rodwell32

Analysis Increments (12hr window): T5000.5m/s0.5m/s

Unit = 0.01K

-44 -20 -12 -4 4 12 20 28 -44 -20 -12 -4 4 12 20 28

0.5m/s0.5m/s

Unit = 0.01K

-22 -10 -6 -2 2 6 10 30 -22 -10 -6 -2 2 6 10 30

Unit=0.01K

Unit=0.01K

MJO+

MJO-

δMJO ~0.4Kday-1

Main problem is not associated with the MJO. Need to reduce wet bias.

Model “MJO convection” ~90% of true signal?

M.J. Rodwell33

Scale-Dependent Verification

● Northern Mid-latitudes Spring 2009

DiagnosticsR&J 34

Z500 Mean-Squared Error and Activity: Ops

Forecasts in spring 2009 were less skilful than in 2008

DiagnosticsR&J 35

Z500 D+5 MAM: 2009-2008

RMSE: Operational Forecast EPS Spread: Operational Forecast

Eady Index: Operational Analysis Synoptic Activity: Operational Analysis

mm

m

… More synoptic activity partly the reason?

DiagnosticsR&J 36

Z500 Scale-Dependent MSE and Activity: Ops

Both synoptic and planetary waves were problematic. There was more synoptic activity but less planetary activity

DiagnosticsR&J 37

Mean Z500 Anomalies

-20

-20

-20

-20

20

20

20

20

20

20

20

60

10°N

10°N

10°N

10°N

20°N

20°N

20°N

30°N

40°N

50°N

60°N

70°N

80°N

180160 W140 W

W

W

W

W60°E

80°E

100°E

120°E

140 E160 E

-160

-120

-100

-80

-60

-40

-2020

40

60

80

100

120

160

-20

-20

-20

20

20

20

20

20

20 60

10°N

10°N

10°N

10°N

20°N

20°N

20°N

30°N

40°N

50°N

60°N

70°N

80°N

180160 W140 W

W

W

W

W

140 E160 E

MAM 2008 MAM 2009

Planetary waves were very different in 2009 compared to 2008

DiagnosticsR&J 38

Z500 Scale-Dept. MSE and Activity: ERA-Interim

Similar results when same model cycle used in both years. IFS has problem with 2009 planetary waves(?)

DiagnosticsR&J 39

Z500 MSE and Activity: FC versus CF

High-resolution model performs better than EPS control

DiagnosticsR&J 40

Z500 Scale-Dependent MSE and Activity: FC,CF

… For synoptic and planetary waves

M.J. Rodwell41

Diagnostic Verification

● Verification measures that aid diagnosis of error● Verification of smaller-scale quantities: Precipitation

DiagnosticsR&J 42

Linear Error in Probability Space & Equitable Skill

● “LEPS” – Ward and Folland (1991) in terms of seasonal rainfall totals● Equitable scores – Gandin and Murphy (1992)

Observation

Probability p0 p1 … pn

Forecast

Category 0 1 … n

0 s00 s10 … sn0

1 s01 s11 … sn1

: : : · :n s0n s1n … snn

Equitable Categorical Skill Score

-200 2000

1

Precipitation (mm)

Obs

Cum

ulat

ive

Prob

abili

ty

Linear Error in Probability Space

pΔ

pΔ

,

1

0

0 1

i iii

i iji

j i ij ji j j

p s

p s j

q p s q

=

= ∀

= =

∑

∑

∑ ∑

Constant FC

Random FC

Perfect FC

Aim: to combine these two concepts into a score for daily precipitation forecasts

DiagnosticsR&J 43

A New Score for Precipitation (“SLEEPS”)

Observation

Probability p q q

Forecast

Category No Rain

Light Rain

Heavy Rain

No Rain 0 c c+aLight Rain d 0 aHeavy Rain d+b b 0

● Better score ⇒ Better system● Less sensitive to sampling uncertainty● Aggregation of errors from different

climatic regions

Semi-Linear error in probability space

0 TPmax

0

1

Precipitation (mm)

Obs

Cum

ulat

ive

Prob

abili

ty

Use of Cumulative Distribution

q

q

p

“Light Rain” “Heavy Rain”No Rain

x

Enhanced Equitability Constraints

Nearest Grid-box to Point-Observation● Monitor scores as

resolution improves● 24hr accumulation

improves representativeness

1pdq)b(p12c)q(a

=++=+

qapd1qapd

==+

● Clear link between forecast score and model error (compared to, e.g., correlation)

DiagnosticsR&J 44

Extra-Tropical ‘SLEEPS’ Lead-time to Error of 0.6

For the Northern Hemisphere, improvements must be due to model formulation, resolution or data assimilation methodology (not increasing number of observations)

DiagnosticsR&J 45

0 0.099 1 2 5 10 20 61 0 0.099 1 2 5 10 20 96.82 0 0.3 0.4 0.5 0.6 0.7 0.8 1

Dry Light Heavy Dry Light Heavy 0 0.3 0.6 0.9 1.2 1.5 1.8 6.9

(a) Observation (b) Forecast (c) Probability No Precip.

(d) Observed Category (e) Forecast Category (f) SLEEPS

Ten Worst D+4 Precip. Forecasts of 2008: #8

Forecast is penalised for not predicting dry weather in (generally wet) Northern Europe in winter

DiagnosticsR&J 46

0 0.099 1 2 5 10 20 38 0 0.099 1 2 5 10 20 28.05 0 0.3 0.4 0.5 0.6 0.7 0.8 1

Dry Light Heavy Dry Light Heavy

(a) Observation

0 0.4 0.8 1.2 1.6 2 2.4 8

(b) Forecast (c) Probability No Precip.



Forecast penalised for not predicting convective rain in (generally dry) Southern Europe in summer

DiagnosticsR&J 47

0 0.099 1 2 5 10 20 57 0 0.099 1 2 5 10 20 65.96 0 0.3 0.4 0.5 0.6 0.7 0.8 1

Dry Light Heavy Dry Light Heavy 0 0.4 0.8 1.2 1.6 2 2.4 9.2

(a) Observation (b) Forecast (c) Probability No Precip.



Forecast penalised for not predicting heavy rain over (generally dry) northern Europe in summer

DiagnosticsR&J 48

Summary

● Diagnostics must become ever more powerful and precise● Higher resolution● More observations● Better forecast models● Smaller signal-to-noise ratio

● Initial Tendency diagnosis of model error● Highlights Local causes – Hints at solutions● Higher statistical significance● Zonal-mean errors● Asian monsoon errors● MJO errors

● Scale-dependent verification● Separation of planetary waves (tropical origin?) and synoptic

● Diagnostic Verification● Careful design of scores can aid error detection: Precipitation

Documents

Diagnostics at ECMWF