Huug van den Dool and Suranjana Saha Prediction Skill and Predictability in CFS

Huug van den Dool and Suranjana Saha

Prediction Skill and Predictability

in CFS

Definitions Prediction Skill and Predictability

Opinion: Literature fuzzies up ‘predictability’ vs ‘prediction skill’

Definition 1: Evaluation of skill of real time prediction; the old-fashioned way. Problems: a) Sample size! , b) Wait a long time(and funding agents are impatient)

Definition 1: Evaluation of skill of real time prediction; the old-fashioned way. Definition 2: Evaluation of skill of hindcasts; hard, not impossible.Problems: a) Sample size, b) ‘honesty’ of hindcasts

Definition 1: Evaluation of skill of real time prediction; the old-fashioned way.

Definition 2: Evaluation of skill of hindcasts; hard, not impossible.

Definition 3: Predictability of the 1st kind (~ sensitivity due to uncertainty in initial conditions)

Definition 1: Evaluation of skill of real time prediction; the old-fashioned way. Sample size!

Definition 2: Evaluation of skill of hindcasts; hard, not impossible

Definition 3: Predictability of the 1st kind (~ sensitivity due to uncertainty in initial conditions)

Definition 4: Predictability of the 2nd kind due to variations in external boundary conditions (AMIP; Potential Predictability; Reproducibility; Madden’s approach)

Predictability (theoretical/intrinsic) is a ceiling for actual prediction skill.

Any other ‘kinds’ of predictability?

CFS forecast:X (space, lead, member ,year)

• Space is 2.5oX2.5o (Z500) or 1oX2o (SST/mask), or 1.875 by Gaussian (Soilw, T2m, Precip)

• Basic data used is monthly mean• Lead = 0, 8 in units of months; member = 1, 15• Year = 1981 – 2003 (increases annually)• Example: ‘Initial’ Month is August (= lead 0); • Note IC is Jul 11/21/Aug 1 for SST, and Jul 09-13/ 19-23

/ Jul 30-Aug3 for atmosphere and soil. • ‘Member’ 16 is ensemble average• ‘Member’ 17 is matching observed field• X = ( Z500, SST, Soilw, T2m, Precip)

ASPECTS

• Prediction skill (member i vs member 17)• Predictability (member i vs member j)• Monthly mean• Seasonal mean• Ensemble average• Predictability of 1st kind only.

Two types of climatology plus complications

• Xclim_mdl (space, lead) is average over years and (14 or 15) members, depending.

• Xclim_verif (space, lead) is ave over (same) years for either member 17, or member i, i=15.

• Anomaly = X minus Xclim, whichever is relevant• Systematic error (SE) is automatically corrected

by the above• CV of the SE correction (exclude from Xclim the

member and the year to be verified). Not trivial.

Prediction Skill

Monthly

0

0.1

0.2

0.3

0.4

0.5

a s o n d j f m a

Anom.Corr vs LeadZ500 NH

0

0.1

0.2

0.3

0.4

0.5

a s o n d j f m a

monthly

monthly ens ave


0

0.1

0.2

0.3

0.4

0.5

a s o n d j f m a

monthly

monthly ens ave

seasonal ens ave


0

0.1

0.2

0.3

0.4

0.5

a s o n d j f m a

monthly

monthly ens ave

seasonal ens ave

prdctblty seasonal ens ave


0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

a s o n d j f m a

skill monthly

skill monthly ens ave

skill seasonal ens ave


Anom.Corr vs LeadSST TR (20S-20N;0-360)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

a s o n d j f m a

skill monthly




Anom.Corr vs LeadSST Nino34 (5S-5N;170W-120W)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

a s o n d j f m a

skill monthly




Anom.Corr vs LeadSST NorthAtl (30-60N;80W-0W)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

a s o n d j f m a

skill monthly




Anom.Corr vs LeadSST Tropical Atl (20S-20N)

Seasonal Prediction Skill Z500 NHCFS 1981-2003

0

0.1

0.2

0.3

0.4

0.5

djf

jfm fma

mam am

j

mjj jja jas

aso

son

ond

ndj

djf

jfm fma

mam am

j

mjj

corr

elat

ion

Jan Mar May Jul Sep Nov

Seasonal Predictability Z500 NHCFS 1981-2003

0

0.1

0.2

0.3

0.4

0.5

djf

jfm fma

mam am

j

mjj jja jas

aso

son

ond

ndj

djf

jfm fma

mam am

j

mjj

corr

elat

ion


0

0.1

0.2

0.3

0.4

0.5

0.6

djf fma amj jja aso ond djf fma amj


Seasonal Prediction Skill T2m (NH-landCFS 1981-2003

0

0.1

0.2

0.3

0.4

0.5

0.6



Seasonal Predictability T2m (NH-land)CFS 1981-2003

0

0.1

0.2

0.3

0.4

0.5

0.6



Seasonal Prediction Skill Prc (NH-landCFS 1981-2003

0

0.1

0.2

0.3

0.4

0.5

0.6



Seasonal Predictability Prc (NH-landCFS 1981-2003

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1



Seasonal Predictability (NH-landSoilw CFS 1981-2003

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1



Seasonal Prediction Skill (NH-landSoilw CFS 1981-2003

-60-50-40-30-20-10

010

corr

elat

ion(

%)

0 1 2 3 4 5 6 7 8 9 10111213target month (jan=1)

NCEP model observed ens average

W, T correlation at lag 1 mo

Conclusions (monthly data)

• CFS data is a goldmine.• CFS has enough (?) data for forecast evaluation

(and diagnostics)• Member i vs member j unifies predictability of 1st

and 2nd kind in CFS output • CFS has some prediction skill. In order of skill:

SST, {tropical variables}, soilw,T2m, Precip• CFS has some more predictability (as defined), but

ceiling is ‘low’ in mid-latitudes.• Seasonality (no surprise)

To do:

• Identify interdecadal skill source (if any)• Identify soil moisture skill source (are models still

too strong on local effects? How about non-local effects)

• Daily data for the finer temporal scales in skill/predictability.

• Why do models like CFS have predictability in so few d.o.f. (and is that really all there is)

• Further ideas about ‘new’ predictability notions

A case for the importance of knowing the effective number of degrees of freedom (edof) in which we have forecast skill.

Considerations:-) physical models have one clear strength: they can execute the non-linear terms-) a model needs at least 3 degrees of freedom to be non-linear (Lorenz, 1960)-) a non-linear model with nominally a zillion degrees of freedom, but skill in only <= 3 dof is functionally linear in terms of the skill of its forecasts - and, to its detriment, the non-linear terms add random numbers to the tendencies of the modes with predictability.

==> Therefore: Physical models need to have skill in, effectively, > 3 dof before they can be expected to take advantage of non-linearity. (In a forecast setting). ( Note: not any 3 degrees of freedom will do.)

0

20

40

60

80

100co

rrel

atio

n

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30forecast lead (days)

AC(Z) TC(spave(Z))TC(area Z-Below)

SH Z500

95

97.5

100

corr

elat

ion

0 1 2 3forecast lead (days)

AC(Z) TC(spave(Z))TC(area Z-Below)

SH Z500

‘Lingering memory’Cai+Van den Dool(2005); Schemm et al calibration data set,(CFS daily data set will be used also).

Documents

Huug van den Dool and Suranjana Saha Prediction Skill and Predictability in CFS