Weather Forecast Verification Training - Recent Advancements - Pertti Nurmi Meteorological Research...

Weather Forecast

Verification

Training- Recent Advancements -

Pertti NurmiMeteorological Research & Development

Finnish Meteorological Institute

Laurie WilsonMeteorological Service of Canada

1

trainingmodules

2

0. Introduction

5 July 2007CALMet - Beijing

2pertti.nurmi@fmi.fi

Forecast Verification Training

2680 + 302800

= 96,8 %2680 + 302800

= 96,8 %

TornadoTornado observed

forecastYes No fc S

Yes 30 70 100

No 20 2680 2700

obs S 50 2750 2800

Verification(training)History, 1884

Tornados in USA”The Finlay case”

0

2750 + 02800

= 98,2 %

Never forecasta Tornado

2750 + 02800

= 98,2 %

Never forecasta Tornado !

TornadoTornado observed

forecastYes No fc S

Yes 0 0 0

No 50 2750 2800

obs S 50 2750 2800

Hypothesis

5 July 2007CALMet - Beijing

3pertti.nurmi@fmi.fi

Forecast Verification Training

Goals of Verification Training: Understand the basic properties and relationships of

(common) verification measures

Learn to extract useful information from (graphical) verification results

Increase interest in forecast verification and the methods

Apply verification in everyday fc practice at every fc office

0

Generally HEAVILY neglected topic in training activities !

5 July 2007CALMet - Beijing

4pertti.nurmi@fmi.fi

Forecast Verification Training

”WMO WWRP/WGNE JWGV” (*

Harold Brooks(NSSL, USA; not shown)

Barbara Brown(NCAR, USA; Chair)

Barbara Casati(MSC, Canada)

Ulrich Damrath(DWD, Germany)

Beth Ebert(BMRC, Australia)

Anna Ghelli (ECMWF)

Pertti Nurmi (FMI, Finland)

David Stephenson(U. Reading, UK; not shown)

Clive Wilson (UKMO, UK)

Laurie Wilson(MSC, Canada)

0

*) WMO ~ World Meteorological Organization WWRP ~ World Weather Research Program WGNE ~ Working Group on Numerical Experimentation JWGV ~ Joint Working Group on Verification

InternationalActivities

5 July 2007CALMet - Beijing

5pertti.nurmi@fmi.fi

Forecast Verification Training

Goals of JWGV: • Cooperation between forecasters, users and verifiers

• Development and application of verification methodologies

• Promote awareness of the importance of verification via training

Recent activities of JWGV:• Research and training workshops on verification methods • Recommendations for verification methodologies

• Test-bed activities

• Beijing 2008 Olympics Real-Time Forecast Verification (RTFV)

• THORPEX and TIGGE

• EUMETCAL verification training modules

”WMO WWRP/WGNE JWGV”

2

0

1

InternationalActivities

5 July 2007CALMet - Beijing

6pertti.nurmi@fmi.fi

Forecast Verification Training

www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html

JWGV website Google search:

“Forecast > verification”

gives this as the first prompt !

0

5 July 2007CALMet - Beijing

7pertti.nurmi@fmi.fi

Forecast Verification Training

0

5 July 2007CALMet - Beijing

8pertti.nurmi@fmi.fi

Forecast Verification Training

www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/lecture_notes/index.html

1

5 July 2007CALMet - Beijing

9pertti.nurmi@fmi.fi

Forecast Verification Training

1

Argentina, Australia , Brasil, Canada, Checz Republic, China, Croatia, Denmark, Ethiopia, Finland, France, Germany, Hong Kong, India, Italy, Kenya, Lithuania, Nigeria, Norway, Romania, Senegal, Slovakia, South Africa, South Korea, Sweden, Switzerland, Ukraine, USA

31 students,One / country

5 July 2007CALMet - Beijing

10pertti.nurmi@fmi.fi

Forecast Verification Training

Components:

i. Tutorial ~ 2 days

ii. Scientific workshop and presentations ~ 3 days

i. Tutorial contents:

• Use/interpretation of “standard” verification methods• Application of new methodologies (e.g. spatial forecast verification)

• Hands-on laboratory exercises

• ”R” software (verification components) training

ii. Workshop contents:

• Invited keynotes & contributed presentations

• User-oriented forecast verification

• Verification of extreme/ high impact weather, EPS, uncertainty …

1

5 July 2007CALMet - Beijing

11pertti.nurmi@fmi.fi

Forecast Verification Training

Tutorial contents:

i. Introduction to R (Matt Pocernich)

ii. Basic Verification Concepts (Barb Brown)

iii. Verification of Continuous Predictands (Barbara Casati)

iv. Verification of Categorical Predictands (Pertti Nurmi)

v. Verification of Probabilistic Predictands and Ensemble Forecasts (Beth Ebert)

vi. Confidence Intervals and Hypothesis Tests (Simon Mason)

vii. Projects (Laurie Wilson)

1

5 July 2007CALMet - Beijing

12pertti.nurmi@fmi.fi

Forecast Verification Training

1

www.r-project.org

(i)

5 July 2007CALMet - Beijing

13pertti.nurmi@fmi.fi

Forecast Verification Training

1(ii)

5 July 2007CALMet - Beijing

14pertti.nurmi@fmi.fi

Forecast Verification Training

1(ii)

5 July 2007CALMet - Beijing

15pertti.nurmi@fmi.fi

Forecast Verification Training

1(iii)

5 July 2007CALMet - Beijing

16pertti.nurmi@fmi.fi

Forecast Verification Training

1(iii)

5 July 2007CALMet - Beijing

17pertti.nurmi@fmi.fi

Forecast Verification Training

1(iii)

5 July 2007CALMet - Beijing

18pertti.nurmi@fmi.fi

Forecast Verification Training

1(iv)

5 July 2007CALMet - Beijing

19pertti.nurmi@fmi.fi

Forecast Verification Training

1(iv)

5 July 2007CALMet - Beijing

20pertti.nurmi@fmi.fi

Forecast Verification Training

1(v)

E.Ebert@bom.gov.au

5 July 2007CALMet - Beijing

21pertti.nurmi@fmi.fi

Forecast Verification Training

1(v)

5 July 2007CALMet - Beijing

22pertti.nurmi@fmi.fi

Forecast Verification Training

1(v)

5 July 2007CALMet - Beijing

23pertti.nurmi@fmi.fi

Forecast Verification Training

1(v)

5 July 2007CALMet - Beijing

24pertti.nurmi@fmi.fi

Forecast Verification Training

1(vi)

5 July 2007CALMet - Beijing

25pertti.nurmi@fmi.fi

Forecast Verification Training

Project No. 1: Verification of ensemble-based PoP forecasts (Dataset: Tong/ Hong Kong)

Calculate the predictand… Verify the POP forecasts... Prepare and plot a reliability table... Compute the BS…

Project No. 2: Temperature verification (Dataset: Diaconu/ Romania)

Choose a verification problem… Look at the data… Produce and examine scatter plots…Choose verification scores…

Project 3: Cloudiness forecasts in 3 categories (Dataset: Diaconu/ Romania)

Choose a verification issue... Look at the data… Produce contingency tables... Choose verification measures…

Project No. 4: Wind speed forecasts(Dataset: Moreau/ Canada)Choose the verification problem… Look at the data… Plot

forecast-observation scatterplots... Choose verification measures... Calculate the scatter index...

Project No. 5 Climatological analysis(Dataset: Korecha/ Ethiopia)Produce box and whisker plots… Calculate the annual

statistics… Do the same for each month separately...

Project No. 6: Precipitation forecasts(Dataset: Suaya/ Argentina)Look at the data… Prepare histogram plots… Pose the

relevant verification questions… Carry out the verification and produce graphs to display your results…

1(vii) Projects

Laurie WilsonEnvironment Canada

Lawrence.Wilson@ec.gc.ca

deterministic

5 July 2007CALMet - Beijing

26pertti.nurmi@fmi.fi

Forecast Verification Training

1 “Probably the best trainingcourse I ever had.

Thank you to all of you! “

Overall evaluationby students:

> 2.5 / 3

5 July 2007CALMet - Beijing

27pertti.nurmi@fmi.fi

Forecast Verification Training

2What’s included in the modules:

1. Introduction Goals and general guidelines

2. Continuous variables

3. Categorical events

4. Probability forecasts

Glossary + References

+ Practical exercises ~ Quiz !

finished spring 2006

finished winter 2007

finished summer 2007

CAL trainingmodules

5 July 2007CALMet - Beijing

28pertti.nurmi@fmi.fi

Forecast Verification Training

2 www.eumetcal.org

5 July 2007CALMet - Beijing

29pertti.nurmi@fmi.fi

Forecast Verification Training

2 www.eumetcal.org

Demonstration on Sat 10:10-11:00 WELCOME TO TEST

www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm

… produced by Pertti Nurmi, Laurie Wilson & Sigbritt Näsman, 2006-07

CAL modules !

2

Demonstration on Sat 10:10-11:00 WELCOME TO TEST

www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm

2

”Help button”

”Glossary button”

”Communication tool”

”Return to Index”… open browser !

5 July 2007CALMet - Beijing

32pertti.nurmi@fmi.fi

Forecast Verification Training

Jolliffe and Stephenson, Eds. (2003): Forecast Verification:A Practitioner’s Guide, Wiley & Sons.

Nurmi (2003): Recommendations on the verification of local weather forecasts. ECMWF Tech. Mem. 430.

Wilks (2006): Statistical Methods in the Atmospheric Sciences, Chapter 7 (Forecast Verification). Academic Press.

WMO Joint Working Group on Verification Website: www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html

ECMWF Verification Methods Training Workshop Lectures: www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/

lecture_notes/index.html

EUMETCAL Forecast Verification Learning Modules: www.eumetcal.org/resources/ukmeteocal/verification/www/

english/courses/msgcrs/index.htm

2

0

1

References:

5 July 2007CALMet - Beijing

33pertti.nurmi@fmi.fi

Forecast Verification Training

There is nothing as difficult as

forecasting the weather (correctly) ...

… … exceptexcept (proper)(proper) forecast forecast

verification !verification !- Allan Murphy, RIP- Allan Murphy, RIP

Training is one key issue !

Finishing motto

Pertti Nurmi

Meteorological Research & Development

Finnish Meteorological Institute

Thank You