Embed Size (px)
Citation preview
Weather Forecast
Verification
Training- Recent Advancements -
Pertti NurmiMeteorological Research & Development
Finnish Meteorological Institute
Laurie WilsonMeteorological Service of Canada
1
trainingmodules
2
0. Introduction
5 July 2007CALMet - Beijing
Forecast Verification Training
2680 + 302800
= 96,8 %2680 + 302800
= 96,8 %
TornadoTornado observed
forecastYes No fc S
Yes 30 70 100
No 20 2680 2700
obs S 50 2750 2800
Verification(training)History, 1884
Tornados in USA”The Finlay case”
0
2750 + 02800
= 98,2 %
Never forecasta Tornado
2750 + 02800
= 98,2 %
Never forecasta Tornado !
TornadoTornado observed
forecastYes No fc S
Yes 0 0 0
No 50 2750 2800
obs S 50 2750 2800
Hypothesis
5 July 2007CALMet - Beijing
Forecast Verification Training
Goals of Verification Training: Understand the basic properties and relationships of
(common) verification measures
Learn to extract useful information from (graphical) verification results
Increase interest in forecast verification and the methods
Apply verification in everyday fc practice at every fc office
0
Generally HEAVILY neglected topic in training activities !
5 July 2007CALMet - Beijing
Forecast Verification Training
”WMO WWRP/WGNE JWGV” (*
Harold Brooks(NSSL, USA; not shown)
Barbara Brown(NCAR, USA; Chair)
Barbara Casati(MSC, Canada)
Ulrich Damrath(DWD, Germany)
Beth Ebert(BMRC, Australia)
Anna Ghelli (ECMWF)
Pertti Nurmi (FMI, Finland)
David Stephenson(U. Reading, UK; not shown)
Clive Wilson (UKMO, UK)
Laurie Wilson(MSC, Canada)
0
*) WMO ~ World Meteorological Organization WWRP ~ World Weather Research Program WGNE ~ Working Group on Numerical Experimentation JWGV ~ Joint Working Group on Verification
InternationalActivities
5 July 2007CALMet - Beijing
Forecast Verification Training
Goals of JWGV: • Cooperation between forecasters, users and verifiers
• Development and application of verification methodologies
• Promote awareness of the importance of verification via training
Recent activities of JWGV:• Research and training workshops on verification methods • Recommendations for verification methodologies
• Test-bed activities
• Beijing 2008 Olympics Real-Time Forecast Verification (RTFV)
• THORPEX and TIGGE
• EUMETCAL verification training modules
”WMO WWRP/WGNE JWGV”
2
0
1
InternationalActivities
5 July 2007CALMet - Beijing
Forecast Verification Training
www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html
JWGV website Google search:
“Forecast > verification”
gives this as the first prompt !
0
5 July 2007CALMet - Beijing
Forecast Verification Training
www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/lecture_notes/index.html
1
5 July 2007CALMet - Beijing
Forecast Verification Training
1
Argentina, Australia , Brasil, Canada, Checz Republic, China, Croatia, Denmark, Ethiopia, Finland, France, Germany, Hong Kong, India, Italy, Kenya, Lithuania, Nigeria, Norway, Romania, Senegal, Slovakia, South Africa, South Korea, Sweden, Switzerland, Ukraine, USA
31 students,One / country
5 July 2007CALMet - Beijing
Forecast Verification Training
Components:
i. Tutorial ~ 2 days
ii. Scientific workshop and presentations ~ 3 days
i. Tutorial contents:
• Use/interpretation of “standard” verification methods• Application of new methodologies (e.g. spatial forecast verification)
• Hands-on laboratory exercises
• ”R” software (verification components) training
ii. Workshop contents:
• Invited keynotes & contributed presentations
• User-oriented forecast verification
• Verification of extreme/ high impact weather, EPS, uncertainty …
1
5 July 2007CALMet - Beijing
Forecast Verification Training
Tutorial contents:
i. Introduction to R (Matt Pocernich)
ii. Basic Verification Concepts (Barb Brown)
iii. Verification of Continuous Predictands (Barbara Casati)
iv. Verification of Categorical Predictands (Pertti Nurmi)
v. Verification of Probabilistic Predictands and Ensemble Forecasts (Beth Ebert)
vi. Confidence Intervals and Hypothesis Tests (Simon Mason)
vii. Projects (Laurie Wilson)
1
5 July 2007CALMet - Beijing
Forecast Verification Training
1
www.r-project.org
(i)
5 July 2007CALMet - Beijing
Forecast Verification Training
Project No. 1: Verification of ensemble-based PoP forecasts (Dataset: Tong/ Hong Kong)
Calculate the predictand… Verify the POP forecasts... Prepare and plot a reliability table... Compute the BS…
Project No. 2: Temperature verification (Dataset: Diaconu/ Romania)
Choose a verification problem… Look at the data… Produce and examine scatter plots…Choose verification scores…
Project 3: Cloudiness forecasts in 3 categories (Dataset: Diaconu/ Romania)
Choose a verification issue... Look at the data… Produce contingency tables... Choose verification measures…
Project No. 4: Wind speed forecasts(Dataset: Moreau/ Canada)Choose the verification problem… Look at the data… Plot
forecast-observation scatterplots... Choose verification measures... Calculate the scatter index...
Project No. 5 Climatological analysis(Dataset: Korecha/ Ethiopia)Produce box and whisker plots… Calculate the annual
statistics… Do the same for each month separately...
Project No. 6: Precipitation forecasts(Dataset: Suaya/ Argentina)Look at the data… Prepare histogram plots… Pose the
relevant verification questions… Carry out the verification and produce graphs to display your results…
1(vii) Projects
Laurie WilsonEnvironment Canada
deterministic
5 July 2007CALMet - Beijing
Forecast Verification Training
1 “Probably the best trainingcourse I ever had.
Thank you to all of you! “
Overall evaluationby students:
> 2.5 / 3
5 July 2007CALMet - Beijing
Forecast Verification Training
2What’s included in the modules:
1. Introduction Goals and general guidelines
2. Continuous variables
3. Categorical events
4. Probability forecasts
Glossary + References
+ Practical exercises ~ Quiz !
finished spring 2006
finished winter 2007
finished summer 2007
CAL trainingmodules
5 July 2007CALMet - Beijing
Forecast Verification Training
2 www.eumetcal.org
Demonstration on Sat 10:10-11:00 WELCOME TO TEST
www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm
… produced by Pertti Nurmi, Laurie Wilson & Sigbritt Näsman, 2006-07
CAL modules !
2
Demonstration on Sat 10:10-11:00 WELCOME TO TEST
www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm
2
”Help button”
”Glossary button”
”Communication tool”
”Return to Index”… open browser !
5 July 2007CALMet - Beijing
Forecast Verification Training
Jolliffe and Stephenson, Eds. (2003): Forecast Verification:A Practitioner’s Guide, Wiley & Sons.
Nurmi (2003): Recommendations on the verification of local weather forecasts. ECMWF Tech. Mem. 430.
Wilks (2006): Statistical Methods in the Atmospheric Sciences, Chapter 7 (Forecast Verification). Academic Press.
WMO Joint Working Group on Verification Website: www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html
ECMWF Verification Methods Training Workshop Lectures: www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/
lecture_notes/index.html
EUMETCAL Forecast Verification Learning Modules: www.eumetcal.org/resources/ukmeteocal/verification/www/
english/courses/msgcrs/index.htm
2
0
1
References:
5 July 2007CALMet - Beijing
Forecast Verification Training
There is nothing as difficult as
forecasting the weather (correctly) ...
… … exceptexcept (proper)(proper) forecast forecast
verification !verification !- Allan Murphy, RIP- Allan Murphy, RIP
Training is one key issue !
Finishing motto
Pertti Nurmi
Meteorological Research & Development
Finnish Meteorological Institute
Thank You
