Verification of Tropical Cyclone Forecasts

Beth Ebert (BOM)

Barb Brown (NCAR)

Laurie Wilson (RPN)

Tony Eckel (ERT)

8th TIGGE Working Group Meeting

22-24 February 2010, Geneva

What kind of TC forecasts?

Deterministic• TC track

• Intensity – maximum wind– central pressure– temporal trend (rapid

intensification)

• Wind field / radii• Precipitation• Storm surge• Temporal consistency

Ensemble• Track distribution

– strike prob., cone of uncertainty

• Intensity distribution– mean / median– spread– 90th percentile

• Prob (wind > threshold)• Prob (precip > threshold)• Prob (surge > threshold)• Temporal consistency

Different users need different kinds of verification information

• Public and emergency managers– Simple, graphical– Focus on impact

• Forecasters– Information on how to interpret forecasts– Timing errors

• Modellers– Systematic errors

• How to improve the model

• How to improve ensemble distribution / spread

Quality of deterministic forecasts

• What are the track errors (along-track, cross-track)?

• What are the intensity errors?• Are temporal intensity trends correctly predicted?• What is the error in timing of landfall?• What is the error in forecast maximum wind

(rain)?– Multi-day total precipitation

• Is the spatial distribution of wind (rain) correct?

Quality of ensemble forecasts

• Does the ensemble enclose the observed track?• Are the ensemble probabilities skillful and reliable

predictions for – strike probability (track)– intensity (max wind, central pressure)– wind– precipitation

• Does the ensemble produce an appropriate spread for these variables?

Data issues

• Verification data– Data sources

• Best track, Dvorak, surface instruments, radar, …– Problems measuring in extreme conditions

• Forecast data– Size of ensemble– Model grid resolution– Details of cyclone tracker

• Reference forecasts– Statistical forecast (e.g., CLIPER)

Verification methods for deterministic TC forecasts

• Example 1 – visual comparison

Courtesy Noel Davidson, BOM

Analysis Track/intensity verificationModel

Verification methods for deterministic TC forecasts

• Example 2 – along-track and cross-track errors

Courtesy James Franklin, NHC

Verification methods for deterministic TC forecasts

• Example 3 – cumulative distribution of track errors

Courtesy James Franklin, NHC

Verification methods for deterministic TC forecasts

• Example 4 – distribution of intensity errors

HFIP High-Resolution Hurricane Test – DTC Final Report Sept 2009

Verification methods for deterministic TC forecasts

• Example 5 – rapid intensification

Count

Hours since onset of observed RI

Observed

High resolution model

Low resolution model

AOML / WRF – 69 cases

HFIP High-Resolution Hurricane Test – DTC Final Report Sept 2009

Verification methods for deterministic TC forecasts

• Example 6 – rain intensity distribution

Marchok et al. 2008

Verification methods for ensemble TC forecasts

• Example 1 – visual comparison

Verification methods for ensemble TC forecasts

• Example 2 – probabilistic scores and methods

MOGREPS 120 h forecast forstrike probability (within 120 km)

TC Anja, November 2009

fake

^

Verification methods for ensemble TC forecasts

• Example 3 – spread of track and intensity forecasts

20-member FIM ensemble

track intensitytrack intensity

New approaches for verifying TCs

• Spatial verification methods

Precipitation andwind fields

Storm characteristics • location• size• intensity• shape, etc.

New approaches for verifying ensemble TC forecasts

• Minimum spanning tree – multi-variate rank histogram

• Ensemble of object properties– ensemble mean object properties– distributions – use standard

methods for ensembles and probability forecasts

– relationship of TC genesis to the number of ensemble members predicting the TC at day 1+?

– correspondence ratio

observation

ensembleforecastensembleforecast

Reporting guidelines

• Provide all relevant information– Model(s), grid, range of dates, forecast lead times,

verification data source, etc.

• Aggregation and distribution of results

• Confidence intervals / uncertainty

Verification tools for TCs

• US Developmental Testbed Centre (DTC) – Tracker (Marchok)– Code for verification of TC track and intensity

• Other international tool sets??

Document on TC verification – commented literature review to be written this year

Contents

1. Introduction

2. Verification strategy

3. Reference data

4. Verification methods

5. Reporting guidelines

6. Summary

References

Appendices:

a. Brief description of scores

b. Guidelines for computing aggregate statistics

c. Confidence intervals for verification scores

d. Examples of graphical verification products