Spatial Verification Methods for Ensemble Forecasts of Low-Level Rotation in Supercells Patrick S. Skinner 1, Louis J. Wicker 1, Dustan M. Wheatley 1,2,

Spatial Verification Methods for Ensemble Forecasts of Low-Level

Rotation in Supercells

Patrick S. Skinner1, Louis J. Wicker1, Dustan M. Wheatley1,2, and Kent H. Knopfmeier1,2

1 - NOAA/National Severe Storms Laboratory

2 - Cooperative Institute for Mesoscale Meteorological Studies, University of Oklahoma

Thanks To: Thomas Jones, Corey Potvin, Gerry Creager, Jeff Snyder, and Chris Karstens

National Weather Association 40th Annual Meeting

• “Next Day”• May 31st, 2013• 6-18 hour forecast probability swath of updraft helicity

(shading) and SPC storm reports (yellow markers)

Goal is to verify forecasts of regions of thunderstorms

Wheatley et al. 2015

Schwartz et al. 2015

Convection allowing ensemble forecast with mesoscale data assimilation:

Convection allowing ensemble forecast with storm-scale (radar) data assimilation:

• “Next Hour”• May 31st, 2013• 0-1 hour forecast probability swath of vertical vorticity

(shading), reflectivity (contours), and El Reno, OK tornado track (blue outline)

Goal is to verify forecasts of specific thunderstorms

Verification of Low-level Rotation

Overall goal of verification is to reproduce subjective interpretation of ensemble low-level rotation forecasts

• Neither observations or forecasts resolve the hazard being forecast

• NEWS-e horizontal grid spacing is 3 km: Can partially resolve mesocyclones, but not storm-scale processes (i.e. cyclic mesocyclogenesis) (Potvin and Flora 2015)

• The predicted model fields cannot be directly verified (e.g. only have WSR-88D data)

• Requires development of proxies for tornado likelihood in both forecasts and observations

• Best available proxy for tornado occurrence in modeled supercells is the presence of a low-level (z<1km AGL) mesocyclone (Trapp et al. 2005)

• Best available proxy for low-level mesocyclone occurrence in WSR-88D data is low-level azimuthal wind shear (AWS)

Postprocessing Example: 20 May 2013, Moore, OK Supercell

NEWS-e ζ (s-1)25-min Forecast

Valid: 20:10Ens. Mem. 14

KTLX AWS (s-1)

Valid: 20:10

• KTLX azimuthal wind shear field interpolated to model grid

• Postprocessing method: Based on Method for Object-Based Diagnostic Evaluation (MODE; Davis et al. 2006a,b)

Damage track ofMoore Tornado

Contour of 0.001 s-1

final KTLX AWS

Step 1:



KTLX AWS (s-1)

Valid: 20:10

• Apply 3x3 grid point maximum value filter to spread information for both modeled vorticity and AWS fields


Step 2:



KTLX AWS (s-1)

Valid: 20:10

• Apply 5x5 grid point convolution filter to smooth information


Step 3:



KTLX AWS (s-1)

Valid: 20:10

• Threshold smoothed AWS field at 0.001 s-1 and restore unsmoothed values to grid points exceeding threshold

• Threshold smoothed ζ field at 0.003 s-1 and restore unsmoothed values to grid points exceeding threshold


Step 4:

NEWS-e ζ (s-1)1945 - 2045 Forecast

Ens. Mem. 14

KTLX AWS (s-1)1945 - 2045 Rotation Track

• Values may be combined over time to create rotation tracks

• Note: A poorly performing ensemble member is shown to illustrate differences between observations and forecasts


Time Aggregation:

NEWS-e 1945 - 2045 Probabilistic Forecast

KTLX AWS (s-1)1945 - 2045 Rotation Track

• Rotation tracks for each ensemble member may be used to create a probabilistic rotation forecast


Ensemble Aggregation:

20 May 2013: Subjective Verification

Moore tornado genesis at

~2000

Improved probabilistic swaths in successive forecasts for Moore supercell

• Four 60-min NEWS-e forecasts issued at 15-min intervals prior to Moore tornado

• Increasing probabilities along the rotation track of the Moore supercell

• Lower probabilities along the tracks of the weakly tornadic supercell in the center of the domain and nontornadic southern supercell

Forecast Time

20 May 2013: Subjective Verification

• Orange: Model forecast information• Dots represent vorticity maxima in individual

members• Contours are ensemble mean reflectivity of 20, 30,

40, and 50 dBZ

• Gray: WSR-88D information:• Large dot is observed maximum in AWS

• Forecast accuracy degrades with increasing forecast time

• Strong bias in storm speed observed in all members

• Similar to many recent storm-scale data assimilation studies (e.g. Wheatley et al. 2015, Yussouf et al. 2015)

15-min NEWS-e forecast


Object-Based Verification



KTLX AWS (s-1)

Valid: 20:10

Verification object

Verification object

Forecast object

Forecast object

Forecast object

Each low-level mesocyclone isolated

in forecast and verification fields is

considered an object and is compared to all other objects in space

and time

20 May 2013: Object-Based Verification

• Total interest score (Iij; Davis et al. 2006) is calculated for each pair of verification and forecast objects (“ij” represents obs/fcst object)

• 90% of weight is on proximity in space in time

• 10% of weight is on area difference between objects

• Match: Highest total interest between objects, provided Iij > 0.2

• False Alarm: Unmatched forecast objects

• Miss: Unmatched verification objects



15-min NEWS-e Objects

45-min NEWS-e Objects


• Object matches, false alarms, and misses can be used as quadrants in the standard contingency table (e.g. Wilks et al. 2006)

• Allows an Object-based Threat Score (OTS) to be calculated (Johnson et al. 2011)

• OTS scores reproduce subjective assessment of NEWS-e forecasts

• Scores improve with each successive forecast • Scores decrease with increasing forecast time• Results are more dramatic when subdomain containing only the Moore supercell (right panel) is

considered

Initial AWSmaximum observed

Comparison Between Different Cases

“Good”“Average”“Poor”

• OTS score can capture relative differences in quality between different forecasts

El Reno, OK 2013

El Reno, OK 2013

El Reno, OK 2013

El Reno, OK 2013

El Reno, OK 2013

El Reno, OK 2013

Moore, OK 2013

Moore, OK 2013

Moore, OK 2013

Moore, OK 2013

Moore, OK 2013

Moore, OK 2013

Nebraska, 2014

Nebraska, 2014

Nebraska, 2014

Nebraska, 2014

Nebraska, 2014

Nebraska, 2014

T-30 Min

T-15 Min

T-0 Min T-0 Min

T-15 Min

T-30 Min T-30 Min

T-15 Min

T-0 Min

Summary

• Object-based verification can reproduce subjective assessments of forecast quality for ensemble forecasts of low-level rotation

• Object characteristics, such as centroid distance, may be used to quantify specific forecast errors

• Summary object-based threat score can discriminate between relative forecast quality for different cases

Long term goal is to develop automated, near real time capabilities for assessing Warn-on-Forecast system performance

• Quantify improvements resulting from changes in system configuration

• Quantify system performance for varying storm modes and environments

[email protected]


• Characteristics of matched objects can be used to quantify specific errors• Centroid distance between forecast and observed rotation objects in the

Moore supercell captures positive bias in storm speed

Documents

Spatial Verification Methods for Ensemble Forecasts of Low-Level Rotation in Supercells Patrick S. Skinner 1, Louis J. Wicker 1, Dustan M. Wheatley 1,2,