Verification of Precipitation Areas
Beth EbertBureau of Meteorology Research Centre
Melbourne, [email protected]
Outline
1. “Eyeball” verification - use of maps
2. QPF verification using gridpoint match-ups
3. Space-time verification of pooled data
4. Entity-based (rain “blob”) verification
5. Summary
1. “Eyeball” verification - some examples
Accumulated rain over eastern Germany and western Poland, 4-8 July 1997
WWRP Sydney 2000 Forecast Demonstration Project
RAINVAL - Operational verification of NWP QPFs
2. QPF verification using (grid)point match-ups
All verification statistics can be applied to spatial estimates when treated as a matched set of forecasts/observations at a set of individual points!
. . . .. .... . . . .. ...
. . . .. ...
. . . .. ... . . . .. ... . . . .. ...
. . . .. .... . . . .. ..
. . . .. .... . . . .. ...
. . . .. ...
. . . .. ... . . . .. ... . . . .. ...
. . . .. ... . . . .. ...
Observed Forecast
Method 1: Analyze observations onto a grid
Observed Forecast
Method 2: Interpolate model forecast to station locations
Q: Which verification approach is better?
A: It depends!
Arguments in favor of grid:• point observations may not represent rain in local area
• gridded analysis of observations better represents the grid-scale values that a model predicts
• spatially uniform sampling
Use to verify gridded forecasts
Arguments in favor of station locations:• observations are “pure” (not smoothed or interpolated)
Use to verify forecasts at point locations or sets of point locations
Note: Verification scores improve with increasing scale!
Preparation of gridded (rain gauge) verification data:
• Real time vs. non-real time
• Quality control to eliminate bad data
• Mapping procedure:– simple gridbox average– objective analysis (Barnes, statistical interpolation,
kriging, splines, etc.)
• Map observations to model grid
• Model intercomparison - map to common grid
• Uncertainty in gridbox values
Continuous statistics quantify errors in forecast rain amount
Score What it measures
Mean difference Forecast bias
Mean absolute error Average error magnitude
RMS error Error magnitude, greater emphasis tooutliers
Correlation coefficient Correspondence of forecast to observedrain pattern
Categorical statistics quantify errors in forecast rain occurrence
Score What it measures
Accuracy Correspondence of rain and no-rainareas
Bias score Tendency to under- or over-forecastrain area (occurrence)
Probability of detection Ability to predict observed rain
False alarm ratio Tendency to predict rain when/ wherenone occurred
Threat score Penalizes both misses and false alarms
Equitable threat score As above, accounts for regime
Hanssen & Kuipers (trueskill) score
Accuracy for events + accuracy fornon-events - 1
Heidke skill score Can use multiple rain categories
Verification of QPFs from NWP models
Vary rain threshold from light to heavy
Equitable threat scoreBias score
Verification of NWP QPFs over Germany
equitable threatw.r.t. chance
equitable threatw.r.t. persistence
Verification of nowcasts in Sydney 2000 FDP
—— Nowcast- - - - Persistence
3. Space-time QPF verification
(a) Pool forecasts and observations in SPACE AND TIME summary statistics
Caution: Results may mask regional and/or seasonal differences
annual
winter
summerModel performance in Australian tropics
(b) Pool forecasts and observations in SPACE but NOT TIME maps of temporal statistics
1.0-1.1
1.1-1.2
1.2-1.5
1.5-2.0
2.0-3.0
3.0-4.0
0.0-0.2
0.2-0.4
0.4-0.6
0.6-0.8
0.8-0.9
0.9-1.0
No data
Bias scoreJune 1995-November 1996
(c) Pool forecasts and observations in TIME but NOT SPACE time series of spatial statistics
OBSLAPS 24 hLAPS 36 hLAPS 48 h
1-30 Apr 2001Australian region
Limitations to QPF verification using (grid)point match-ups:
• Some seemingly good verification statistics may result from compensating errors
– too much rain in one part of the domain offset by too little rain in another part of the domain
– interseasonal rainfall variation captured but shorter period variation not captured
• Conservative forecasts are rewarded
• Some rain forecasts look quite good except for the location of the system; unfortunately, traditional verification statistics severely penalize these cases
4. Entity-based QPF verification (rain “blobs”)
Verify the properties of the forecast rain system against the properties of the observed rain system:
• location• rain area• rain intensity (mean, maximum)
Observed Forecast
Define a rain entity by a Contiguous Rain Area (CRA), a region bounded by a user-specified isohyet.
Some possible choices of CRA thresholds are:
1 mm d-1: ~ all rain in system5 mm d-1: “important” rain 20 mm d-1: rain center
Observed Forecast
Determining the location error:
• Horizontally translate the QPF until the total squared error between the forecast and the analysis (observations) is minimized in the shaded region.
• The displacement is the vector difference between the original and final locations of the forecast. Arrow shows optimum shift.
Observed Forecast
CRA error decomposition
The total mean squared error (MSE) can be written as:
MSEtotal = MSEdisplacement + MSEvolume + MSEpattern
The difference between the mean square error before and after translation is the contribution to total error due to displacement,
MSEdisplacement = MSEtotal – MSEshifted
The error component due to volume represents the bias in mean intensity,
where and are the CRA mean forecast and observed values after the shift.
The pattern error accounts for differences in the fine structure of the forecast and observed fields,
MSEpattern = MSEshifted - MSEvolume
2)( XFMSEvolume
XF
Example: Nowcasts from Sydney 2000 FDP
Example: Australian regional NWP model
Rain area Mean rain intensity
North of 25°S
South of 25°S
Maximum rain intensity Rain volume
Displacement error
Event forecast classification
Two most important aspects of a “useful” QPF:
• Location of predicted rain must be close to the observed location
• Predicted maximum rain rate must be “in the ballpark”
Forecast Maximum Rain Rate
TooLittle
Approx.Correct*
TooMuch
Close**Under-estimate
HitOver-estimateDisplacement
of forecastrain pattern Far
MissedEvent
MissedLocation
FalseAlarm
Example: Proposed event forecast criteria for 24h NWP QPFs
Good location: Forecast rain system must be within 2° lat/lon or one effective radius of the rain system, but not farther than 5° from the observed location
Good intensity: Maximum rain rate must be within one category of observed (using rain categories of 1-2, 2-5, 5-10, 10-25, 25-50, 50-100, 100-150, 150-200, >200 mm d-1)
Event forecast classification
Australian 24h QPFs from BoM regional model, July 1995-June 1999 (2066 events)
Error decomposition
Australian 24h QPFs from BoM regional model, July 1995-June 1999 (2066 events)
Advantages of entity-based QPF verification:
• intuitive, quantifies “eyeball” verification
• addresses location errors
• allows decomposition of total error into contributions from location, volume, and pattern errors
• rain event forecasts can be classified as "hits", "misses", etc.
• does not reward conservative forecasts
Disadvantages of entity-based verification:
• more than one way to do pattern matching (i.e., not 100% objective
• forecast must resemble observations sufficiently to enable pattern matching
5. Summary
Spatial QPF success* can be qualitatively and quantitatively measured in many ways, each of which tells only part of the story
*Note: “success” depends on the requirements of the user!!
Objective
Subjective
Point Area
(grid)point match-ups
Precision
Meaningmaps
entities