J13.1 MVT – An Automated Mesoscale Verification Tool

Scott A. SandgatheLeah Heiss

University of Washington Applied Physics Laboratory, Seattle, Washington

1. INTRODUCTION*

Traditional verification schemes tend topenalize mesoscale numerical weatherprediction (NWP) systems which realisticallyportray high amplitude, short durationmesoscale phenomena. Small phase, timingor location errors for high amplitude featuresresult in apparent poor performance whentraditional verification schemes based onsynoptic observations or grid point analysesare employed (Brown, 2002). Yet these sameNWP systems provide much more realisticand often more operationally useful depictionsof weather events than smoother global NWPor ensemble-mean systems (Mass, et.al.,2003). Unfortunately, taking into considerationsmall phase or timing errors generally requireslabor-intensive case studies which are unableto address the large number of cases requiredfor reliable verification of sophisticatedmesoscale NWP systems or mesoscaleensemble systems. NWP centers andforecasters need automated, rapid and morerealistic evaluation of mesoscale NWP andensemble performance.

We are developing a mesoscale verificationtool to address these issues. After discussionswith developers and forecasters, we havedecided such a tool should have the followingattributes:

• Automated and flexible requiring onlysimple setup commands

• Adaptable to today’s forecast issues interms of parameters, timing andintensity

• Capable of evaluating structuraldistortion (rotation or dilation) andtiming errors as well as amplitude (orintensity) errors

* Corresponding author address: Scott Sandgathe,Univ. of Washington Applied Physics Lab, 1013NE 40th St., Seattle, WA 98105-6698

• Able to address large numbers ofcases and multiple models rapidly toidentify systematic errors or to detectsmall performance differences

While simple and flexible, any evaluationsystem should also be statistically sound andprovide easily interpreted results. OurMesoscale Verification Tool (MVT) describedbelow is the first iteration of our attempt todevelop a verification tool that contains allthese attributes.

2. VERIFICATION MECHANICS

MVT currently separates forecast error into anamplitude component and a phase (or timing)component following the method of Van Galen(1970) and Hoffman, et.al., (1995). Thistechnique has been demonstrated forprecipitation predictions by Du and Mullen(2000) and Ebert and McBride (2000).Hoffman and colleagues have describedimprovements to this method in severalpublications, the most recent being Nehrkorn,et. al., (2003). We have chosen to follow theoriginal method; however, as it is most easilyadaptable to computer accelerationtechniques required to meet our near-real timeobjectives.

2.1 ‘Full Search’ Technique

The full search technique as developed byVan Galen or Hoffman compares an area(structure) of a specified size or number of gridpoints on the analysis with the correspondingarea on the forecast. The error computationfor this comparison represents the total errorof the forecast for that area. The analysis ‘box’is then moved over adjacent regions on theforecast field, until the error is minimized. Thiserror represents the error in the forecast if thephase would have been correctly predicted,i.e., the amplitude error. The difference of thetotal error and the amplitude error representsthe error due to distortion (Hoffman, 1995).See Figure 1. below. Distortion here is

timing) error; however, it contains componentsdue to rotation and dilation, which we hope toaddress in later releases of MVT. The errorcomputation may be any of a number of errormetrics such as root mean square error, meanabsolute difference, or mean square error(MSE). We use MSE as we have found itperforms more consistently in regions ofweaker gradients.

Figure 1. Full Search Method

2.2 Acceleration Techniques

The University of Washington Short RangeMesoscale Ensemble or SREF (Mass, et. al.,2002) currently has from eight to twenty fivemembers with a 126 by 150 point outer (36km)grid and a 99 by 102 inner (12km) grid.Verification using the full search method, evenwith advanced computer processors, is still tooslow to be utilized as a near real-time system.Consequently, we have investigated a numberof procedures, predominately based on imagematching from the motion picture industry, toaccelerate the verification. We haveimplemented two techniques, the LayeredStructure Algorithm (LSA) and the Inter-BlockMotion Algorithm (IBM) detailed in a reviewarticle by Chan (1993). The combination ofthese two algorithms results in an easy 30 to50 times acceleration in the verification, withessentially no loss in accuracy (<1% of phaseMSE error). This enables a forecaster toquickly verify a series of model predictions,ensuring they will be more willing to considerrecent model performance in their forecasts.

LSA assumes the existence of a locallymonotonic error field and is simply the

technique of searching a large fieldintermittently, i.e., skipping a fixed number ofgrid points, to identify local minima, thensearching in detail around the local minima toidentify the true minimum in the MSE field.See Figure 2. It is a ‘layered’ search and canbe performed in two or more steps dependingon the complexity and size of the search area.In MVT, we use a simple two-step procedurethat captures three local minima by skippingevery other grid point in the search area.

1 1 1 1 1

2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2

1 2 2 1 1 2 2 1 2 2 1

2 2 2 2 2 2 2 2

2 2 2 2 2 2 2 2

1 1 0 1 1

2 2 2 2 2

2 2 2 2 2

1 1 2 2 1 2 2 1 1

2 2 2 2 2

2 2 2 2 2

1 1 1 1 1

Figure 2. Two-step LSA

IBM assumes groups of adjacent points on thegrid are correlated, and consequently, findingthe minima for intermittent points allows you topredict the location of the minima for theadjacent points. It can be used either to skipthe first step of the LSA (for intermediate gridpoints), predicting the location of the minimabased on the surrounding points, or to skip theLSA entirely, by assuming the minima is givenby the average of the surrounding points. Inthe case of MVT, we simply perform a two-step LSA on every other grid point, thencompute the location of the minima for theremaining points from the surrounding fourgrid points. In addition to being very rapid withlittle loss in accuracy, it tends to dampen noisein the computed phase shift.

3. VERIFICATION TOOLS

The mesoscale verification tool has two web-based graphical user interfaces; theMesoscale Verification Tool (MVT) intendedfor forecaster use, and Mesoscale DataManipulator (MDP) intended for developeruse. Both use the same search andverification engines described in section 2.

3.1 MESOSCALE VERIFICATION TOOL

The MVT is a web-based tool that is part of aforecaster tool kit (“MURI Uncertainty Monitor”or MUM) being developed by the University ofWashington for the U.S. Navy. The MVTallows a forecaster to quickly select anumerical prediction for any ensemblemember, model domain (12 or 36km grid),parameter, level, and forecast hour and verifyit against the latest analysis. MVT includes amanual mode that allows the forecaster toquickly select a specific feature or region to beverified. It also allows the forecaster to goback in time and verify historical cases. TheMVT outputs both tabular and spatial maps oferror fields including MSE and phase error.The spatial maps are overlain with phase errorvectors depicting the phase shift betweenstructures on the analysis and on the forecast.The GUI and resultant output are shown inFigures 3 and 4.

Figure 3. MVT GUI

Figure 4. MVT with phase error vectors

3.2 Mesoscale Data Manipulator

The MDP is accessed via a powerful web-based GUI that allows the user to set up andexecute large verification cases. The user ispresented with the screen shown in Figure 5below.

Figure 5. GUI for the MDP

Selectable features of the MDP are (all basedon the current University of WashingtonSREF):

• Range of dates from one day to entiredata record.

• Model initialization time - 00Z or 12Z• Model domain - 36 or 12km• Verifying field• Ensemble members to verify**

• Forecast hour to verify**• Time lagged verification**• Forecast parameters to verify**• Output desired - Table, RMSE,

Vectors**• Search mechanics

** Multiple selections allowed for theseparameters.

The user can quickly manipulate the GUI toexecute complex and large verificationsequences. The GUI allows the user to submitthe verification sequence for execution on thedata server. When the verification is complete,the user is notified automatically by e-mail andgiven a web address that contains tabulatedresults and figures.

4. DEMONSTRATION CASES

Several demonstration cases will be presentedat the meeting.

5. SUMMARY

MVT and MDP are powerful web-basedverification tools developed to improveforecaster and developer evaluation ofmesoscale NWP products. At the time of thissubmission, MVT had been extensively tested;however, many MDP features had not beencompletely tested. While it is clear that currentverification techniques based on synopticobservations or labor intensive case studiesare inadequate to rigorously evaluatemesoscale NWP systems, it hasn’t beendemonstrated that the MVT phase-basedverification techniques are the solution to thisproblem. Tools in hand, we hope todemonstrate their usefulness over the comingmonths.

6. ACKNOWLEDGEMENTS

This work was supported by ONR and theDoD Multidisciplinary University Research

Initiative (MURI) program administered by theOffice of Naval Research under GrantsN00014-01-10745 and N00014-01-G-0460.

7. REFERENCES

Brown, B., 2002: Development of an Object-basedDiagnostic Approach for QPF Verification. USWRPScience Symposium, April 2002.

Chan, E., 1993: Review of Block Matching BasedMotion Estimation Algorithms for VideoCompression. CCECE/CCGEI.

Du, J., and S. L. Mullen, 2000: Removal ofDistortion Error from an Ensemble Forecast. Mon.Wea. Rev., 128, 3347-3351.

Ebert, E. E., and J. L. McBride, 2000: Verification ofprecipitation in weather systems: determination ofsystematic errors. J. Hydro., 239, 179-202.

Hoffman, R. N., Z. Liu, J.-F. Louis, and C. Grassotti,1995: Distortion representation of forecast errors.Mon. Wea. Rev., 123, 2758-2770.

Mass, C.F., et. al., 2002: Does IncreasingHorizontal Resolution Produce More SkillfulForecasts?, BAMS, Vol.83, 3, pp.407-430.

Nehrkorn, T., R. Hoffman, C. Grassotti and J.-F.Louis, 2003: Feature calibration and alignment torepresent model forecast errors: Empiricalregularization. QJRMS, 129, pp.195-218.

Reilly, C., P. Price, A. Gelman, and S. Sandgathe,2003: Using Image and Curve Registration forMeasuring the Goodness of Fit of Spatial andTemporal Predictions. (Draft submitted forpublication)

Van Galen, J., 1970: A new method for verifyingdeterministic predictions of meteorological scalarfields. Tellus, XXII, 32-42.