Ed Tollerud In collaboration with ESRL, RFC’s, WFO’s, and CDWR HMT-DTC Task

Ed Tollerud

In collaboration with ESRL, RFC’s, WFO’s, and CDWR

HMT-DTC Task

DTC Goals and the HMTProvide early step up for front-line

forecasters vis-à-vis ensemble usage

Demonstrate usefulness of prototype ensemble forecast systems; thereby

provide long-range guidance for ensemble forecast systems

Enhance and motivate MET development

HMT-West ensemble QPF evaluation is current focus to attain these goals

Major accomplishments - past 6 months

Online WRF ensemble verification in near-real-time:Regionalization of HMT domain30-day summariesObservational data options

Object-based (MODE/MET) verification of historical atmospheric rivers

Direct comparison between GFS and WRF ensemble mean

Online QPF verification for HMT-West

Equitable Threat Scores (ETS) for ensemble members

Forecasts initiated at 1200 UTC 1/17 2010

Ensemble mean (black) GFS (green)

30-day summary boxplots, ETS

January 2010 Large HMT domain Intensity

thresholds: .01,.1,.5,1,3 in

Planned activities - next 6 months

Revise and expand winter exercise verification demonstration website

Prepare and present preliminary results from 2009-2010 winter exercise verification

Develop probabilistic capabilities for QPF in the HMT

Continue retrospective study of MET/MODE applications to atmospheric river forecasts

Anticipated major accomplishments - next 6 months

Following features added to HMT-West winter verification exercise: 1-2 new additional baseline models (SREF,

NAM,…)Time series display of MODE attributes Displays of reliability and ROC diagrams6h gage accumulation options

Plans for AOP 2011Incorporate spatial verification statistics for

ensemblesImplement METviewer online displaysCollaborate with DET to apply MET-based

probabilistic verification techniquesDevelop procedures to assess gridded QPE

products and their impact on verificationExtend verification to additional

meteorological fieldsImplement prototype AR verification utility

Anticipated major accomplishments – AOP 2011Revised and expanded verification options

appropriate for HMT (West and East)Comprehensive suite of probabilistic

verification utilities applied to QPF for HMTObject-based verification techniques

implemented for AR forecastingReports and presentations from HMT-West

verification activities

Resource requirements for AOP 2011Staff Category FTE

Scientists 1.0

Software Engineers 0.5

Students 0.1

Scientists Software Engineers

Ed Tollerud John Halley Gotway

Tara Jensen Paul Oldenburg

CIRES associate

Randy Bullock

RAL statistician Brent Ertz

Tressa Fowler Chris Harrop

Barb Brown

Non-salary Cost

Travel $7 K

Publications $2 K

Challenges (Science and Resources)Establishing synergistic roles of DET and DTC

vis-à-vis ensemble verificationDetermining potential use and advantages for

regional and episodic verificationDeveloping appropriate down-scaling methods for baselining and comparisonImproving forecaster access to results of evaluation (ALPS? AWIPS?)Validating long-range point precipitation forecast for Tillamook for September 2011

Tara Jensen

HWT-DTC Collaboration

Collaborating with:NOAA: SPC, NSSL, HPC, AWC, ESRL, EMCUniversities: OU/CAPSNCAR: MMMUCAR: COMET

HWT-DTC Collaboration Goals• Explore a different R2O Paradigm

• “O” is Forecasters, WFOs, and Prediction Centers• Evaluate convection allowing models on leading edge

of NWP• Gather robust dataset for ensemble and deterministic

studies• Datasets and algorithms leveraged by DET

• Demonstrate utility of MET-based objective evaluation in a forecast environment• Provide near real-time evaluation during Spring Experiment• Application of MET feeds back to MET development

• Facilitate faster R2O through focused evaluation of key topics identified during Spring Experiment• Perform retrospective studies using DTCs METviewer

Major accomplishments - past 6 months

Enormous expansion of near real-time evaluation capability Evaluation of 30 models during Spring Experiment

(CAPS ensemble+3 operational baselines) 10 Deterministic and 4 Ensemble products evaluated

using traditional and spatial verification methods. Three additional research models available for

retrospective studies

DTC staff participation in each week of Spring Experiment 2010

Manuscript accepted by AMS Weather and Forecasting.(Kain et. al, 2010 on Impact of radar assimilation on short-term forecasts)

2010 HWT Spring Experiment – 30 models

CAPS SSEF Ensemble PM Mean

CAPS 1 km Model

CAPS SSEF

ARW-CN (control w/

radar assimilation)

3 km HRRR

12km NAM

CAPS SSEF ARW-C0

(control w/o radar

assimilation)

• Assimilation continues to show greater skill over no-assimilation

• CAPS Ens. Mean generally hashigher skill than 1km deterministic and Operational baselines

Planned activities - next 6 monthsComplete re-run of

Spring Experiment evaluation

Add additional probabilistic measures(Brier Skill Score, ReliabilityDiagrams, and ROC Curves)

Perform retrospective evaluation of all members/products

Probabilistic Verification ofEnsemble Products (for a Single Run)

Brier Score and decomposition

CAPS SimpleCAPS NeighborhoodSREF Simple

Anticipated major accomplishments - next 6 monthsAnalysis for presentations (Severe Local Storms

and AMS)Investigate physics perturbations on Accumulated Precip.

and ReflectivityEvaluate performance of probability products for QPFDetermine meaningful metrics from object-oriented

evaluation of Radar Echo Top

Report on impact of Radar Assimilation on 0-12 hr forecast

Targeted analysis using METviewer to facilitate 2011 Spring Experiment planning

Quality-checked and complete dataset available to collaborators for their own investigation

Plans for AOP 2011Contribute to R2O into SPC, HPC and

AWC? by providing objective evaluation for their Spring Experiment foci:SevereConvective Initiation/LightningQPF/Heavy RainEnsemble Product Evaluation

Perform retrospective studiesProvide feedback to researchers on ensemble

configuration and product generationLeverage DET Verification ModuleHelp develop link between DET and HWT

Anticipated major accomplishments – AOP 2011Near real-time objective evaluation for:

Simulated Satellite Lightning ProductQPF and Probabilistic (PQPF) fieldsReflectivity (if desired)

Evaluation products available within 30 min – 2 hours of valid time (rather than 6-18 hours)

DTC staff member participation during each week of Spring Experiment

Submit Journal article on QPF/PQPF evaluation


Scientists 0.54


Students 0.15


Tara Jensen {System Engineer}

Michelle Harrold

Paul Oldenburg

Jamie Wolff* John Halley Gotway

Isidora Jankov* Brent Ernst

Tressa Fowler*

Lisa Coco

Non-salary Cost

Travel $9 K

Workshops $1 K

Publications $1.5 K

Other (see below)

Compute Resources: HPC facility like NOAA-Jet system or NCAR Bluefire systemGoal is to use DET Verification Module.

* Spring Experiment participant

ChallengesHow to balance expectations with budget

HWT Collaboration started smallSuccess has garnered more requests and expectationBudget has not grown

How to serve the needs of additional testbedsAWC planning to start separate Aviation Weather Testbed

Indicated interest in DTC participation - How do we fund activity?HPC planning to continue using HWT facility

What if their evaluation needs expand?

Defining HWT-DTC Collaboration and DET interactionWhat is appropriate evolution of these complimentary

tasks?

Zoltan Toth

DTC Ensemble Testbed

Collaborators: NOAA/NCEP/EMC

DET website: http://www.dtcenter.org/det

Goals for setting up new DET task

Develop and maintain DET infrastructureSupport its use by community

Establish NCEP operational system as benchmark

Test and evaluate new community methods

Transition successful methods to NCEP and other agencies

Link up with ensemble work in other test-beds

Module 1: Configuration

Module 2: Initial

Perturbations

Module 4: Statistical

Post-Processing

External Input(HMT, HWT, HFIP,

etc)

Module 3: Model

Perturbations

Module 5: Product

Generation

Module 6: Verification

DET M

OD

ULES

Major accomplishments - past 6 monthsInitial plans developed for

Overall architecture of DET infrastructureEach of 6 modulesTest and Evaluation

Initial tests for establishing basic capabilities for 2 modulesConfigurationInitial perturbations

Collaboration with other DTC tasks / projectsHMT – Joint plans for testing DET & HMT ensemble

generationHWT – Joint plans for evaluation of 2011 HWT ensemble

OutreachOrganized DET Workshop and engaged with WRF Ensemble

WGActivities coordinated with NCEP/EMC via regular meetings

DET Workshop

Organized by DTC with input from WRF Ensemble WGAug. 18-19, Boulder

Excellent opportunity for connecting DET with community50 plus participants from community & agencies

Total of 6 Breakout sessions focused onConfigurationPreliminary plans for testing

Extensive set of important recommendationsTo be used for updating DET plans

WRF Ensemble WG meeting, Aug 20Additional recommendations

Planned activities - next 6 months

Complete planning for overall design and 6 modules

Develop and test initial perturbation methodsCollaborate closely with NCEP/EMC for their

next implementationComplete protocol for T&EContinue coordination / collaboration with

HMT & HWTLeverage winter 2010/11 HMT ensemble for

DET testingPrepare for joint evaluation of 2011 HWT

ensemble

Anticipated major accomplishments - next 6 monthsDetailed plan for infrastructure – overall

design plus 6 modulesBasic capability for modules 1-2

Initial tests contributing to next NCEP implementation

In collaboration with NCEPProtocol for T&E & basic capability for

verification module (#6)Joint plans / activities with HMT & HWT

Testing of generation & verification of ensemble (HMT)

Evaluation plan of ensemble (HWT)

Basic capability – Functionality tested / supported

Benchmark – Methods used in operations can be replicated

Plans for AOP 2011Benchmark for module 2

Contribute to next NCEP implementation; Replicate next NCEP configuration

T&E for module 2Use T&E protocol and verification module

Contribute to NCEP implementation for module 2 (FY11-12)Test improved method, contribute with algorithm & software

Basic capability for module 3Initial tests with NEMS compatible and WRF models

Contribute to design and evaluation of HMT & HWT ensemblesCollaborative effort

Anticipated major accomplishments – AOP 2011Benchmark for module 2

Milestone: Build initial perturbation module replicating next NCEP SREF implementation

T&E for module 2Milestone: Complete tests of new initial perturbation configuration

Contribute to NCEP implementation for module 2 (FY11-12)Milestone: Collaborate with NCEP scientists and contribute to

development of new initial perturbation methodBasic capability for module 3

Milestone: Demonstrate basic capabilities for model-related uncertainty (NEMS / WRF)

Contribute to design and evaluation of HMT & HWT ensemblesMilestones:

Test DET benchmark configuration for initial perturbation module in HMT ensemble

Contribution to evaluation of HWT ensemble


Scientists 2.1



Barbara Brown Paula McCaslin

Isidora Jankov Linda Wharton

Tara Jensen Eugene Mirvis

Ed Tollerud Ann Holmes

New Hire (DET Lead)

Non-salary Cost

Travel $8 K

Workshops $15 K

Publications $2 K

ChallengesIdentify CPU resources for computer intensive DET work

Internal GSD & NCAR resources very limitedPrepare & submit proposals for external resources

Agencies and community desire quick setup of & extensive testing by DETMismatch between funding and expectationsHow can DTC build benchmarks faster?How can DTC engage other agencies / programs / projects?

HFIP, NUOPC, NEXTGEN, FAA, AFWA, etc

Contingency plan assuming additional fundingAccelerate integrated development of & community support

for modules 4-6Basic capability for modules 4-5 – AOP11Benchmark for modules 4-6 – AOP12

DET TIMELINE – STARTING MARCH 2011AOP11 deliverables – Basic capabilities, benchmarks

Basic capability for module 3Benchmark + T&E for module 2Contribute to NCEP implementation for modules 2 (FY11-12)Contribute to design and evaluation of HMT & HWT ensembles

AOP12 deliverables – Benchmarks, T&EBenchmark + T&E for module 3Basic capability for modules 4-5T&E of community methods for module 2

AOP13 deliverablesBenchmark for modules 4 & 6Contribute to NCEP implementation for module 4 (FY13-14)T&E of community methods for all modulesUpdate benchmarks

Laurie Carson

DTC NEMS

Collaborators: NOAA/NCEP/EMC, NOAA/ESRL

DTC NEMS Goals

Define community software goals for the NEMS modeling framework and operational model configurations

Gain expertise in the NEMS framework, by participating in and assisting the on-going development, with a community support focus

Develop an O2R and R2O ops-concept that supports the needs of both the research and the operational user

Progress to-date: past 6 monthsExplore approaches and ideas to help define DTC NEMS

community software goals and timeline On-going: internal (DTC) and external (EMC, NCAR, GSD)

discussionsNEMS software status and near-term plans at EMC1-2 year plans including FIM, RapidRefresh, SREFTechnical expert meeting planned to focus on the research and

operations connections between WRF and NEMS

Develop DTC expertise in NEMS softwareOn-going: technical discussions with NEMS developers

Code repositoryTask/Work plan for Software Engineer hire

Software Engineer hire to focus on NEMS development (at EMC)starting Oct 4, 2010

Technical Meeting: October 5, 2010

Planned activities - next 6 monthsDTC Software Engineer

Will reside at EMC to further facilitate the information exchange and DTC contributions

Add in-depth knowledge of NEMS to the DTCContribute to the development of NEMS softwareBegin work on a draft code management plan

Hold a technical information exchange meetingFall 2010: DTC, NOAA-EMC, NOAA-GSD, NCAR-MMMoverlap between WRF and NEMS with respect to

community supportmaintain a connection with the broad research communityuse this as a path-finder to explore R2O and O2R

mechanisms

Anticipated major accomplishments - next 6 monthsRefine work plan for DTC-NEMS software

engineerAdd I/O layer capabilities to NEMS that will benefit

research communitiesComputing Platform Portability (systems,

compilers)Inter-operability with other community softwareUsers Guides, sample datasets, and examples of

useTechnical meeting Oct 5, 2010 in Boulder

Focus on upcoming interactions between WRF and NEMS components

Plans for AOP 2011Strengthen the foundation of DTC

expertise with the NEMS software Continue to contribute to the development

process for NEMS, with a community user focus.

Refine DTC NEMS community software goals and timeline

Develop, with EMC, a NEMS code management plan to support community software and operational system goals

Anticipated major accomplishments – AOP 2011DTC NEMS code management planContinued Technical Interchange MeetingsDTC NEMS community software plan,

including a path from R2O and O2R


Scientists 0.1



Jamie Wolff Eugene Mirvis

Laurie Carson

Non-salary Cost

Travel $12 K

Workshops $300

ChallengesEngagement of the NWP research community

is a critical component to making contributions to the operational NWP capabilities

NEMS is still under active development, so any decision to package a software release to the research community must be done with careful coordination between EMC and DTC

A fully supported community software package requires considerable resources from DTC. Is this the best use of DTC resources?

Documents

Ed Tollerud In collaboration with ESRL, RFC’s, WFO’s, and CDWR HMT-DTC Task