Upload
timothy-davidson
View
18
Download
0
Embed Size (px)
DESCRIPTION
D.Kiktev, E.Astakhova, A.Muravyev, M.Tsyrulnikov. Perfomance of the WWRP project FROST-2014 forecasting systems: Preliminary assessments (FROST = Forecast and Research in the Olympic Sochi Testbed). WWOSC-2014, 16-21 August 2 014. PRESENTED by S. BELAIR (with additions). - PowerPoint PPT Presentation
Citation preview
Perfomance of the WWRP project FROST-2014 forecasting systems: Preliminary assessments
(FROST = Forecast and Research in the Olympic Sochi Testbed)
D.Kiktev, E.Astakhova, A.Muravyev, M.Tsyrulnikov
WWOSC-2014, 16-21 August 2014
PRESENTED by S. BELAIR (with additions)
Items in this presentation:
Brief reminder
Online monitoring, evaluation (deterministic and ensemble)
Interesting cases
Integrated forecasting
What’s next
Goals of WMO WWRP RDP/FDP FROST-2014:
• To improve and exploit:– high-resolution deterministic mesoscale forecasts of meteorological conditions in winter complex terrain environment;
– regional meso-scale ensemble forecast products in winter complex terrain environment;
– nowcast systems of high impact weather phenomena (wind, precipitation type and intensity, visibility, etc.) in complex terrain.
• To improve the understanding of physics of high impact weather phenomena in the region;
• To deliver deterministic and probabilistic forecasts in real time to Olympic weather forecasters and decision makers.
• To assess benefits of forecast improvement (verification and societal impacts)
• To develop a comprehensive information resource of alpine winter weather observations;
3rd meeting of the project participants (10-12 April 2013)
International participants of the FROST-2014 project
• COSMO, • EC, • FMI, • HIRLAM, • KMA, • NOAA, • ZAMG
under supervision of the WWRP WGs on Nowcasting, Mesoscale Forecasting, Verification Research
Observational network in the region of Sochi
- About 50 AMS; - C-band Doppler Radar WRM200; - Temperature/Humidity profiler – HATPRO; - Wind – Scintec-3000 Radar Wind Profiler;- Two Micro Rain vertically pointing Radars (MRR-2);- 4 times/day upper air sounding in Sochi
Forecasting systems participating in RDP/FDP FROST-2014
Nowcasting: ABOM, CARDS, INCA, INTW, MeteoExpert, Joint (Multi-system forecast integration)
Deterministic NWP: COSMO-RU with grid spacing 1km, 2.2 km, 7km; GEM with grid spacing 2.5 km, 1 km, 0.25 km; NMMB – 1 km; HARMONIE – 1 km; INCA – 1 km
Ensemble NWP: COSMO-S14-EPS (7km), Aladin LAEF (11km), GLAMEPS (11km), NNMB-EPS (7km ), COSMO-RU2-EPS (2.2km), HARMON-EPS (2.5km)
FROST-2014 Online Monitoring of Forecast Quality: Role of resolution - GEM-2.5 km vs GEM-1 km vs GEM-250 mForecast mean absolute errors as a function of forecast lead time.
Location: Mountain skiing finish (Roza-Khutor-7 station).
Period: 15 Jan – 15 March
Effect is not straightforward. It depends on meteorological variable, location, lead time etc.
Some examples of diagnostic verification: Role of spatial resolution. COSMO-S14-EPS (7km grid spacing) vs COSMO-RU2-EPS (2km grid spacing)
Parameter: T2m, Location: Biathlon Stadium (1075m), Verification Period: 15.1.2014-15.3.2014, Verification approach: Nearest point
COSMO-S14-EPS (7km grid spacing)
COSMO-S14-EPS (7km grid spacing)
COSMO-RU2-EPS (2km grid spacing)
COSMO-RU2-EPS (2km grid spacing)
Better shape on Q-Q-plots and higher variability for the downscaled ensemble forecasts.
Role of spatial resolution for ensemble forecasts – continuedCOSMO-S14-EPS (7km grid spacing) vs COSMO-RU2-EPS (2km grid spacing)
Station BIAS (for 6/12/18hr lead time) Mean Absolute Error (for 6/12/18hr lead time)
COSMO-S14-EPS COSMO-RU2-EPS COSMO-S14-EPS COSMO-RU2-EPS
Sledge(~700m)
-1.3 / -2.0/ -1.4 0.2 / -1.9 / -0.1 1.6 / 2.2 / 1.6 1.4 / 3.5 / 1.7
Freestyle(~1000m)
-2.0 / -1.8 / -1.9 0.3 / -0.7 / 0.0 2.1 / 2.0 / 2.1 1.6 / 2.4 / 1.7
Biathlon Stadium(~1500m)
-1.4 / -1.3 / -1.4 0.9 / 0.0 / 0.5 2.0 / 1.8 / 2.1 2.1 / 2.6 / 2.3
Mountain Skiing(start)(~2000m)
1.6 / 2.2 / 1.6 0.6 / 0.2 / 0.1 2.8 / 3.1 / 2.8 2.1 / 2.2 / 2.6
• T2m: Some positive effect of downscaling from 7 to 2 km resolution.
• Wind Speed: No positive effect of dynamical downscaling was found.
Verifications for ensemble meanVerification Period: 15.1.2014-15.3.2014
ROCA
BSS
BS
COSMO-S14-EPS – redCOSMO-RU2-EPS – orangeLAEF-EPS – brownNMMB-EPS – blackHARMON-EPS – blueGLAMEPS – green
Verification approach: 13 stations in the area of Krasnaya Polyana were clustered for matching to forecasts.
Some ensemble verifications:
ROC Area, Brier Skill Score, and Brier Score for Precip > 0.01 mm/3h
COSMO-S14-EPS, NMMB-EPS and COSMO-RU2-EPS look most informative.
Lead time
ROCA, BSS, and BS scores for Precip > 5 mm/3h
For higher Precip threshold (w.r.t the low threshold):= COSMO-S14-EPS, NMMB, and HARMON-EPS become worse. = In contrast, LAEF and GLAMEPS become better.
BSS
ROCA
BS
COSMO-S14-EPS – redCOSMO-RU2-EPS – orangeLAEF-EPS – brownNMMB-EPS – blackHARMON-EPS – blueGLAMEPS – green
17.02.2014 . Camera shots from Gornaya Carousel-1500
FROST-2014 experience demonstrates that direct forecast of visibility is a serious challenge. However, some results were encouraging.
Example: 17 February 2014, 11:00-12:00 UTC (Biathlon venue) – Forecast of time slot for competitions during the 3-days period with low visibility.
Forecast of wind direction and relative humidity (as proxy of visibility)by COSMO-Ru1 (1km grid spacing)
Wind and RH at 850 hPa. Forecast from 12 UTC 16.02.2014
Biathlon Stadium
11:00 UTC 11:30 UTC 12:00 UTC
11:00 UTC 13:00 UTC12:00 UTC
RH at 2m:Forecast and observations
Biathlon Stadium Biathlon Stadium
How was the window of good visibility on 17 February predicted by various systems?
List of other interesting casesCase Meteorological
process/phenomenonModels’ behavior Impact on
competitions07.02.14 Foehn Poor T forecast by most models at
Biathlon Stadium(forecast errors negative: 1.4…3.7°С)
15.02.14 Poor Max Wind Speed (Vmax) forecast by most models at Krasnaya Polyana(forecast errors negative: 3.5…7 m/s)
16.02.14 Low visibility Postponed competitions at Laura and Extreme Park
18.02.14 Cold front Good precipitation forecast by most models
22.02.14 Foehn Poor T forecast by most models(negative forecast errors: -2.4...-4.4°С,
most markedly at height 1500 m)
11.03.14 Cold front. Low visibility Tmax not good by most models (maximum T forecast at noon, whereas
in reality maximum T occurred in the morning)
Postponed skiing competitions at Roza
Khutor
13.03.14 “Weak process”. Precip. Poor precipitation forecast by most models at heights above 1500 m
17.03.14 Cold front Poor Vmax forecast (underestimation) by most models at heights above 1500 m
N
iiii tbtfttOttF ))()(()())(1()()(
It was not simple for forecasters to deal with such an amount of information under the operational time
constraints => Integrated Forecast
F(t) – integrated forecast (t – forecast time);O – last available observation; fi(t) – forecast of i-th participating forecasting system;α(t), βi(t) - weights;bi(t) - bias for i-th forecasting system
• F. Woodcock and C. Engel: Operational Consensus Forecasts, Weather and Forecasting, 2005;
• L.X. Huang and G.A. Isaac: Integrating NWP Forecasts and Observation Data to Improve Nowcasting Accuracy, Weather and Forecasting, 2012
FROST-2014 weather data feed for the Olympic information system
Integrated objective multi-model forecasts served as a first guess for preparation of the “official forecasts” for the Olympic information system.
Web-editor was developed for forecasters for correction of objective forecasts.
ATOS Requrements: - 1-hour update frequency;- Temporal resolution:
for a current day – 1 hour;for subsequent days – 3 hours;
- Forecast outlooks for a current day and next 5 days;- Alert Warnings
Forecasters’ subjective evaluationModel
Grid mesh size
Overall usefulne
ss
Forecast accuracy: Visualization (appearance)
Timeliness and
reliability
Comments
T Precip
Wind
Gusts
Vis
COSMO-Ru77 km
2.4 1.9 2.3 2.3 2.1 2.9 2.9 The basic model for the forecasters. Reasonable precip fact. Overestimated precip intensity. Tmin, Tmax poor. Wind poor. dT/dt OK.
COSMO-Ru2
2.2 km
2.7 2.3 2.3 2.3 2.1 2.9 2.7 The basic model for the forecasters. In general better than Cosmo-Ru7.
COSMO-Ru1
1.1 km
2.3 1.5 2.0 2.0 2.3 2.8 2.4 Comments are contradictory. The majority of forecasters considered COSMO-Ru2 to be more useful than COSMO-Ru1. Some forecasters preferred Cosmo-Ru1 (helpful wind, humidity). Overestimates precip intensity.
COSMO-S14-EPS
7 km
2.1 2.0 2.0 2.0 2.0 2.7 2.7 Precip reasonable. Good tendencies. Wind poor. Was available well before the Olympics that was helpful to get used to this information.
NMMB1 km
2.0 2.0 2.0 1.3 1.3 2.0 2.3 2.3 Good in T and Precip. Informative visibility
NMMB-EPS 7 km
2.1 2.0 2.0 1.3 2.0 1.7 2.2 2.7 Nice. Informative visibility. Precip reasonable. Tmin, Tmax poor
GEM-2.52.5 km
2.3 2.0 1.9 1.7 1.6 1.6 2.2 2.4 Good precip, humidity.
GEM-11 km
2.2 2.0 2.0 1.7 1.5 1.5 2.2 2.3 Good precip, humidity.
GEM-250250 m
2.4 2.2 2.2 2.0 2.0 1.8 2.3 2.3 Good precip, humidity. Very detailed maps.
Forecasters’ subjective evaluationModel
Grid mesh size
Overall
usefulness
Forecast accuracy: Visualizati
on (appearanc
e)
Timeliness and reliability
CommentsT Preci
pWin
dGust
sVis
GLAMEPS11 km
1.5 1.8 1.8 1.8 2.0 2.3 2.7 Informative tendencies. Issues with absolute values.
GLAMEPS calibr.,frequen
t update11 km
2.0 2.0 2.0 2.0 2.0 2.2 2.7 Interesting and helpful.
HarmonEPS2.5 km
1.3 1.5 1.3 1.3 1.3 2.2 1.8 In general good in T and Precip, but there were problems with T in anticyclones and Foehn.
Harmonie1 km
2.3 2.3 2.3 2.0 2.0 2.3 2.3 Good T, Precip.
ALADIN LAEF11 km
2.0 1.8 1.8 2.0 2.0 2.5 2.7 Good Wind, including Vmax. Nice plots
WRF600 m
2.1 2.0 2.3 2.1 2.2 2.5 2.1 1.6 Useful but late
COSMO-Ru2-EPS
2.2 km
1.7 1.3 1.7 1.7 2 2.3 2.3 Experimental
Project Social and Economic Impacts
Socially significant project application areas:• Education• Understanding• Transfer of technologies• Practical forecasting – first guess for operational official forecasts.
Integrated project forecasts were used as a first guess for the data feed to the Olympic information system.
Further steps
• Enhanced quality control of the project observations archive. • Additional diagnostic tools and export facilities on the project web-site http://frost2014. • Open access for international research community.
• Validation and intercomparison of the participating forecasting systems, case studies and numerical experiments, assessments of predictability of various weather elements;
• Update of developed technologies and transfer of positive experience into operational practice.
What’s next…
Workshop this Fall in Moscow (difficult to attend for some of the participants)
Interesting case studies for the international community
Joint paper?
Lessons learned (after Vancouver and in preparation for upcoming events)
http://frost2014.meteoinfo.ru
Thank you!
Gratitude to all the participants !
Along with traditional verification measures some new scores were implemented.
EDI - Extremal Dependence Index
NOTES: - Pictures will refer to thresholds 0.01 and 3, and the last threshold at which any of the three EDI curves remains not interrupted in the 0-36h interval- The base rate has the following approximate values: P(0.01mm/3h)=0.3; P(1mm/3h)=0.2; P(2mm/3h)=0.15; P(3mm/3h)=0.1; P(4mm/3h)=0.055;P(5mm/3h)=0.05
EDI = (logF – logH) / (logF+logH)
EDI is especially recommended for low base-rate thresholds, but it will give a good comparative estimate of accuracy for all thresholds (“Suggested methods for the verification of precipitation forecasts against high resolution limited area observations” by the JWGFVR (Laurie Wilson, Beth Ebert et al.)
COSMO-S14-EPSHighest threshold: 8 mm/3h
Lower decision-making level
Blue: EDI for 50% probability thresholdGreen: for 66%Red: for 90%
Conclusions, EDI
• Extremal Dependence Index, EDI, can be used for decision making, especially for rare events when other scores, such as PSS, approach zero. Constructing EDI for different probability decision levels (50, 66, and 90%) showed that the participated EPSs demonstrate skill for all these levels up to the following precipitation thresholds:
COSMO-S14-EPS and NMMB-EPS – informative up to 8mm/3h;
COSMO-RU2-EPS, Harmon-EPS, ALADIN LAEF - informative up to 6mm/3h;
GLAMEPS – informative up to 4mm/3h.
• Sampling effects are evident for all the models, especially for higher thresholds of variables.
• It is not possible to single out “the best ensemble producing system”, but still some conclusions can be drawn.