Evaluation of Selected Winter ’04/’05 Performance Results

Seth Linden and Jamie Wolff

NCAR/RAL

Evaluation of Selected Winter ’04/’05 Performance Results

Weather Forecast Verification

• Consensus (RWFS) forecast is compared to individual model components

• Air-temperature, dewpoint, wind-speed and cloud-cover forecasts– 18 UTC runs for the entire

season (1 November 2004 to 15 April 2005)

• Error (RMSE) calculated for:– Colorado Plains: 176 sites– Mountains: 119 sites

Blizzard of March 2003

Air temperature RMSE

Colorado Plains

RWFS

Colorado Mountains

Dewpoint RMSE

Colorado Plains

Forward Error Correction

Colorado Mountains

Due to 3-hour MOS data

Wind Speed RMSE

Colorado Plains

Colorado Mountains

Colorado PlainsCloud Cover

RMSE

Colorado Mountains

•The ensemble approach utilized by the RWFS does improve the predictions on average for all verifiable parameters

•No single model performs better for all parameters

•A blend of weather models will provide better results

Summary/Recommendations

Forecast Model Weights Used by the RWFS

• System automatically weights forecasts based on skill

• Distribution of weight values per lead time for air-temperature, dewpoint, and wind-speed– 18 UTC run on 3 May 2005

• Weights looked at for two sites:– Denver International Airport– I-70 at Genesse

Which models have the most skill?

Air Temperature Model Weights

Denver Int. Airport

ETA

GFSMOSMOSI-70 at Genesee

RUC

Dewpoint Model Weights

Denver Int. Airport

I-70 at Genesee

Denver Int. Airport Wind SpeedModel Weights

MM5I-70 at Genesee

WRF

Insolation Weights

0

100

200

300

400

500

600

700

800

12 0 12 0 12 0 12 0 12 0 12 0 12 0 12 0 12 0 12 0 12 0

Time (UTC)

W/m

2MSH_obEtaGFSMM5WRFeta_12gfs_12mm5_16wrf_16

1/29 1/30 1/31 2/1 2/3 2/4 2/5 2/62/2 2/7 2/8

•No one model consistently outperforms the others•MM5 and WRF forecast hourly instantaneous values, ETA forecasts 3-hour instantaneous values and GFS forecasts 3-hour averages

Clear Conditions

For MDSS static weights were applied:- 50/50 split between MM5 and WRF for the

0-23 hour forecast- All Eta for the 24-48 hour forecast

QPF Weights

Model GFS EtaMM5 2hr

MM5 3hr

MM5 4hr

Total MM5

WRF 2 hr

WRF 3hr

WRF 4hr

Total WRF RUC

MAV- MOS Total %

TOTAL MM5+WRF Contribution

QPF Weights (%) 9 11 15 12 10 37 17 14 12 43 0 0 100.00 80

•Due to a lack of quality precipitation observations static weights were applied•Weights fixed based on expert opinion•MM5 and WRF were given 80% of the total weight

•Weight distribution reflects that the corrected (dynamic MOS) NWS models (ETA, GFS, and RUC) had the most overall skill

•No one model dominates for all parameters•The limitation of the NWS models is their 3-hr temporal resolution

•WRF and MM5 were given the highest static weights for Insolation and QPF


Road Temp Observation Variance

• Tr variance across E-470 corridor– Shading by permanent structures or passing

clouds– Make/model/installation/age of temperature

sensors

E-470 Road/Bridge SitesColorado Blvd Platte Valley

(road and bridge)

6th Ave Pkwy

Plaza ASmokey Hill Rd

(road and bridge)

SCT BKN OVC

LOCAL TIME (19 = noon, 07 = midnight)

27 Nov 2004 28 Nov 2004

OVC CLRBKN SCT


29 Nov 2004 30 Nov 2004


• Large variations in observed road and bridge temperatures– Over relatively small area

(10s of miles)

• Makes prediction and verification of pavement temperatures very challenging– Difficult to establish ground

truth

Road/Bridge Forecast Verification

• Road and bridge temperature forecasts– Using recommended

treatments from MDSS

• Error (MAE) and bias calculated for:– For each lead time (0-48hrs)

18 UTC runs

– E-470: 6 roads/2 bridge (1 Nov 2004 – 15 Apr 2005)

– Mountains: 5 roads (1 Feb 2004 – 15 Apr 2005)

East bound lane of I-70 at the summit of Vail Pass

Consistent low bias

Lead Time (0 = 18 UTC = noon, 18 = 12 UTC = 6am)

Peak insolation

Morning hours

E-470 road sites

Perfect forecast

Lead Time (0 = 18 UTC ~ noon, 18 = 12 UTC ~ 6am)

Shadowing?

evening

morning

E-470 bridge sites

Lead Time (0 = 18 UTC = noon, 18 = 12 UTC = 6am)

evening

morning

CDOT mountain road sites


• Larger Tr differences during times of high solar insolation likely due to several factors:– Errors in measuring pavement skin temp– Mountain shading during low sun angle– Limitations in insolation prediction in models– Limitations in pavement heat balance model

• Simplified assumptions about pavement characteristics

• Tb analysis compromised by:– Sensors shadowed by bridge rail– Bias results suggest tuning may be beneficial

• Overall Issue:– Actual/Recommended treatments not the same

Case Study Analysis• 183 day demonstration

– 16 winter weather days• 10 light snow• 5 moderate snow• 1 heavy snow

November 27-29, 2004

• First significant snow storm of the season– 5-8” in the Denver area

• Large variations in parameter predictions– Forecast vs. observations

• Denver International Airport• Ta, Td, Wspd, Cloud Cover and Precipitation

• 12 UTC 28th examined – Captured the start time of event

-12

-10

-8

-6

-4

-2

0

2

4

6

12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11

Time (UTC)

Air

T (

C)

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

SN


28 Nov 2005

8C/14F diff2C/4F diff

Air Temperature

Snow

-12

-10

-8

-6

-4

-2

0

12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11

Time (UTC)

Td

(C

)

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

SN


28 Nov 2005

6C/11F diff

Dewpoint Temperature

Snow

0

1

2

3

4

5

6

7

8

9

10

12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11

Time (UTC)

WS

d (

m/s

)

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

SN


28 Nov 2005

Snow

Wind Speed

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11

Time (UTC)

CC

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

OVC

BKN

SCT

FEW

CLR

SN

FEC


28 Nov 2005

Cloud Cover

Snow

0

0.05

0.1

0.15

0.2

0.25

0.3

12 13 14 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11

Time (UTC)

QP

E (

in)

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

SN


28 Nov 2005

Quantitative Precipitation Forecast

Snow

March 13, 2005

• Moderate Snow Event – 4-6” along the E-470 corridor

• Warm air temps before start of snow– Dropped from 11C (52F) to -2C (29F) in 5 hours

• Large variations in parameter predictions– Forecast vs. observations

• Denver International Airport• Ta, Wspd, Cloud Cover and Precipitation

• 00 UTC 13 March 2005 run examined– Captured both start and end times

-8

-6

-4

-2

0

2

4

6

8

10

12

14

16

18

20

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Time (UTC)

Air

T (

C)

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

SN


13 March 2005

Air Temperature

Snow

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Time (UTC)

WS

d (

m/s

)

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

SN


13 March 2005

Wind Speed

Snow

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Time (UTC)

CC

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

OVC

BKN

SCT

FEW

CLR

SN


13 March 2005

SCT - OVC

Cloud Cover

Snow

actual forecast

Start timeactual forecast

End timeLOCAL TIME (18 = noon, 07 = midnight)

13 March 2005

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Time (UTC)

QP

F (

in)

OB

RWFS

Eta

GFS

MM5_2

MM5_3

MM5_4

RUC

WRF_2

WRF_3

WRF_4

Quantitative Precipitation Forecast


• Large discrepancies between weather models in predicting state weather parameters– All too dry for Td and cloud cover– Low wind speed bias during windy conditions– Overall, no ONE model outperforms => Ensemble

approach key

• Supports probabilistic forecast presentation– Atmosphere is unpredictable– Best approach to present uncertainty to end users?

Thank You!

Questions?

Documents

Evaluation of Selected Winter ’04/’05 Performance Results