ECMWF Slide 1CAS2K3, Annecy, 7-10 September 2003 Report from ECMWF Walter Zwieflhofer European Centre for Medium-Range Weather Forecasting

ECMWFCAS2K3, Annecy, 7-10 September 2003 Slide 1

Report from ECMWFReport from ECMWF

Walter Zwieflhofer

European Centre for Medium-Range Weather Forecasting


Topics covered

Operational forecasting system

Planned improvements

High-performance computing system

Data handling system

Reanalysis activities


Operational forecasting system

Forecasts of the atmosphere and ocean waves to ten-days ahead

- Twice per day

- T511 (40km) 60-level deterministic model

- T255 (80km) 40-level ensemble prediction system

- All of the above are coupled to an ocean-wave model

Forecasts to one month ahead- Ensembles using a T159 (125km) L40 atmospheric model coupled

to an ocean circulation model (110km)

- Currently run experimentally once per fortnight

Seasonal forecasts- Once per month, using a T95 (210km) L40 atmospheric model and

the same ocean model used for the one-month forecasts


Data assimilation

Four-dimensional variational (4D-Var) analysis with- 12-hour data window

- Inner-loop minimizations at up to T159 (125km) L60 resolution

- Outer-loop resolution of T511 (40km) L60

Short cut-off analyses - 3D-Var with 6-hour data window

- 4 times per day

- Used to supply boundary conditions to several Member States for Limited Area Modelling


N.HEM LAT 20.000 TO 90.000 LON -180.000 TO 180.000

ANOMALY CORRELATION FORECAST

500hPa GEOPOTENTIAL

ECMWF FORECAST VERIFICATION 12UTC

Forecast Day MA = 12 Month Moving Average

19801981198219831984198519861987198819891990 1991199219931994199519961997199819992000200120024.5

5

5.5

6

6.5

7

7.5

8

8.5

9

SCORE REACHES 60.00

SCORE REACHES 60.00 MA

ECMWF forecast verification (12UTC)


Evolution of the number of products disseminated per day

Total is now is 1.3 M per day

Was 300k in 2000


Research directions & operational targets

For 2004:- Assimilation of MSG data and additional ENVISAT and AIRS data

- Assimilation of rainfall measurements

- Increased vertical resolution, particularly in the vicinity of the tropopause

- Preparations for upgrades in horizontal resolution

Target resolutions for 2005:- Deterministic forecast: T799 (25km) L91

- EPS: T399 (50km) L65

- 4D-Var (inner/outer): T255/T799 L91

Compute resources required for the above will be delivered by the IBM Phase 3 system


Slide from CAS-2001: Planned changes

New DHS

NewServers

New HPC systemNew

Network


Computer configuration (September 2003)


Current HPC configuration

Two IBM cluster 1600s with 30 p690 servers eachEach p690 is partitioned into four 8-way nodesEach of the clusters has 120 nodes

- 12 of the nodes in each cluster have 32 GB memory

- All other nodes have 8 GB memory

- Processors run at 1.3 GHz

- 960 processors per cluster for user work

- Dual-plane colony switch with PCI adapters

- 4.2 terabytes of disk space per cluster

Both clusters are configured identically


HPC Migration: project milestones

June 02 Delivery of cluster-A of Phase 1

August 02 Delivery of cluster-B of Phase 1

31 October 02 Passed the Functional Test

(including performance

commitments)

08 December 02 Passed the Operational Test

17 January 03 Full service for external users

07 February 03 Passed the Reliability Test

(availability was > 99%)

04 March 03 Switch-over of the operational suite

31 March 03 End of the Fujitsu service


Migration: general experience

The system needed a larger number of nodes than originally envisaged to meet the contracted performance commitments

The in-depth testing performed in autumn 2002 identified a number of problems that had to be addressed before the acceptance process could complete; IBM worked very hard to resolve these in a timely manner

A lot of attention went into system related issues such as minimising boot times

The migration of the operational suite went smoothlyThe system is very stable and the reliability is high


User availability of the HPC systems

98.0

98.2

98.4

98.6

98.8

99.0

99.2

99.4

99.6

99.8

100.0

Q3-2002 Q4-2002 Q1-2003 Q2-2003

VPP5000 VPP700 HPCA HPCB


HPC upgrade planned for 2004

Phase 3 of the contract with IBM will be a replacement of the currently installed hardware (apart from the reuse of disk space)

Based on p690+ servers and the Federation switch Two p690+ servers are already installed at ECMWF

for testing purposesThe test setup will be enhanced over the next two

months to include more nodes and the new switchMain delivery will start in early 2004Phase 3 will deliver around 2 teraflops of sustained

performance on ECMWF’s main codes (around 5x the sustained performance of the final Fujitsu configuration)


Architecture change: Fujitsu VPP to IBM Cluster 1600

Peak performance9.6 Gflops per processor

Fujitsu VPP5000100 vector processors

IBM p6902 x 960 scalar processors

Peak performance5.2 Gflops per processor

8 processors per shared memory node

8 GB memory per node4 GB memory per CPU

1999 2002


0

500

1,000

1,500

2,000

2,500

3,000s

ec

on

ds

IFS TL1023L60 (480 time steps)

VPP5000 36 CPUs 6036 secs

IBM P690 320 CPUs 5753 secs

TL1023 ~ 20 km


TL511L60 forecast model (26R3)

100

150

200

250

300

350

128 256 384 512Processors

Fo

recast

Days/D

ay

1 thread (MPI only) 2 threads 4 threads


Comparison of 8-CPUs v 7-CPUs (TL511)

Speedup in elapsed time using 16 LPARs

tasks CPUs/LPAR

128 x 1 thread (8 CPUs) = 1.0

112 x 1 thread (7 CPUs) = 1.06

32 x 4 threads (8 CPUs) = 1.20

Speedup in elapsed time using 32 LPARs

tasks CPUs/LPAR

256 x 1 thread (8 CPUs) = 1.0

224 x 1 thread (7 CPUs) = 1.24

64 x 4 threads (8 CPUs) = 1.42


Data handling system

ECMWF has used Tivoli Storage Manager (TSM) for several years but encountered scalability issues

In 2001, HPSS was chosen as a replacementProjected data storage volumes (including backups)

- mid-2003: 1.4 petabytes



Experience with HPSS is very positive:- Complex system but very scalable

- Excellent support when problems are encountered

Timely migration of the data was a major concern at the project planning stage


Migration of data from TSM to HPSS

• Planned in 2002:

• Current Plan:

Jan2002

Jan2003

Jan2004

Jan2005

MARS RDMARS OD & ERA

ECFS

Jan2002

Jan2003

Jan2004

Jan2005

RDOD & ERA

ECFS

Ended 24 Jan (3 months)Started 21 Oct

Planned: around 12 months.

The exact start-date depends on the availability of high-capacity drivesand HPSS 5.1


ERA- 40 project

A global re-analysis of observations covering the period September 1957 to August 2002

Based on cycle 23r4 of ECMWF forecasting system - operational from June 2001 to January 2002

Six-hourly 3D-Var analysis- operations uses 12-hourly 4D-Var

T159 horizontal resolution (~125km grid) - operations uses T511 (~39km grid)


ERA- 40 status

Produced with considerable external support:- Most of the older observations were supplied by NCAR via NCEP

- SST and sea-ice analyses were produced by the Met Office and NCEP

- EUMETSAT supplied reprocessed satellite winds

- Validation partners provided valuable feedback

- Practical support from EU, Fujitsu, IAP, JMA, PCMDI, WCRP, GCOS …

Production was completed in April 2003Full set of products is available from ECMWF MARS2.5o products are available on a public data server

Data is being supplied to NCAR for UCAR use and

general use in USA for research and education

Counts of observations accepted by ERA-40 system

AIRCRAFT

SATELLITE WINDS

SATELLITE RADIANCES

1960 1970 1980 1990 2000

BUOYS

1973 1979


Anomaly correlations of 500hPa height forecasts

Ops 1980

Ops 2002/3

Ops 2001ERA 2001

Australia/New Zealand

%

Documents

ECMWF Slide 1CAS2K3, Annecy, 7-10 September 2003 Report from ECMWF Walter Zwieflhofer European Centre for Medium-Range Weather Forecasting