Changes at ECMWF

ECMWFCAS 2001, Annecy Slide 1

Changes at ECMWF

Walter Zwieflhofer

Head of Computer Division


Operational system (October 2001)Data assimilation

4D-Var with a 12-hour data window Horizontal resolution: 120 km for inner and 40 km for outer loops

Global atmospheric forecasts Forecasts to 10 days from 12 UTC at 40 km resolution Ensemble forecasts to 10 days with 51 members at 80 km resolution Four forecasts per day to 3 days to provide boundary conditions for

some of the ECMWF Member States (40 km, 3D-Var)

Ocean wave forecasts Global forecasts to 10 days at 40 km resolution European waters forecasts to 5 days at 15 km resolution

Seasonal forecasts Global forecasts to 6 months (ensemble with 30 members)


Forecasting-system changes since 1999

N Hem Mean reduction in day-one error measured in pre-operational trials S Hem

19r2 Increased resolution in stratosphere21r1 Assimilation of raw TOVS/ATOVS radiances21r4 New error statistics for 4D-Var Jb, SSMI wind data, corrected use of humidity observations, increased PBL resolution, improved cloud, convection and orography.22r1 Improved SSM/I QC and second SSM/I satellite

22r3 New error variances (for Jb and Jo) additional ATOVS data, improved land-surface, sea-ice and LW radiation23r1 12-hourly 4D-Var23r3 T511/159 resolution, second ATOVS satellite, increased angular resolution in wave model


Evolution of forecast skill for northern and southern hemispheres


Error distribution for Day 5 (Europe)


Annual-mean RMS errors against analyses from WMO scores


Planned improvements2002 Increase of the inner loop resolution of

4D-Var from 120 km to 80 km2003-2004 Assimilation of cloud and rain in 4D-Var

Higher vertical resolution in free troposphere and lower stratosphere

Assimilation of AIRS data

Operational running of a Severe Weather forecasting system

2005-2006 Assimilation of IASI data


Computer configuration – October 2001


PrepIFSECMWF tool to prepare forecast and data assimilation

experimentsUses the operational configuration of the forecasting

system and allows researcher to introduce modifications to:

input data (including the selection of observations) the model code the resolution to be used and many other parameters

Written Java, with a client part that can be installed on remote machines

Allows registered external users to configure and submit experiments via the Internet


Major changes to thecomputing environmentDesktop systems (procurement completed in 2000)

Decision was taken to replace all proprietary UNIX workstations with PCs running LINUX over a period of three years

Data Handling System – DHS (procurement in progress) A scalable system that will be able to support the expected

growth over 6+ years is required Invitation to Tender for a new system was issued earlier this year

Supercomputer – HPC (procurement in progress) Fujitsu systems are rented and the contract ends on 31 Dec 2002 Invitation to Tender for a replacement to cover the period

2003-2006 was issued in March this year


DHS procurementArchive holds currently some 400 TB of data (primary data)

and has been growing by some 60% per annumTwo services need to be supported at ECMWF:

ECFS, a large-scale UNIX-like filesystem for general use MARS, a customised meteorological archival and retrieval system

Vendors were asked to offer a scalable storage system and were given the option to offer a replacement for the ECFS and/or MARS software layer

DHS and HPC procurements were separated because: Lifetime of the data handling systems tends to be considerably

longer than that of the HPC systems Allows a simpler HPC procurement not affected by the complexities

associated with the replacement of the archive system


HPC procurementECMWF asked for a phased introduction of resources covering

the period 2003-2006, with a parallel run during the second half of 2002

The specifications allowed for some flexibility, enabling vendors to adapt their response to their product development plans. This included the option to propose a 3-month shift of the service period

The funds available were publishedVendors were required to commit to performance increases,

expressed as factors, over the performance of the Fujitsu systems installed at ECMWF

Requirements for disk space, I/O and network bandwidth were expressed as function of these factors


The Benchmark SuiteIFS tests (for the performance commitments)

4D-VAR (T511/255 L60; 2, 4 or 6 copies depending on PHASE) DETERMINISTIC FORECAST (T799 L90; 2 or 4 copies) EPS (T399 L60; 50 or 100 copies)

Burton Smith taxonomy: Size * g >> 2 * O + L Suitable for both T-type and C-type systems

IFS model runs well on all architectures4D-Var is very sensitive to communication bandwidth and

has sections on which scalar machines do wellResolutions representative for the workload planned for

2003-2006 were used


Observations Vendors were allowed to optimize the codesHowever, changes that would affect the readability or

portability were not permittedSome vendors used a hybrid MPI / OpenMP approachSome vendors found that such a hybrid approach did not

improve the results obtained by using MPI only Percentages of peak:

4D-Var scaled not as well the deterministic forecast and the EPS Scalar machines: 3% to 12%, depending on test/product Vector machines: 14% to 38%, depending on test/product

Rule of thumb for ECMWF’s codes was confirmed: ~10% for scalar architectures and ~30% for vector architectures


Status of both procurementsThe Tender Evaluation Boards have completed their work and

have made recommendations to the ECMWF DirectorThe Director has awarded both contracts, subject to approval

by ECMWF CouncilDocuments detailing the respective procurement process and

the proposed solutions have been sent to the technical and financial committees of the ECMWF Council

These committees will meet in November to consider ECMWF’s proposals and to make recommendations to Council

Council will meet in December and (hopefully) authorize the Director to sign the appropriate contracts


Changes over the next 18 months

New DHS

NewServers

Linux PCs

New HPC systemNew

Network