The dynamical core of the Met office's Unified Model ... · PDF fileThe challenge Computational ... ENDGame grid Scalar variables, w ... (John Thuburn, Colin Cotter) C-grid finite-element

© Crown copyright Met Office

The dynamical core of the

Met office's Unified Model:

challenges of future computer architectures

Nigel Wood, Dynamics Research, UK Met Office

© Crown copyright Met Office © Crown copyright Met Office

Outline

The Unified Model – where we are now

The need for change

The challenge, now and the future

GungHo!

Summary


Current Unified Model “New Dynamics” Davies et al. (2005)

Dynamics:

• Regular lat/lon grid.

• Non-hydrostatic dynamics with a deep atmosphere.

• Semi-implicit time integration with 3D semi-Lagrangian advection.

• Atmospheric tracer advection

Physics:

• Spectral band radiation

• Diagnostic or prognostic cloud

• Mixed-phase ppn

• Mass flux convection

• Boundary layer

• Gravity wave schemes

Coupling possible to non-atmospheric components:

• Land surface model

• Ocean model

• Sea ice model

• Chemistry/aerosol model …


Operational NWP Models

Global 25km 70L

4DVAR – 60km

60h forecast twice/day

144h forecast twice/day

+24member EPS at 60km 2x/day

NAE 12km 70L

4DVAR – 24km

60h forecast

4 times per day

+24member EPS at 18km 2x/day

UK-V (& UK-4) 1.5km 70L

3DVAR (3 hourly)

36h forecast

4 times per day


And the forecast for today...

Relative performance

100

150

200

250

300

350

400

450

2003

01

2004

01

2005

01

2006

01

2007

01

2008

01

2009

01

2010

01

2011

01

Met OfficeECMWFUSAFranceGermanyJapanCanadaAustralia

2003 2011

3 day Northern Hemisphere surface pressure errors © Crown copyright Met Office


Milestone reached October 2012

Forecast length where surface pressure RMSE over N.

Atlantic and Europe matches one day forecast in 1980

Performance metric for CMIP3 climate models

Reichler and Kim, 2008, BAMS

HadGEM1

HadCM3

HadGEM2-AO

Reanalyses

Mean model

Better Worse © Crown copyright Met Office


The need for change…

Weymouth Bay wind & wave


Weymouth Bay models

333m UM weather forecast model

Used for Weymouth Bay wind climatology

Captures local wind/weather detail missed by 1.5km UKV: frost/fog hollows, beach vs town, park vs urban

Could be deployed for UK in 2020, given HPC resources & prioritised requirement

250m SWAN wave forecast model

Resolves headlands missed by 12km operational wave model including blocking of SW swell by Portland

Represents shallow water physics better than Wavewatch III including focussing of swell on Kimmeridge Bay

Aim to add physics to WWIII, couple with NEMO-SHELF ocean model & configure to forecast within 1km of UK shoreline by 2020


X = New Orleans

Simulations initialised 3.5 days prior to

observed landfall

Hurricane Katrina

Stu Webster © Crown copyright Met Office

4 km + 1.5 km make landfall within 6 hours of observed time

Minimum pressure for 4 km + 1.5 km simulations lower than

observed 902 hPa

Hurricane Katrina

4km

1.5km

Obs

24km



A similar story for Super-Typhoon Megi

Obs

1.5km

4km

12km

25km

4km

1.5km Obs

12km

25km

Central Pressure 10m Wind Speed


High resolution climate predictions (60km Atmosphere; 1/40 Ocean)

New model (Operational in seasonal forecasting in 2012):

- Better representation of Gulf Stream

- More realistic blocking

- Better representation of extremes

Observations

Current model

New model

90W 0 90E

Winter blocking frequency 0.30

0.15

0.00


The challenge…

Computational performance critical

Global 25km model (current resolution):

Forecast to: 7 days 3 hours

Timestep: = 10mins 1026 time steps

Resolution 1024 × 768 × 70 = 55M grid points

To run in 60 minute slot, including output


The consequence…

Global 17km model (upgrade next year?):

Timestep = 6 mins

Resolution = 1536 × 1152 × 70 = 124M points

Increase by factor of nearly 4

But time slot unchanged

Algorithmic + code scalability is critical © Crown copyright Met Office

Improved Scalability

dt = 6 mins

ENDGame within the

target for 7 day f/cast

Target

configuration

(17km) Andy Malcolm

Run time (s)

Nodes

(1 node=32 processors) © Crown copyright Met Office

Main relevant changes

Improved (iterative) solution procedure

More implicit, approaching second-order in time

Allows much simpler Helmholtz problem (7 point stencil cf. 45 point)

Much simpler (red/black) preconditioner greatly reduced communications

Improved scalability


Current “New Dynamics” grid

V-at-poles vs. U-at-poles

A “trivial” change to model grid!

Scalar variables, w winds

u winds

v winds

Lon (λ) Lat (Φ)

ENDGame grid

Scalar variables, w winds

u winds

v winds

Lon (λ) Lat (Φ)



Longer term…

Top500 #1 GFLOPS

ECMWF = # 34 & 35

MetO = #43, 51 & 143

Top500 #1 Cores

20 systems have >100,000 cores

1 system has >1,000,000 cores

Top500 projections

100M cores?

Scalability

(17km)

Nodes

(1 node=32 processors)

T24/TN

Perfect scaling

24 nodes



The finger of blame…

At 25km resolution, grid spacing near poles = 75m

At 10km reduces to 12m!


Challenges!

Scalability – remove the poles!


Challenges!

Scalability – remove the poles!

Speed – cannot sacrifice this for low resolution moderate core counts

Accuracy – need to maintain standing of model

Space weather 600km deep model…

Danger: Everything to everyone…or Nothing to anyone?


Globally

Uniform

Next

Generation

Highly

Optimized

GungHo!

“Working together harmoniously”


GungHo Issues

How to maintain accuracy of current model on a GungHo grid?

Principal points about current grid are:

Orthogonality

C-grid

These provide a number of good numerical properties:


From GungHo to not so GungHo

Staniforth & Thuburn 2012 identified ten

“Essential and desirable properties of a dynamical core”:

1. Mass conservation

2. Accurate representation of balanced flow and adjustment

3. Computational modes should be absent or well controlled

4. Geopotential gradient and pressure gradient should produce no unphysical source of vorticity

5. Terms involving the pressure should be energy conserving.

6. Coriolis terms should be energy conserving

7. There should be no spurious fast propagation of Rossby modes; geostrophic balance should not spontaneously break down

8. Axial angular momentum should be conserved



These 5 properties relate to the mimetic properties of the numerics

∇(∇p) = 0

u·∇p+p∇·u = ∇·(up)

u·(Ωu) = 0



9. Accuracy approaching second order

10.Minimal grid imprinting

These are particularly challenging for grids with special points/regions

likely to require higher order schemes…

…whilst maintaining (1)-(8)


Progress

Non-orthogonal finite-difference

TRiSK extension (John Thuburn)

Low-order, primal-dual mixed-finite element method (John Thuburn, Colin Cotter)

C-grid finite-element

Higher-order, mixed finite-element schemes

(Colin Cotter, Andrew Staniforth, Tom Melvin)

Data model changes focus

Grid choice secondary to choice of discretization (David Ham)


Timetable…

Vertical discretization 2013

3D prototype development 2014-2015

Operational…by 2020

Long term step change in scalability


Summary…


Continual improvement…

Forecast length where surface pressure RMSE over N.

Atlantic and Europe matches one day forecast in 1980

?


Globally

Uniform

Next

Generation

Highly

Optimized

GungHo!


Thank you! Questions?

“It would appear that we have reached the limits of what is possible to achieve with computer technology, although one should be careful with such statements, as they tend to sound

pretty silly in five years”

John von Neumann, 1949

And finally…

(with thanks to Mike Ashworth)

Documents

The dynamical core of the Met office's Unified Model ... · PDF fileThe challenge Computational ... ENDGame grid Scalar variables, w ... (John Thuburn, Colin Cotter) C-grid finite-element