Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
Technical aspects of IPSL Earth System model
Arnaud Caubel (IPSL, France)
April 20th 2015, Workshop on Coupling Technologies for Earth System Models
Context : IPSL Earth System model
LMDZ
(atmosphere)
OASIS3-MCT
(coupler)
ORCHIDEE
(land surfaces,
vegetation)
INCA/REPROBUS
(atmospheric
chemistry, aerosols)
OPA
(ocean)
LIM
(sea ice)
PISCES
(marine biogeochemistry)
NEMO
• IPSL Earth System model
– History of IPSL-CM : use of previous versions of OASIS as coupler.
– Different configurations at different resolutions are set up and maintained around the physical atmosphere-land-ocean-sea ice model : carbon cycle, stratospheric chemistry, tropospheric chemistry, aerosols,…
– From paleoclimate to decadal studies
– Web page : http://icmc.ipsl.fr/
Context : IPSL Earth System model
Atmospheric LMDZ
lon-lat grid Oceanic NEMO
curvilinear grid OASIS
• IPSLCM5 version (CMIP5 simulations), IPSLCM6 is the current version.
New developments in IPSLCM6 are :
– Update of components versions
• NEMO_v3.6 (ocean), LIM3 (sea-ice)
• new physics for LMDZ atmospheric model
• …
– OASIS3-MCT parallel coupler
– MPI-OpenMP hybrid parallelization for atmospheric /land /atmospheric chemistry components
– XIOS (XML-IO-Server) as output library for all IPSL components
• library dedicated to IO management of climate code, developped at IPSL by Y. Meurdesoif
• XML configuration file
• attached mode (library) or server mode (asynchroneous transfer), multiple (sequential writing) or single (parallel writing) output file
• NetCDF format (GRIB2 in progress, ICHEC collaboration)
IPSL Earth System model : technical developments
Atmosphere LMDZ
Land ORCHIDEE
Chem/Aerosols INCA
Ocean
Sea-Ice
Biogeochemistry
NEMO
Oasis
MCT
Oasis
MCT
MPI process MPI process
OpenMP thread
OpenMP Master thread XIOS
client XIOS
client
XIOS
server XIOS
server
Asynchroneous mode Asynchroneous mode
one output NEMO ocean file
one output NEMO sea-ice file
one output NEMO bio file
one output atmospheric file
one output land file
…
…
Temporal Series TS files
Two computing centers : Curie TGCC, Ada IDRIS
Atmosphere LMDZ
Land ORCHIDEE
Chem/Aerosols INCA
Ocean
Sea-Ice
Biogeochemistry
NEMO
Oasis
MCT
Oasis
MCT
MPI process MPI process
OpenMP thread
OpenMP Master thread XIOS
client XIOS
client
XIOS
server XIOS
server
Asynchroneous mode Asynchroneous mode
one output NEMO ocean file
one output NEMO sea-ice file
one output NEMO bio file
one output atmospheric file
one output land file
…
…
Temporal Series TS files
Two computing centers : Curie TGCC, Ada IDRIS
Atmosphere LMDZ
Land ORCHIDEE
Chem/Aerosols INCA
Ocean
Sea-Ice
Biogeochemistry
NEMO
Oasis
MCT
Oasis
MCT
MPI process MPI process
OpenMP thread
OpenMP Master thread XIOS
client XIOS
client
XIOS
server XIOS
server
Asynchroneous mode Asynchroneous mode
one output NEMO ocean file
one output NEMO sea-ice file
one output NEMO bio file
one output atmospheric file
one output land file
…
…
Temporal Series TS files
Two computing centers : Curie TGCC, Ada IDRIS
Atmosphere LMDZ
Land ORCHIDEE
Chem/Aerosols INCA
Ocean
Sea-Ice
Biogeochemistry
NEMO
Oasis
MCT
Oasis
MCT
MPI process MPI process
OpenMP thread
OpenMP Master thread XIOS
client XIOS
client
XIOS
server XIOS
server
Asynchroneous mode Asynchroneous mode
one output NEMO ocean file
one output NEMO sea-ice file
one output NEMO bio file
one output atmospheric file
one output land file
…
…
Temporal Series TS files
Two computing centers : Curie TGCC, Ada IDRIS
Atmosphere LMDZ
Land ORCHIDEE
Chem/Aerosols INCA
Ocean
Sea-Ice
Biogeochemistry
NEMO
Oasis
MCT
Oasis
MCT
MPI process MPI process
OpenMP thread
OpenMP Master thread XIOS
client XIOS
client
XIOS
server XIOS
server
Asynchroneous mode Asynchroneous mode
one output NEMO ocean file
one output NEMO sea-ice file
one output NEMO bio file
one output atmospheric file
one output land file
…
…
Temporal Series TS files
Two computing centers : Curie TGCC, Ada IDRIS
IPSLCM6 : recent technical developments : what is the gain ?
Components
– MPI-OpenMP hybrid parallelization for atmospheric /land /atm chem components
• more computing cores used (distributed memory for horizontal decomposition, shared memory for vertical decomposition)
• reduces elapsed time of a simulation
• target efficiently SMP architectures (shared memory intra-node, distributed memory inter-node)
– XIOS (XML-IO-Server) as output library
• server mode (asynchroneous mode for writing) reduces elapsed time of the simulation
• post-processing steps (i.e output files rebuild step, times series) not needed anymore= running environment more reliable.
• XML configuration file : flexibility and easy handling
Whole coupled model
– OASIS3-MCT parallel coupler
• parallel use of interpolation weights
• no significative impact when increasing resolution, coupling frequency,…
– Complex application
• MPMD model, diffferent levels of parallelization (MPI for ocean, MPI+OpenMP for atmosphere, IO servers distributed on each computing node,… )
• Not a standard application for computing centres (process assignment, tools,…)
• IPSL context : several computing centres
IPSLCM6 : performances
• Load balancing (difference between computing time of components) ?
– elapsed time of coupled model = elapsed time of slowest component (concurrent mode)
– how to measure load balancing ?
• Not easy to use standard tools to analyse : « vampir », « paraver »,…
• LUCIA : tool to measure performances within an OASIS3-MCT-based coupled system.
• Developped by E. Maisonnave (CERFACS)
• Easy to use : activation via OASIS namcouple, succint metrics (computing time and waiting time are integrated over all model processes and over all the simulation).
• http://www.cerfacs.fr/globc/publication/technicalreport/2014/lucia_documentation.pdf
Waiting time model 2
model 1
lmdz
model 2
nemo
Waiting time
model 2
Vampir trace LUCIA
Waiting time model 2
• IPSLCM5 version used for CMIP5 simulations in 2010-2011
• IPSLCM6 used for CMIP6 simulations in 2016-2017, different resolutions :
– Low Resolution : Atm: 2.5⁰ x 1.5⁰ L79 Oce: 1⁰ L75
– High Resolution : Atm: 0.9⁰ x 0.4⁰ L79 Oce: 0.25⁰ L75
IPSLCM6 : performances and plans
0
2
4
6
8
10
12
0 100 200 300 400 500 600 700
NB
of
sim
ula
ted
yea
rs p
er d
ay
NB cores
ORCA1_LIM3_PISCES
LMDZ 144x143x79 (NP)
LR Ocean component NEMO 1° L75
LR Atmospheric component LMDZ 2° L79
Curie SMP machine
Intel Sandybridge
• IPSLCM5 version used for CMIP5 simulations in 2010-2011
• IPSLCM6 used for CMIP6 simulations in 2016-2017, different resolutions :
– Low Resolution : Atm: 2.5⁰ x 1.5⁰ L79 Oce: 1⁰ L75
– High Resolution : Atm: 0.9⁰ x 0.4⁰ L79 Oce: 0.25⁰ L75
IPSLCM6 : performances and plans
0
2
4
6
8
10
12
0 100 200 300 400 500 600 700
NB
of
sim
ula
ted
yea
rs p
er d
ay
NB cores
ORCA1_LIM3_PISCES
LMDZ 144x143x79 (NP)
LR Ocean component NEMO 1° L75
LR Atmospheric component LMDZ 2° L79
4 OMP thread
Speedup 3.75
8 OMP thread
Speedup 5.6
Curie SMP machine
Intel Sandybridge
• IPSLCM5 version used for CMIP5 simulations in 2010-2011
• IPSLCM6 used for CMIP6 simulations in 2016-2017, different resolutions :
– Low Resolution : Atm: 2.5⁰ x 1.5⁰ L79 Oce: 1⁰ L75
– High Resolution : Atm: 0.9⁰ x 0.4⁰ L79 Oce: 0.25⁰ L75
IPSLCM6 : performances and plans
0
2
4
6
8
10
12
0 100 200 300 400 500 600 700
NB
of
sim
ula
ted
yea
rs p
er d
ay
NB cores
ORCA1_LIM3_PISCES
LMDZ 144x143x79 (NP)
LR Ocean component NEMO 1° L75
LR Atmospheric component LMDZ 2° L79
IPSLCM6-LR Atmospheric model LMDZ 2°L79
Ocean model NEMO 1°L75
480 CPUs
6 years per day
Curie SMP machine
Intel Sandybridge
• Recent technical developments of IPSLCM
– MPI-OpenMP, OASIS3-MCT, XIOS : faster than previous IPSLCM5 version
– XIOS : flexibility and running environment more reliable (good point for CMIP6 output requirements)
– more complex application : not standard use on architecture (process assignment, tools,…), not easy on IPSL context with many computing centers.
• CMIP6 simulations : intensive production mode
• Next developments
– LIM3 as separated executable
– XIOS with more functionalities
– parallel reading (forcing files,…)
– on line post-treatment : grid remapping, interpolations, zonal means , grid coarsening, downscaling, global/partial reduction
– Atmospheric component : new dynamical core DYNAMICO (2017 for HighRes MIP ?)
– better performances/scalability
– MPPs, MICs, GPUs architectures.
IPSLCM conclusions and perspectives
Lon-lat grid Icosahedral grid