View
3
Download
0
Category
Preview:
Citation preview
The Modular Earth Submodel System (MESSy):Atmospheric Chemistry in Earth System Models
http://www.messy-interface.org
P. Jöckel Max Planck Institute for Chemistry, Mainz, Germany
ScicomP13, July 16-20, 2007, Supercomputing Center Garching
The Modular Earth Submodel System
MESSy communityMax Planck Institute for Chemistry,
Air Chemistry Department, Mainz
DLR Institute for Physics of the Atmosphere,Oberpfaffenhofen
Institute for Meteorology and Climate Research,Karlsruhe Research Centre
MPG Supercomputing Center Garching (RZG), Garching
Institute for Meteorology, Free University Berlin
Max Planck Institute for Meteorology,Hamburg (ECHAM5)
Aristotle University of Thessaloniki
The Modular Earth Submodel System
MESSy hall of fame
Baumgärtner, Andreas; Brinkop, Sabine; Brühl, Christoph; Buchholz, Joachim;Burrows, Susannah; Butler, Tim; Dameris, Martin; Erhardt, Gabriele;
Ganzeveld, Laurens; Giner, Bernhard; Giorgetta, Marco; Grewe, Volker;Gromov, Sergey; Jöckel, Patrick; Kerkweg, Astrid; Ketelsen, Klaus ;Kirner, Ole;
Kubin, Anne; Kunze, Markus; Kurz, Christian; Lawrence, Mark; Langematz, Ulrike; Lauer, Axel; Lelieveld, Jos; Metzger, Swen; Nissen, Katrin;
Ponater, Michael; Pozzer, Andrea; Rhodin, Andreas; Riede, Hella; Ruhnke, Roland; Sander, Rolf; Sausen, Robert; Schumann, Ulrich;Steil, Benedikt; Steinkamp, Jörg; Stier, Philip; Tanarhte, Meryem;
Taraborrelli, Domenico; Tost, Holger; Tourpali, Kleareti; Traub, Michael; van Aalst, Maarten; van Aardenne, John; ...
Developers, contributors, users, promoters:
... and much more data users ...
The Modular Earth Submodel System
I. MotivationII. Overview of the concept of MESSyIII. MESSy for atmospheric chemistryIV. Evaluation of the AC-GCM
ECHAM5/MESSy1 (E5/M1)V. Some notes on the implementation
and performance
Outline
The Modular Earth Submodel System
I. Motivation
The Modular Earth Submodel System
Atmospheric Chemistry:● major player in the Earth Climate system● provides link between different domains and processes:
● radiative budget (CO2, Ozone, clouds)
● interaction with ocean and biosphere (C-,N-,S-,...-cycles)● human impact (pollution) and impact on humans (air quality)
● self-cleansing capacity of the atmosphere prevents runaway
Atmospheric Chemistry in Earth System Models (ESMs):● a large number of processes involved ...● ... which are “chemically” coupled● highly non-linear processes:
>>> avoid recompilation wherever possible● wide range of time-scales (10-2 s ... 106 years)● large number of species (all relevant ?)
Motivation
The Modular Earth Submodel System
Emission
Transformation
Deposition
Transportnatural
emissions
anthropogenicemissions
gas phase clouds
aerosol
advection convection diffusion
wetdeposition
sedimentation
drydeposition
scavenging
OCEAN AND BIOSPHERE
Motivation
The Modular Earth Submodel System
MotivationDevelopment aspects:● always state-of-the-art● prepared for the future
Scientific aspects:● increasing complexity requires increasingly transparent
control● variable complexity depending on the scientific
questions/demands -> high flexibility● minimized effort for extension (plug&play, exchangeable code)● efficient future developments (standardised interfaces)● feedback mechanisms between processes (quantification)● process oriented● high degree of internal consistency● reproducibility● ...
continuous further development
The Modular Earth Submodel System
... consistency
many projects / ONE code
com
ple
xity
simulation time
GCM
BOX
MESSy
climatestudies
processstudies
Motivation
The Modular Earth Submodel System
II. Overview of the conceptof MESSy
The Modular Earth Submodel System
process oriented approach of MESSy
MESSy is (a project providing) ...● an interface with infrastructure to couple 'processes' (=submodels) to a GCM (= base model)● a set of processes coded as switchable submodels● a (simple) coding standard
What is MESSy ?Jöckel et al., ACP, 2005
advection
radiation
chemistry
microphysics
convection...
clock/runcontrol(base model)
standard interface
soilvegetation
land use
...
dynamics
biologychemistry
... ... AtmosphereOcean
Land surface
...
MESSy The Systemis a coupled
set ofcommunicating
processes.
The Modular Earth Submodel System
The key: Standardisation
Fortran90/95 ISO/IEC 1539:1991languagelevel
user interface:namelists
subroutines as 'operators'in smallest meaningful entity
(box, column, column-vector, ...)
INTENT(IN) INTENT(OUT)process
level
interfacelevel
SWITCH /CALL
infra-structure
level
MEMORY /OUTPUT
TRACER RE-GRIDDING
TOOLS ...
Standardisation on the lowest possible level ...
process specific interface
“sciencecode”
The Modular Earth Submodel System
How does MESSy work ?
Base Model Interface Layer: multiple socket outlet
Submodel InterfaceLayer: connector
Submodel Core Layer:the machinery ...
Base Model Layer:power supply
The Modular Earth Submodel System
INITIALISATIONPHASE
INTEGRATIONPHASE
FINALISINGPHASE
user friendly flexibility: 'model setup'
PERFORMANCE !!!time loop, decomposition loop, ...
INPUT of boundary conditions
OUTPUT of results
Basic flow chart
The Modular Earth Submodel System
III. MESSy for atmosphericchemistry
The Modular Earth Submodel System
Atmospheric chemistry
Emission Transformation
Deposition Transport
ConVective TRANSport
LightningNOx
ONLineEMissions
OFFLineEMissions
TNUDGE(pseudo emissions/depositions)
CONVECTion
GCM
advection
radi ationRAD4ALL
DRYDEPosition
AIRSEAexchange
MECCA
incloud chem.,SCAVenging &wet deposition
CLOUD
JVAL/PHOTO
Aerosol:MECCA_AERO,M7, SEDI, PSC,HETCHEM, EQSAM
coupling viaTRACER
The Modular Earth Submodel System
TRACER
META-DATA
MEMORY
_FAMILY
_PDEF
● transport families● tagging (linearisation)
● keep positive● track corrections● global mass diagnostics
SET 1
META-DATA
MEMORY
SET 2
META-DATA
MEMORY
SET n
● “generic” submodel (i.e., infrastructure coded as submodel)● provides “chemical coupling” between different processes
(e.g., emission -> chemical reaction -> advection -> deposition)
TRACER
The Modular Earth Submodel System
TRACERMETA-DATA: characteristics of tracers (= chemical species)
-> concatenated list of nested Fortran90-structures
TYPE t_trinfo TYPE(t_ident) :: ident ! IDENTIFICATION TYPE(t_proc) :: proc ! PROCESS FLAGS TYPE(t_infra) :: infra ! INFRASTRUCTURE TYPE(t_typ_single) :: typ_single ! TYPE: SINGLE TYPE(t_typ_family) :: typ_family ! TYPE: FAMILY TYPE(t_typ_isotope) :: typ_isotope ! TYPE: ISOTOPE TYPE(t_med_aerosol) :: med_aerosol ! MEDIUM: AEROSOLEND TYPE t_trinfo
TYPE t_trinfo_list TYPE(t_trinfo) :: info TYPE(t_trinfo_list), POINTER :: nextEND TYPE t_trinfo_list
TYPE t_typ_single REAL(DP) :: molarmass = 0.0_DP ! REAL(DP) :: henry = 0.0_DP ! henry coefficient [mol/L/atm] REAL(DP) :: dryreac_sf = 0.0_DP ! dry reaction coeff.END TYPE t_typ_single
The Modular Earth Submodel System
chemical equation setsulfurhalogenshydro-
carbonsbasic
tro
po
sph
.st
rato
sph
.aq
ueo
us
kpp solver
radau5
ros3
ros2
auto
solver type timestep
manually
log
9
log
5
...
chosen chemical setup:e.g. basic gas phase
chosen solver:e.g. ros3-auto
F90-code
MBL5) MBL MBL
MBLMBLMBL
MA-ECHAM4/CHEM3)
MATCH1) MISTRA2)
MISTRAMISTRA
1) v. Kuhlmann et al. 20032) v. Glasow et al. 20023) Steil et al. 19984) Meilinger, 20005) Kerkweg, 2005
MSBM4)
MECCA(_AERO)high flexibility >>> automatic code generation
Sander et al., ACP, 2005Kerkweg et al., ACP, 2007
The Modular Earth Submodel System
MECCA(_AERO) / KPP
The Modular Earth Submodel System
MECCA(_AERO) / KPP
The Modular Earth Submodel System
IV. Evaluation of the AC-GCM ECHAM5/MESSy1 (E5/M1)
The Modular Earth Submodel System
Evaluation simulation (S1):
● ECHAM5/MESSy● T42 resolution (~2.8°x~2.8°)● 90 vertical levels (up to 0.01 hPa)● nudged in free-troposphere (200-700 hPa) towards
analysis data from ECMWF operational model● 245 reactions including hydrocarbons up to 4 carbon
atoms● feedback between chemistry and dynamics via
radiation● off-line anthropogenic emissions compiled for the
year 2000● 8 years of simulation (1998-2005)
Evaluated Setup
The Modular Earth Submodel System
MECCA* MECCA_MBL
AIRSEA(air-sea Δc)
PTRAC*(tracer properties)
DRYDEP(or EMDEP)
CVTRANS(massflux)
CONVECT(convectivecloud/rainproperties)
CLOUD(large scalecloud/rainproperties)
SEDI
M7*(aerosol mass, number, radius, σ, ρ)
SCAV*
LNOX PSC(psc region, khet*)
TROPOP(tropopause height)
TNUDGE
ONLEM(emission fluxes)
(or EMDEP)
HETCHEM(khet*)
JVAL(or PHOTO)
(J_*)
OFFLEM(emission fluxes)
H2O*
RAD4ALLTRACER_FAMILY*
couplingswitchable
* definetracers
coupling todynamics
modifytracers
usetracers
no tracermodific.
not used
Complexity
The Modular Earth Submodel System
4 year average (2000-2004, excl. 2002) of ozone (DJF) [μmol/mol]
consistentsimulation of(dynamical and chemical)state of theatmospherebetween surface and 0.01 hPa
OzoneJöckel et al., ACP, 2006
The Modular Earth Submodel System
Jan
Jan Jun
Jun
40 hPa
400 hPa
Jan
Jan Jun
Jun
OzoneComparison with O
3 database (monthly averages)
Logan et al., JGR, 1999; Jöckel et al., ACP, 2006
The Modular Earth Submodel System
E5/M1NOAA/GMD
Example:tropospheric CO
Carbon monoxide
Novelli et al., JGR, 1998Jöckel et al., ACP, 2006Pozzer et al., ACP, 2007
The Modular Earth Submodel System
Example: stratospheric temperature (SON 2003)
E5/M1 (S2) E5/M1 - MIPAS
K K
Temperature
Jöckel et al., ACP, 2006
The Modular Earth Submodel System
E5/M1 (S2) TOMSTotal Ozone [DU] 26 Sep 2002
SH vortex split 2002 reproduced
SH vortex split 2002
Jöckel et al., ACP, 2006
The Modular Earth Submodel System
Christmas Island (C2H
6)Japan (C
3H
8)
Emmons et al.,JGR, 2000; Pozzer et al., ACP, 2007
Organicspecies
Example: Propane Example: Ethane
Organic species
The Modular Earth Submodel System
Philipine Sea (C3H
6)South Atlantic (C
3H
6)
Emmons et al.,JGR, 2000; Pozzer et al., ACP, 2007
Example: Propene
Organic species
...somefail...
The Modular Earth Submodel System
Wet deposition
Example: Nitrate [mg(N)/m2/year]
Tost et al., ACP, 2007
The Modular Earth Submodel System
Tost et al., ACP, 2007
pH
precipitation pH
cloud pH
The Modular Earth Submodel System
V. Some notes on theimplementation andperformance
The Modular Earth Submodel System
Evaluation simulation (S1):● ECHAM5/MESSy T42L90MA, 1998-2005● IBM P4 “regatta”; 256 CPUs; (psi @ RZG):
~ 8 hours (wall clock) / simulation month~ 1 Tb output / simulation year
>>> parallelisation of ECHAM5: ● regional decomposition in grid-point space
NCPUS = NPROCA x NPROCB● “vector”-packing on every CPU:
(NGPBLKS-1)xNPROMA + NPROMZ
>>> 70-80% of CPU time consumed by chem. kinetics>>> high I/O load (MPI, dedicated I/O CPU, gather)
-> parallel-netCDF (MPI-2); work in progress
Performance... is highly dependent on the chosen setup ...
The Modular Earth Submodel System
NCPUS=16 NPROCB = 2N
PR
OC
A =
8
CPU 0
CPU 0CPU 1
CPU 1
CPU 7CPU 7
CPU 8
CPU 8CPU 9
CPU 9
CPU 15CPU 15
EXAMPLE: T21L19 >>> 64 longitudes x 32 latitudes x 19 levels
CP
U#
Parallelisation
choice ofNPROCANPROCB(NCPUS)
limitedby
advection
The Modular Earth Submodel System
CPU 0
CPU 0
CPU 11
CPU 11
NPROMA = 101JP=1JP=1 ...
JP
JP= ...101
JROW=1 ...
JROW= ... 2
JP=1
Vectorisation
optimalchoice ofNPROMA ?
vector-architectures: as large as possible
scalar-architecture: cache ???
on each CPU ...
The Modular Earth Submodel System
Example: T42L90MA (128 longitudes x 64 latitudes x 90 levels)WITH complex chemistry setup
NPROMAmax=NLON×NLATNCPUS
(32)(1024)
Vectorisation
The Modular Earth Submodel System
Problem (with parallelisation):role of involved processes (with different computational demands)varies with space and time
Examples:- convection / convective transport / scavenging
>>> tropics vs. polar region- chemical transformations
>>> annual and diurnal cycle (photo-chemistry !)>>> involved species~ ~ ~ stiffness of the ODE system describing the chemical
kinetics ...(reaction rates vary over several orders of magnitude)SOLVER:- 'fast'
- 'reliable', accurate- stable (!)
Performance issues
>>> 3rd order Rosenbrock with automatic time stepping
The Modular Earth Submodel System
Kerkweg et al., ACP, 2007number of kpp (ros3-auto) substeps
lowest modellayer (~ 70m)with (sea-salt-)aerosolchemistry
>>>
very stiffODE system
22 April 1999, 05:00 UTC
Automatic time-stepping
The Modular Earth Submodel System
Kerkweg et al., ACP, 2007number of kpp (ros3-auto) substeps
70 hPa(~ 50 km)only gasphasechemistry
>>>
lower stiffnesof the ODEsystem
22 April 1999, 05:00 UTC
Automatic time-stepping
The Modular Earth Submodel System
Performance ???
“FAST CHEMISTRY”
CODE OPTIMISATIONS REDUCED CHEMICAL SETUP
complex setups:- NMHC, > C
5
- Aerosol Chem.- SOA
reduced setups:- simplified chemistry
on long time scales
The Modular Earth Submodel System
Vectorisation of KPP code
Δt(1)
OUTER LOOP: grid-boxINNER LOOP: solver (automatic time stepping)
OUTER LOOP: solver (automatic time stepping) (applied to vector of grid-boxes)
+ (de-)compression algorithm
Δt(2)
Δt(3)
Δt(4)
Δt(5)
Δt(6)
Δt(7)
Δt(8)
Δt(...)
Δt(1)
Δt(2)
Δt(3)
Δt(4)
Δt(5)
Δt(6)
Δt(7)
Δt(8)
Δt(...)
“sort-out” every box for which solver finished time-stepping
“scalar”implementation
“vector”implementation
The Modular Earth Submodel System
on p4/p5:
scalarimplementationis much fasterthan vectorimplementation
>>>
overhead of(de-compression)exceeds gainof cache usageoptimisation
Vectorisation of KPP code
The Modular Earth Submodel System
no de-indexing 64.97s -qnooptde-indexing (1) 35.08 s -qnoopt
de-indexing (2) 13.84 sde-indexing (2) 13.04 s JVS(:) -> JVS(LU_NONZERO)de-indexing (2) 13.11 s JVS(:) -> JVS(LU_NONZERO)
-Q-kppdecompde-indexing (1) 12.85 s JVS(:) -> JVS(LU_NONZERO)
-Q-kppdecomp
scalar sparse LU-decomposition
p4256 CPUsT42L90MAcomplex chemistry
average wall-clock timeper simulation time step
... loop with indirect indexing ... replaced by sequence of operations
The Modular Earth Submodel System
ScalingExample: T42L90MA (128 longitudes x 64 latitudes x 90 levels)
WITH complex chemistry setup (1 day integration) ;measured with “-pg” - Option on p4/p5
P5 (8 CPUs) P4 (256 CPUs)(CPU 2) (CPU 40)
TOTAL 6938 s 645 s 10.76CHEM. SOLVER 5382 s 262 s 20.54ADVECTION 381 s 23 s 16.57
256 / (2*8) = 16
!!! preliminary results: choice of arbitrary CPU for comparison isNOT appropriate
The Modular Earth Submodel System
www.messy-interface.org
special issue in Journal ofAtmospheric Chemistryand Physicshttp://www.atmos-chem-phys.net/special_issue22.html
Thank you very muchfor your attention !
Recommended