Upload
vic
View
75
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Status Review of NOAA Implementation of ESMF Architecture (i.e. NEMS) Mark Iredell March 26 (#1-12) & April 2 (#12-32), 2008. NEMS People NEMS Projects NEMS Issues. Tom Black Atm , NAM Huiya ChuangPost Ed ColonInfrastructure Mike EkLand Jim GeigerLand (NASA) Bob GrumbineSea ice - PowerPoint PPT Presentation
Citation preview
1
Status Review of NOAA Implementation of ESMF Architecture (i.e. NEMS)
Mark IredellMarch 26 (#1-12) & April 2 (#12-32), 2008
• NEMS People• NEMS Projects• NEMS Issues
2
NEMS People
• Tom Black Atm, NAM• Huiya Chuang Post• Ed Colon Infrastructure• Mike Ek Land• Jim Geiger Land (NASA)• Bob Grumbine Sea ice• Henry Juang Atm, Dyn, GFS• Sujay Kumar Land (NASA)• Sarah Lu AQ• Avichal Mehra Ocean
• S. Moorthi Atm, Phy, GFS
• Ken Mitchell Land• Christa Peters Land (NASA)• Youhua Tang AQ• Hendrik Tolman Wave• Ratko Vasic Infrastructure• Jun Wang Infrastructure• Weiyu Yang Infrastructure• Mike Young Infrastructure• Shujia Zhou Land (NASA)• Yanqiu Zhu GSI
Biweekly UMIG (Unified Modeling Infrastructure Group) meeting
3
NEMS People
• V. Balaji GFDL• Brian Gross GFDL• Niki Zadeh GFDL
• Stan Benjamin GSD• Tom Henderson GSD• Jin Lee GSD• Jacques Middlecoff GSD
Monthly UMIT (Unified Modeling Infrastructure Telecon) meeting
Includes local UMIG group plus…
Occasionally the telecon invites a much wider group, including DTC and Navy.
4
NEMS Projects• ESMF• NEMS atmosphere• Write history and Post processor• Nesting• Aerosols and Chemistry• Land• Ocean, waves and sea ice• Ionosphere• Ensemble• Data assimilation
5
ESMF
• Earth System Modeling Framework• Community effort, partially supported by
NOAA• Proper ESMF Superstructure required for
all NEMS components• ESMF Infrastructure optional• NEMS will require ESMF 3.1.0r at this time
6
An ESMF Component
An ESMF “Component”“Parent” Driver (User Code)
SetServices()
Initialize()
Run()
Finalize()
ESMF library codeESMF library code
Run()
Import State
Export State
Internal State
Data flow in an ESMF Component Run method
7
ESMF Component rules• Only SetServices is public; Initialize, Run, and Finalize
are private but exposed by SetServices.• Import State is input; Export State is output. Both should
be fully described using ESMF metadata (names, grid, decomposition, etc).
• Coupler Component likely has Import and Export States on different grids. A Coupler Component is user code.
• Internal State is private to the component but persists across calls to Initialize, Run, and Finalize. Even other instances of the same component will have their own private internal state. An access exception may be made to other “friendly” components.
8
ESMF Infrastructure capabilities
ESMF offers or will offer capabilities for decomposed halo update and transposes as well as extensive parallel regridding tools (among other capabilities). The user is free to make use of these or not. Thus far NEMS has taken advantage of existing in-house capability and has minimally made use these ESMF capabilities. This is expected in such transition projects, and ESMF capability will be used as needed.
9
ESMF Portability, Conventions
• ESMF is designed to run on most systems Earth modeling runs on. Plus, ESMF infrastructure will support some ordinary functions that otherwise often hinder portability.
• ESMF supports but does not require the Climate-Forecast metadata conventions.Should NEMS adopt them?– Pro: wide community uses it, including NOAA labs– Con: very long names, but we can make local aliases
10
Units GRIB code
NCEP name NCEP CF
NCEP description CF Standard Name
1 PRES Pa Pa Pressure air_pressure
2 PRMSL Pa Pa Pressure reduced to MSL
air_pressure_at_sea_level
3 PTEND Pa/s Pa s-1
Pressure tendency
tendency_of_air_pressure
5 ICAHT M ICAO Standard Atmosphere Reference Hgt
6 GP m2/s2 m2 s-2 Geopotential geopotential
7 HGT gpm m Geopotential height
geopotential_height
8 DIST m m Geometric height
altitude
9 HSTDV m Standard deviation of height
10 TOZNE Dobson m Total ozone equivalent_thickness_at_stp_of_atmosphere_o3_content
CF convention examples fromhttp://cf-pcmdi.llnl.gov/documents/cf-standard-names/ncep-grib-code-cf-standard-name-mapping
11
NEMS Component Library
• Create a NEMS component library (in SVN).• Each component will have some documentation.
Moreover, each component should have a sample MAIN to run it in stand-alone mode for testing. For instance, Dynamics components may have Held-Suarez, Physics may have a single-column model.
• Each type of component will have an assigned NCEP librarian to coordinate the sub-library.
12
NEMS Atmosphere
Atmosphere
Dynamics PhysicsDyn-PhyCoupler
NMM-B
Spectral
FIM
Color KeyComponent class
Coupler class
Completed Instance
Under Development
NAM Phy
GFS PhySimple
unified atmosphereincluding digital filter
Future Development
• The goal is one unified atmospheric component that can invoke multiple dynamics and physics.• At this time, dynamics and physics run on the same grid in the same decomposition, so the coupler literally is very simple.
ARW
FVCORE
FISL
NOGAPS WRF Phy
Navy PhyCOAMPS
Regrid,Redist,Chgvar,Avg, etc
13
Original FIM run() Call Order FIM run:
do timestep = first, last dyn_1 ! 1st half of dyn calls call physics() dyn_2 ! 2nd half of dyn callsend do
Must re-order operations without changing model results
Slide courtesy Stan Benjamin
14
FIM run() Call Re-Ordering Re-order time stepping loop:
do timestep = first, last+1 if (timestep > first) & dyn_2(timestep-1) if (timestep <= last) then dyn_1(timestep) call physics(timestep) endifend do
Slide courtesy Stan Benjamin
15
New FIM run() Call Order Combine dyn_1 + dyn_2 into dyn_run and
push “if” statements inside Push physics() and if statement into
phy_rundo timestep = first, last+1 call dyn_run() call phy_run()end do
Slide courtesy Stan Benjamin
MI: Note NMM and GFS have gone through the exact same process.
16
Write history and Post processor
Atmosphere
Dynamics Physics
Color Key
Completed Instance
Under Development
Future Development
Write
Binary file
NEMSIO
Post
WriteWrite
• The Write component transfers state from model tasks to “quilt” tasks and then writes out from there.• NEMSIO is intended to be a unified optimized parallelized I/O package that can write several formats. Ideally, NEMSIO would be used for all NEMS output data and metadata (not necessarily restart data).• The unified post-processor NCEP_POST will run on the quilt tasks.• Perhaps in the future, the Dynamics and Physics will write their own history files.
unified atmosphere
Component class
Coupler class
17
Nesting
Atmosphere
Color Key
Completed Instance
Under Development
Future Development
• Parent creates children and creates proper boundary conditions in their import states.• Recursively, children create grandchildren.• Children run on different tasks from parent in concurrent nesting, same tasks in sequential nesting.• Two-way concurrent nesting would require a different time integration scheme that needs to be tested.
Atmosphere
1-way conc. NAM-NAM
1-way conc. GFS-NAM
2-way seq.
2-way conc.
moving nest
AtmosphereAtmosphere Atmosphere
AtmosphereAtmosphere
Atmosphere
Component class
Coupler class
18
NEMS Nesting• Eventually we will have all forms.
► The parent domain can have any number of children.
• Begin with 1-way grid-associated static nests.
► Telescoping: Children can have any number of children.
► Domains can run concurrently on unique sets of processors.
These three criteria require the need for general andrepeated splitting of the MPI Communicators. --Done
► One-way / Two-way.
► Static / Moving.
► Grid-associated / Not grid-associated.
Slide courtesy Tom Black
19
ATM_RUN(Fcst_time)DO Timeloop over Fcst_time at interval Δtinternal
IF(My_Compute_Tasks) PREDICT(Δtinternal) IF N_CHILDREN>0
DO N= 1, N_CHILDREN IF(My_Compute_Tasks) ISend BC’s to child IF(Child_Compute_Tasks(N)) Recv from parent CALL ATM_RUN(Child(N), Fcst_time=Δtinternal)
ENDDOIF(Two-way) Send children’s state back to me and blend into
mineENDIF
ENDDOEND ATM_RUN
Nesting – ATM_RUN
Recursive call
“Mini” timeloop
Slide courtesy Tom Black
20
Aerosols and Chemistry
Dynamics Physics
Color Key
Completed Instance
Under Development
Future DevelopmentAerosols
GOCART
• GOCART is Goddard aerosol model. CBM is VOC-NOX-O3 mechanism (chemistry in CMAQ model).• Aerosol and chemistry components compute sources, sinks, and chemical transformation.• Aerosol and chemistry components may need to expose internal state to avoid memory copies.• Atmosphere must be ready to do convective fluxes and removal as well as advection and diffusion.
CBM-5
reduced chem
Atmosphere
WRF-chem
Component class
Coupler class
21
Land (alternative 0)Color Key
Completed Instance
Under Development
Future Development
• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?
Dynamics Physics Land
LIS-Noah
others
Atmosphere
Component class
Coupler class
22
Land (alternative 0b)Color Key
Completed Instance
Under Development
Future Development
• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?
Dynamics Physics
Land
LIS-Noah
others
Atmosphere
Component class
Coupler class
Main
23
Land (alternative 1)Color Key
Completed Instance
Under Development
Future Development
• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?
Dynamics Radiation Land
LIS-Noah
others
PBL/Moist
Atmosphere
Component class
Coupler class
24
Land (alternative 1b)Color Key
Completed Instance
Under Development
Future Development
• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?
Dynamics RadiationLand
LIS-Noah
others
PBL/Moist
Atmosphere
Component class
Coupler class
25
Land (alternative 2)Color Key
Completed Instance
Under Development
Future Development
• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?
Dynamics
Atmosphere
Radiation Land PBL/Moist
Physics
Aerosols/Chemistry
Component class
Coupler class
26
Land (alternative 3)Color Key
Completed Instance
Under Development
Future Development
• Land is responsible for computing surface fluxes.• Land may need to be invoked inside physics, since it needs radiation and surface layer needs it.• Implicit solving would require iteration between Land and PBL/Moist.• How deep should ESMF go?
Dynamics
Atmosphere
Radiation PBL/Moist Aerosols/Chemistry
Land Ice Waves Ocean
Physics
Surface
Component class
Coupler class
27
GEOS-5 AGCM COMPONENT STRUCTURE
AGCM
DYNAMICS
GWDFVCORE
RADIATION
COLUMN PROCESSES
SOLAR
IR
MOIST
TURBSURFACE
CHEM
LAKE
SALTWATER
CAP
(qv, ql, qi, cl,…)
(u, v, T, p)
(Tc, qc, Td,...)
(Ts,Fi...)VEGDYNCATCH
HISTORY
ICELAND(Ts,Fi...)
(O3,…,Qdust,…)
(Tskin,Hskin,...)(Ci,...)
OGCM
Slide courtesy Max Suarez
28
Ocean, waves and sea ice
Atmosphere
Color Key
Completed Instance
Under Development
Future Development
• Under ESMF NEMS, there should be not much difference between SPMD and MPMD codes. However, the MPMD code might not be as portable. MPMD will allow more flexible optimizations though.• Due to some rapidly interacting physics (sea ice thermodynamics and wave roughness), some coupling would have to be frequent.
Atm-OcnCoupler Ocean
GFS Atm-OcnCoupler MOM4
NCEP CFS non-ESMF MPMD
Waves
GFS WW III MOM4
NEMS-GFS HYCOM
Ice
H-ice
N-ice
Earth system model Component class
Coupler class
29
Ionosphere
Atmosphere
Color Key
Completed Instance
Under Development
Future Development
• GFS-IDEA is the GFS extended up to 600 km (from 60 km). GIP is Global Ionosphere-Plasmasphere• The coupling will be frequent and three-dimensional, providing optimization challenges.
Atm-IonCoupler Ionosphere
GFS-IDEA GIPTBD
Whole atmosphereComponent class
Coupler class
30
Ensemble
Atmosphere
Color Key
Completed Instance
Under Development
Future Development
• The atmosphere ensemble coupler provides stochastic forcing for the ensemble.• The current non-NEMS GEFS capability will be converted to NEMS.
Atm ensCoupler
GFS
NEMS-GFS
Atmosphere Atmosphere
Ensemble
NEMS-any
Component class
Coupler class
31
Data assimilation
Model
Color Key
Completed Instance
Under Development
Future Development
• Frequent coupling of atmosphere and analysis will be necessary as hourly analyses are required.• Whole Earth System Nested Ensemble may include any components in NEMS from previous slides.• 4DVAR is supported in this scenario. The inner loop of the variational system could be entirely within the analysis component, or it could invoke adjoint methods with the NEMS atmosphere.• The model-analysis coupler may change grids and variables. When returning to the model state, only the analysis increments will be interpolated.
Data assimilation
Mdl-AnlCoupler Analysis
GFS GSI
NAM
Whole Earth System Nested
Ensemble
Component class
Coupler class
32
NEMS Issues• ESMF version
– NEMS will require ESMF 3.1.0r, which will be the latest public release in spring 2008. NEMS may later need ESMF 3.1.1.
– All components must use the same version• MAPL
– GSFC ESMF wrapper– Used by MOM4, GOCART, FVCORE– Compatible with NEMS?
• Code Repositories– Operationally secure– Collaboration friendly
• Experiment Launcher compatible with operations• Documentation and Support