Introduction to the Earth System Modeling FrameworkInternational Workshop on Next Generation Climate Models forAdvanced High Performance Computing Facilities

March 12, 2002

Cecelia DeLuca / Scientific Computing Division / NCARChris Hill / Massachusetts Institute of TechnologyV. Balaji / SGI – NOAA Geophysical Fluid Dynamics Laboratory

Outline

• Background and Motivation• ESMF Design Overview• ESMF Codes• Milestones• Development Plan• Beyond 2004: ESMF Evolution• Additional Information and Events

Trends in Earth System ModelingIn climate research and numerical weather prediction … • Increased emphasis on detailed representation of individual physical processes• Requires many teams of specialists to contribute components to a coupled system

In computing technology …• Increase in complexity and variety in high-performance computing hardware

Community Response• Abstraction of underlying hardware to provide uniform programming model across

architectures• Modular design for interchangeable dynamical cores and larger components• Development of prototype frameworks - GFDL (FMS), NASA/GSFC (GEMS), others

The Earth System Modeling Framework (ESMF) aims to unify and extend these efforts

Project Background

Collaborators: NASA/GSFC-DAO, NASA/GSFC-NSIPP, LANL, ANL, University of Michigan, MIT, NCAR-SCD, NCAR-CGD, NCAR-MMM, NOAA/NCEP, NOAA/GFDL

Term: 3 years, start March 2002Budget: $9.8M

NASA’s Earth Science Technology Office (ESTO) proposed the creation of the ESMF in a September 2000 Cooperative Agreement Notice

Large collaboration developed three linked proposals in response:Part I: Core ESMF Development (PI: Killeen, NCAR)Part II: Modeling Applications (PI: Marshall, MIT)Part III: Data Assimilation Applications (PI: da Silva, NASA DAO)

Project Organization

Prognostic ModelDeployment

NASA ESTO

NSF NCAR PI MIT PI NASA DAO PI

Data AssimilationDeployment

Core Framework Development

Part I Proposal Specific Milestones

Part II Proposal Specific Milestones

Part III Proposal Specific Milestones

Joint Milestones Joint Milestones Joint Milestones

Part I

Part II

Part III

Joint Specification TeamRequirements AnalysisSystem Architecture

API Specification

Objectives of the ESMF1. Facilitate the exchange of scientific codes (interoperability)2. Promote the reuse of standardized technical software

while preserving computational efficiency3. Focus community resources to preserve performance

portability in a volatile computing environment4. Share overhead costs of the housekeeping aspects of

software development5. Provide greater institutional continuity to model

development efforts

Framework ArchitectureComponents and Coupling

gridded component interfacedata transfers/couplingcollective I/O

Fields and Grids Fields field and field bundle data and metadatafield I/O

Grids physical grid descriptiongrid decomposition

Parallel Utilities

transpose, halo, etc.abstracted machine layout

Low-Level Utilities event alarmsperformance profilingI/O primitivescommunication primitives, etc.

Application Architecture

External Libraries

Low Level Utilities

Fields and Grids Layer

Model Layer

Coupling Layer

ESMF Infrastructure

User Code

ESMF Superstructure

BLAS, MPI, NetCDF, …

ESMF Codes:Modeling Applications

GFDL FMS B-grid atmosphere at N45L18

FMS spectral atmosphere at T63L18

FMS MOM4 ocean model at 2°x2°xL40

FMS HIM isopycnal C-language ocean model at 1/6°x1/6°L22

MIT MITgcm coupled atmosphere/ocean at 2.8°x2.8°, atmosphere L5, ocean L15

MITgcm regional and global ocean at 15kmL30

NSIPP NSIPP atmospheric GCM at 2°x2.5°xL34 coupled with NSIPP ocean GCM at 2/3°x1.25°L20

NCAR/LANL CCSM2 including CAM with Eulerian spectral dynamics and CLM at T42L26 coupled with POP ocean and data ice model at 1°x1°L40

ESMF Codes:Data Assimilation Applications

DAO PSAS interface layer with 2O0K observations/day

CAM with finite volume dynamics at 2°x2.5°L55, including CLM

NCEP Global atmospheric spectral model at T170L42

SSI analysis system with 250K observations/day, 2 tracers

WRF regional atmospheric model at 22km resolution CONUS forecast 345x569L50

NSIPP ODAS with OI analysis system at 1.25°x1.25°L20 resolution with ~10K observations/day

MIT MITgcm 2.8° century / millennium adjoint sensitivity

Final Milestones

• Tested, optimized core framework software and documentation– Support for many platforms, including commodity clusters

• All ESMF codes will achieve full ESMF compliance– Will use coupling superstructure and utilities

• 8 Interoperability demonstrations– Example: NCAR CAM with MITgcm ocean– Multiple time scales:

• NWP, mid-range forecast experiments• Coupled seasonal forecasts• Interannual/decadal variability• Centennial simulations

– Will require tuning and validation to be scientifically useful

Development Plan• Requirements

– Intensive interaction involving ESMF collaborators and broader community, ESMF Community Requirements Meeting

– High-level results presented in General Requirements Document, detailed requirements documents prepared later

• Design – Design study includes examination of FMS, GEMS, MCT, and

other existing Earth science frameworks– Results presented in Architecture Report, detailed design

documents for specific components prepared later– Community review of ESMF interface specification

Development Plan, cont.• Implementation

– Implementation study will determine implementation language, language interoperability strategy, programming model

– Examine viability of supporting software tools, e.g. CCA, Cactus

– Results presented in Implementation Report• Best Practices

– Early distribution of Software Developer’s Guide will encourage consistent standards, conventions and practices throughout the project

Beyond 2004:ESMF Evolution• Maintenance and management

– NCAR commitment to maintain and support core ESMF software– Persistence of ESMF Executive Committee and Advisory Board

• Technical evolution– Functional extension: support for advanced data assimilation

algorithms, additional grids, new domains– Earth System Modeling Environment, including web/GUI interface,

databases of components and experiments, links to GRID services

Additional Information and Upcoming EventsESMF Community Requirements Discussion and ReviewOpen MeetingIn conjunction with the Spring American Geophysical Union Meeting May 30, 2002

ESMF website: http://www.esmf.ucar.edu

Questions: esmf@ucar.edu