NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH ESMF Infrastructure Layer 2 nd ESMF Community Meeting, GFDL, May 15 th 2003 ,

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH

ESMF Infrastructure Layer

2nd ESMF Community Meeting, GFDL, May 15th 2003

http://www.esmf.ucar.edu, [email protected]


Talk Outline

• What is the infrastructure layer– What is it for– What it contains

• ESMF infrastructure in this release – ESMF version 1.0 function call† examples (code!!!)

(†aka API or methods)

• Next steps….• Infrastructure Utility Example


Infrastructure Layer

1. A standard software platform for enabling interoperability (developing couplers, ensuring performance portability).

2. Set of reusable software for Earth science applications. Streamlined development for researchers.

NCARAtmosphere

GFDLOcean NSIPP

Land

NCARAtmosphere

GFDLOcean NSIPP

Land

MySea Ice


Infrastructure Layer Scope

SupportSupport for• Physical Grids• Regridding• Decomposition/composition• Communication• Calendar and Time• I/O• Logging and Profiling

ESMF Infrastructure

User Code

ESMF Superstructure


2. ESMF provides a toolkit that components use toi. ensure interoperabilityii. abstract common services

Location Within ESMFLocation Within ESMF

Component: run(), checkpoint()

Field: halo(), import(), export() + I/O

Grid: regrid(), transpose() + Metrics

Layout, PEList, Machine Model

Application ComponentGridded

ComponentsCoupler

Components

1. ESMF provides an environment for assembling components.

INFRASTRUCTURE

INFRASTRUCTURE

LAYERLAYER


Talk Outline

• What is the infrastructure layer– What is it for– What it contains



• Next steps….• Infrastructure utility example

Infrastructure Internal Organization

MachineModel

DELayout

DistGridPhysGrid

Bundle

Array

Time, Alarm, Calendar, LogErr, I/O, Attributes

F90

C++

Field

Grid

Regrid

Comm

Data Communications

Route

Two tiers

Class hierarchy for data and communications


ESMF Infrastructure Fields and Grids

Field Contains array, grid, array to grid Contains array, grid, array to grid mapping and metadatamapping and metadata

Array Holds actual data e.g. temperature, wind Holds actual data e.g. temperature, wind speed etc…speed etc…

Grid Contains physical and distributed gridContains physical and distributed grid

Physical Grid Grid metrics, e.g. lat-lon coords, Grid metrics, e.g. lat-lon coords, spacings, areas, volumes etc..spacings, areas, volumes etc..

Distributed Grid Parallel decomposition informationParallel decomposition information


Creating an ESMF Field

1. Fortran Array

2. Array Attach

3. Field Attach

Array

real, dimension(100,100) :: arr

esArr = ESMF_ArrayCreate(arr,…)

ESMF_Array

info.

ESMF_Array

info.

ESMF_FieldAttachArray(esFld,esArr,…)

ESMF_Fieldmetadata

ESMF_Grid


Creating an ESMF Grid

ESMF_Grid

PhysGrid

DistGrid

call ESMF_PhysGridCreate…(…) call ESMF_DistGridCreate…(…)


Field and Grid Together :grid = ESMF_GridCreate(…,layout,…) :field_u = ESMF_FieldCreate(grid, array) :

1. Create field distributed over a set of decomposition elements (DE’s).

2. Domain decomposition determined by DELayout, layout.3. Each object (grid and field_u) has internal representation.4. Other parts of the infrastructure layer use the internal

representation e.g.• Regrid() – interpolation/extrapolation + redistribute over DE’s• Redistribution() - general data rearrangement over DE’s• Halo() – specialized redistribution

ESMF_Array

info.

ESMF_Fieldmetadata

ESMF_Grid


Regrid• Function mapping field’s array to a different physical and distributed grid.

• RegridCreate() – creates a Regrid structure to be used/re-used

regrid = ESMF_RegridCreate(src_field, dst_field, method, [name], [rc])

Source, Dest field can be empty of field data (RegridCreate() uses grid metrics)– PhysGrid, DistGrid info used for setting up regrid– Resulting regrid can be used for other fields sharing same Grid

• Method specifies interpolation algorithm. For example, bilinear, b-spline, etc…

},,{},,{ DPADPA


Regrid Interface (cont)• RegridRun() – performs actual regridding call ESMF_RegridRun(src_field, dst_field, regrid,[rc])

– Communication and interpolation handled transparently.

• RegridDestroy() – frees up memory call ESMF_RegridDestroy(regrid,[rc])

src_field

dst_field


Redistribution• No interpolation or extrapolation• Maps between distributed grids, field’s array and physical grid are the

same i.e.

• Example: layout() created two distributed grids one decomposed in X and one decomposed in Y.

– Redistribution() function maps array data between the distributions (a transpose/corner turn)

– Communication handled transparently

},,{},,{ DPADPA

src_field dst_field


Halo • Fields have distributed index space of

– Exclusive {E}, compute {C} and local {L} region where

• Halo() fills points not in {E} or {C} from remote {E} e.g

{E}

{C}{L}

{C}{L}

{E}

ECL

call ESMF_FieldHalo(field_foo, status)

DE3 DE4

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICHMore functions in reference manual e.g


Bundle and Location Stream

• ESMF_Bundle– collection of fields on the same grid

• ESMF_LocationStream– Like a field but…..– unstructured index space with an associated

physical grid space– useful for observations e.g. radiosonde, floats

• Functions for create(), regrid(), redistribute(), halo() etc…


ESMF Infrastructure Utilities• Clock• Alarm• Calendar• I/O• Logging• Profiling• Attribute• Machine modeland comms

Ensures consistent time between components

Provides field level I/O in standard forms – netCDF, binary, HDF, GRIB, Bufr

Consistent monitoring and messaging

Consistent parameter handling

Hardware and system software hiding. Platform customizable


Time• Standard type for any component

• Calendar (support for range of calendars)

! initialize stop time to 13May2003, 2:00 pmcall ESMF_TimeInit(inject_stop_time, & YR=int(2003,kind=ESMF_IKIND_I8), & MM=off_month, DD=off_day, H=off_hour, M=off_min, & S=int(0,kind=ESMF_IKIND_I8), & cal=gregorianCalendar, rc=rc)

do while (currTime .le. inject_stop_time ) : call ESMF_ClockAdvance(localclock, rc=rc) call ESMF_ClockGetCurrTime(localclock, currtime, rc)end

call ESMF_CalendarInit(gregorianCalendar, ESMF_CAL_GREGORIAN, rc)


I/O• Field level binary, netCDF, HDF, GRIB, bufr

– Currently I/O piped through 1 PE call ESMF_FieldAllGather(field_u, outarray, status) if (de_id .eq. 0) then write(filename, 20) "U_velocity", file_no call ESMF_ArrayWrite(outarray, filename=filename,

rc=status) endif call ESMF_ArrayDestroy(outarray, status)

– Current system fixed textual.


Attributes• Flexible parameter specs, configuration e.g. file

of parameters

Code


Internal Classes

• Machine model– Captures system attributes, CPU, mem,

connectivity graph– Useful for defining decomposition, load-balance,

performance predictions.

• Comms– Communication driver, allows bindings to MPI,

shared memory, vendor system libraries

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICHComms Performance Test

Right mix (green) on Compaq gives x2 realized bandwidth (in large message limit)


Talk Outline• What is the infrastructure layer

– What is it for– What it contains



• Next steps….• Infrastructure utility example


Regrid Next Steps• Support for all ESMF Grids• Support for regridding methods:BilinearBicubic1st-order Conservative2nd-order ConservativeRasterized ConservativeNearest-neighbor distance-weighted averageSpectral transforms1-d interpolations (splines)Index-space (shifts, stencils)Adjoints of many above

Halo Next Steps• Support for periodicity• More general haloing

– in tandem with distributed grid evolution

• Adjoint forms


Distributed Grid • Regular 2d already supported• Next steps

– Generalized 1d decomposition– Extend support for 2d and quasi regular

decomposition – Spectral grid decompositions

• Larger set of metrics, grids• High level routines for rapid definition of common grids.

Physical Grid Next Steps


I/O Next Steps• Broaden format set, binary, netCDF, HDF, GRIB,

bufr• Improve parallelization

• Full support for alarms• Broader functionality

Time/Logging/Profiling Next Steps


Summary

• ESMF Infrastructure Layer– Comprehensive class structure available in

version 1.0– Over the coming year significant extension of

functionality will take place.– Feedback and comments on version 1.0 welcome

http://www.esmf.ucar.edu


ESMF Infrastructure Utility detailed example

Earl Schwab, ESMF Core Team, NCAR

Time Manager


What is Time Manager?

• Clock for time simulation• Time representation• Time calculator• Time comparisons• Time queries• F90 API, C++ implementation


Clock for time simulation

type(ESMF_Clock) :: clock

call ESMF_ClockInit(clock, timeStep, startTime, stopTime, rc=rc)

do while (.not.ESMF_ClockIsStopTime(clock, rc))

! Do application work

.

.

.

call ESMF_ClockAdvance(clock, rc=rc)

end do


Clock (cont.)

• Clock queries/commands

call ESMF_ClockGetCurrTime(clock, currTime, rc)call ESMF_ClockSetCurrTime(clock, currTime, rc)call ESMF_ClockGetTimeStep(clock, timeStep, rc)call ESMF_ClockSetTimeStep(clock, timeStep, rc)call ESMF_ClockGetAdvanceCount(clock, advanceCount, rc)call ESMF_ClockGetStartTime(clock, startTime, rc)call ESMF_ClockGetStopTime(clock, stopTime, rc)call ESMF_ClockGetPrevTime(clock, prevTime, rc)call ESMF_ClockSyncToWallClock(clock, rc)


Time Representation

type(ESMF_Calendar) :: calendar1

call ESMF_CalendarInit(calendar1, ESMF_CAL_GREGORIAN, rc)

- ESMF_CAL_GREGORIAN (3/1/-4800 to 10/29/292,277,019,914)

- ESMF_CAL_JULIAN (+/- 106,751,991,167,300 days)

- ESMF_CAL_NOLEAP

- ESMF_CAL_360DAY


Time Representation (cont.)

type(ESMF_Time) :: time1call ESMF_TimeInit(time1, YR=int( 2003 ,kind=ESMF_IKIND_I8), &

MM= 5 , DD= 15 , H= 15 , cal=calendar1, rc=rc)

- YR, MM, DD, H, M, S F90 optional- D, H, M, S arguments

type(ESMF_TimeInterval) :: timeInterval1call ESMF_TimeIntervalInit(timeInterval1,

D=int( 90 ,kind=ESMF_IKIND_I8), rc=rc)

- D, H, M, S F90 optional arguments


Time Calculator

• Time differencing• Time increment/decrement by a time interval• Time interval arithmetic (+, -, *, /)


Time Calculator (cont.)

call ESMF_TimeInit(time1, YR=int( 2003 ,kind=ESMF_IKIND_I8), & MM= 5 , DD= 15 , cal=gregorianCalendar, rc=rc)

call ESMF_TimeInit(time2, YR=int( 2003 ,kind=ESMF_IKIND_I8), & MM= 3 , DD= 26 , cal=gregorianCalendar, rc=rc)

call ESMF_TimeIntervalInit(timeInterval1, D=int( 90 ,kind=ESMF_IKIND_I8, rc=rc)

timeInterval2 = time2 - time1time1 = time1 + timeInterval1 ! Uses F90 overloaded timeInterval3 = timeInterval1 * 2 ! operators

double precision :: ratioratio = timeInterval1 / timeInterval2


Time Comparisons

>, <, >=, <=, ==, <> F90 overloaded operators between any 2 times or time intervals

if (time1 < time2) then…

end if

if (timeInterval1 .ge. timeInterval2) then…

end if


Time Queries

call ESMF_TimeGet(time1, YR=yr, MM=mm, DD=dd, H=h, M=m, S=s, rc=rc)call ESMF_TimeIntervalGet(timeInterval1, D=d, H=h, M=m, S=s)

call ESMF_TimeGetDayOfYear(time1, dayOfYear, rc) ! double or integercall ESMF_TimeGetDayOfMonth(time1, dayOfMonth, rc)call ESMF_TimeGetDayOfWeek(time1, dayOfWeek, rc)call ESMF_TimeGetMidMonth(time1, midMonth, rc)

call ESMF_TimeGetString(time1, string, rc) ! 2003-05-14T12:20:19 (ISO 8601)call ESMF_TimeIntervalGetString(timeInterval1, string, rc) ! P1DT12H0M0S (ISO)

call ESMF_TimeGetRealTime(time1, rc)


More information

• ESMF User’s Guide• ESMF Reference Manual• ESMF Requirements Document• Time Manager F90 API & examples source code

esmf_1_0_0_r/src/Infrastructure/TimeMgr/interface

esmf_1_0_0_r/src/Infrastructure/TimeMgr/examples

Documents

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH ESMF Infrastructure Layer 2 nd ESMF Community Meeting, GFDL, May 15 th 2003 ,