Parallel grib2 in NCEP POST

Jun Wang

Acknowledgment:

Krishna Kumar, Boi Vuong, Robert Grumbine, Michelle Mainelli, Mark Iredell, Hui-ya Chuang,

Manuel Pondeca, Vera Gerald

Introduction

• Changes are made into NCEP POST to add an option to generate grib2 output files.

1. Using XML post control files 2. Apply NCO’s grib2 templates and g2 lib to generate grib2 message 3. parallel generate grib2 messages and parallel write out the messages

• The changes are backward compatible, nothing need to be changed to run NCEP POST for grib1 output

XML post control file

• Bob Grumbine initiated the investigation to ingest the XML post control file in FORTRAN code, he presented the results at NEMS meeting.

• XML format post control file is considered to accommodate the changes required by generating the grib2 message.

• Benefits to use XML file: readability, flexibility, portability, and easy to maintain

Current NCEP POST control file KGTYPE******I5*******:(00255)********START OF THIS OUTPUT GRID****** IMDLTY *I5* :(00089) DATSET *A6* :(BGDAWP) (PRESS ON MDL SFCS ) SCAL=( 3.0) L=(11000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000) (HEIGHT ON MDL SFCS ) SCAL=(-5.0) L=(11000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000)....***DONE WITH THIS GRID*** KGTYPE******I5*******:(00255)********START OF THIS OUTPUT GRID****** IMDLTY *I5* :(00089) DATSET *A6* :(BGRDSF) (PRESS ON MDL SFCS ) SCAL=( 3.0) L=(10000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000) (HEIGHT ON MDL SFCS ) SCAL=(-5.0) L=(10000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000)...***DONE WITH THIS GRID***

XML post control file template for grib2 <?xml version="1.0"?> <template> <options strict="yes" dynamicstings="no" rootname="postcntrl" /> <typedef name="param_t"> <component name="pname" type="word" length="30" /> <component name="table_info" type="word" length="10"/> <component name="stats_proc" type="word" length="20" /> <component name="fixed_sfc1_type" type="word" length="80" /> <component name="scale_fact_fixed_sfc1" type="integer" dimension="1" /> <component name="level" type="real" dimension="1" default="0."/> <component name="fixed_sfc2_type" type="word" length="80" /> <component name="scale_fact_fixed_sfc2" type="integer" dimension="1" /> <component name="level2" type="real" dimension="1" /> <component name="decimal_scale" type="real" dimension="1" /> <component name="binary_scale" type="real" dimension="1" /> <component name="number_bits" type="real" dimension="1" /> <component name="stat_miss_val" type="integer" /> <component name="leng_time_range_prev" type="integer" /> <component name="time_inc_betwn_succ_fld" type="integer" /> <component name="type_of_time_inc" type="word" length="80" /> <component name="stat_unit_time_key_succ" type="word" length="20" /> <component name="bit_map_flag" type="word" length="20" /> </typedef>

XML post control file template (cont.)

<typedef name="paramset_t"> <component name="datset" type="word" length="6" /> <component name="grid_num" type="integer" /> <component name="sub_center" type="word" length="20" /> <component name="version_no" type="word" length="20" /> <component name="local_table_vers_no" type="word" length="20" /> <component name="sigreftime" type="word" length="20" /> <component name="prod_status" type="word" length="20" /> <component name="data_type" type="word" length="20" /> <component name="gen_proc_type" type="word" length="20" /> <component name="time_range_unit" type="word" length="30" /> <component name="orig_center" type="word" length="50" /> <component name="gen_proc" type="word" length="30" /> <component name="packing_method" type="word" length="20" /> <component name="field_datatype" type="word" length="20" /> <component name="comprs_type" type="word" length="20" /> <component name="param" type="param_t" dimension="1" /> </typedef> <variable name="paramset" type="paramset_t" dimension="1"></variable> </template>

XML post control file - NMMB<?xml version="1.0"?><postcntrl> <paramset> <datset>BGDAWP</datset> <grid_num>255</grid_num> <sub_center>ncep_nco</sub_center> <version_no>v2003</version_no> <local_table_vers_no>local_tab_no</local_table_vers_no> <sigreftime>fcst</sigreftime> <prod_status>oper</prod_status> <data_type>fcst</data_type> <gen_proc_type>fcst</gen_proc_type> <time_range_unit>hour</time_range_unit> <orig_center>nws_ncep</orig_center> <gen_proc>nmm_8km</gen_proc> <packing_method>jpeg2000</packing_method> <field_datatype>fltng_pnt</field_datatype> <comprs_type>lossless</comprs_type>

<param> <pname> PRES</pname> <fixed_sfc1_type>hybrid_lvl</fixed_sfc1_type> <level> 1 2 </level> <binary_scale> 0 0</binary_scale> <decimal_scale> -2 -2</decimal_scale> </param> </paramset>

<paramset> <datset>BGRDSF</datset> <grid_num>255</grid_num> <sub_center>ncep_nco</sub_center> <version_no>v2003</version_no> <local_table_vers_no>local_tab_no</local_table_vers_no> <sigreftime>fcst</sigreftime> <prod_status>oper</prod_status> <data_type>fcst</data_type> <gen_proc_type>fcst</gen_proc_type> <time_range_unit>hour</time_range_unit> <orig_center>nws_ncep</orig_center> <gen_proc>nmm_8km</gen_proc> <packing_method>jpeg2000</packing_method> <comprs_type>lossless</comprs_type>

<param> <pname> PRES</pname> <fixed_sfc1_type>hybrid_lvl</fixed_sfc1_type> <level> 1 </level> <binary_scale> 0</binary_scale> <decimal_scale> -2</decimal_scale> </param>

</paramset></postcntrl>

Fortran90 code to read XML file

• Fortran90 code postcntrl_t.f90 is generated through XML_FORTRAN parser given the XML post template file, the postcntrl_t.f90 will read all the actual xml post control file. This just needs to be done once.

• A library is created from the postcntrl_t.f90 and the xml parser, this library is then linked to the post executable.

•In ncep post to output grb2 message, the READCNTRL_xml.f is created to ingest the xml post control file into POST.

postcntrl_t.f90 Fortran90 code

module xml_data_postcntrl_t use READ_XML_PRIMITIVES use XMLPARSE implicit none integer, private :: lurep_ logical, private :: strict_

type param_t character(len=30) :: pname='' character(len=10) :: table_info='' character(len=20) :: stats_proc='' character(len=80) :: fixed_sfc1_type='' integer, dimension(:), pointer :: scale_fact_fixed_sfc1 => null() real, dimension(:), pointer :: level => null() character(len=80) :: fixed_sfc2_type='' integer, dimension(:), pointer :: scale_fact_fixed_sfc2 => null() real, dimension(:), pointer :: level2 => null() real, dimension(:), pointer :: decimal_scale => null() real, dimension(:), pointer :: binary_scale => null() real, dimension(:), pointer :: number_bits => null() integer :: stat_miss_val=0 integer :: leng_time_range_prev=0 integer :: time_inc_betwn_succ_fld=0 character(len=80) :: type_of_time_inc='' character(len=20) :: stat_unit_time_key_succ='' character(len=20) :: bit_map_flag=''end type param_t

type paramset_t character(len=6) :: datset='' integer :: grid_num=255 character(len=20) :: sub_center='' character(len=20) :: version_no='' character(len=20) :: local_table_vers_no='' character(len=20) :: sigreftime='' character(len=20) :: prod_status='' character(len=20) :: data_type='' character(len=20) :: gen_proc_type='' character(len=30) :: time_range_unit='' character(len=50) :: orig_center='' character(len=30) :: gen_proc='' character(len=20) :: packing_method='' character(len=20) :: field_datatype='' character(len=20) :: comprs_type='' type(param_t), dimension(:), pointer :: param => null()end type paramset_t type(paramset_t), dimension(:), pointer :: paramset => null()Containssubroutine read_xml_file_postcntrl_t(fname, lurep, errout) character(len=*), intent(in) :: fname integer, intent(in), optional :: lurep logical, intent(out), optional :: errout

GRIB2 Sections

• 0 – Indicator Section - ‘GRIB’, length, version, discipline – (16 octets long)

• Section 1 – Identification - Ref time, orig center, table versions, …(length of the section is expressed as the first four

octets (32 bits)

- the section number in the fifth octet and octets beyond 21 are reserved for future use)

• 2 – Local Use Section – optional ================== • 3 – Grid Definition Section ======= |• 4 – Product Definition Section ==== | |

- parameter, level, valid time | | |

• 5 – Data Representation Section | | | - packing algorithm used, info needed to unpack | | |

• 6 – Bit Map Section | | | • 7 – Data Section ==================

-packed data

• 8 – End Section = ‘7777’ Courtesy of Krishna Kumar

Grib1 message

0 – Indicator Section - ‘GRIB’ (octet 1-4 = 32bits), length of message (octet 5-7=24bits), edition number -1 (octet 8=8bits) - 1 octet = 1byte = 8bits - octets are numbered 1, 2, 3, etc., Bit positions within octets are from bit 1 to bit 8. - bit 1 = most significant bit & bit 8 = least significant)

1 – Product Definition Section (PDS) - Length of PDS, parameter table version number, ID of center, generating process ID, grid identification (Table B), Ref time, parameter and units, level, valid time …

2 – Grid Definition Section (GDS) – optional - Length of sec., sec. no, defn of grid surf and geometry of data values within the surface

3 – Bit-Map Section (BMS) – optional - Length of sec., sec. no, indication of presence or absence of data at each of the grid points, as applicable example sst

4 – Binary Data Section (BDS) - Length of sec., sec. no, data values

5 – End Section = ‘7777’

Courtesy of Krishna Kumar

GRIB1 & GRIB2 Major Differences

One GRIB2 message can contain multiple fields. - Sequences of GRIB sections 2 to 7, sections 3 to 7 or sections 4 to 7 may be repeated within a single GRIB message.

- Once a section is repeated all the following sections must appear in order. - A section remains in effect until redefined. - Could be used for vector fields such as u & v components, or ensemble members - IEEE single precision floating point representation used instead of standard IBM

representation. The representation occupies four octets.

GDS, PDS, DRS, and DS are all template driven. Octet assignments within a section are based on the Template specified.

Parameters are defined by three values: Discipline (specified in Section 0) Category (PDS) Parameter number (PDS).

Courtesy of Krishna Kumar

GRIB1 and GRIB2 Major differences contd.

• Table versions, originating center, initial reference time and grid ID removed from GRIB1 PDS. Most are now in Identification Section.

• All longitudes MUST be in degrees East and be between 0 and 360 degrees. No negatives. No degrees West. No change in latitude definitions –900 to 900.

• GDS and BMS no longer optional. They must be included. Only Section 2 (Local Use) is optional.

• Latitudes, Longitudes are in units of micro-degrees (10 –6 degrees). Includes distance between grid points for Lat/Lon grids (GRIB1 was

milli-degrees)• All other grid point increments are in millimeters (GRIB1 was in

meters)• Floating point values are now stored in IEEE format (GRIB1 was

IBM 360/195 format HDS)

Courtesy of Krishna Kumar

GRIB2 in NCEP operations

Encoding Routines

GRIB2 message can contain gridded fields for many parameters on a number of different grids.

Start a new GRIB2 message with a call to:

(1) call GRIBCREATE – encodes sections 0 and 1 at the beginning of message.

(2) call ADDLOCAL – adds a Local Use Section (Section 2) optional

(3) call ADDGRID – encodes a grid definition into Section 3. Defines the geometry of the data values in the fields that follow it. ADDGRID can be called again to change the grid definition describing subsequent data fields.

(4) call ADDFIELD – adds each data field which adds Sections 4, 5, 6, and 7 to the message.

(5) call GRIBEND – adds the final section 8 to the message which updates the length of the message. GRIBEND is required for each GRIB2 message.

Courtesy of Krishna Kumar

Grib2 naming conventionGRIB2 parameter tablediscipline category parameter_number table_information NCEP_mnemonic!! GRIB2 - TABLE 4.2-0-0 PARAMETERS FOR DISCIPLINE 0 CATEGORY 0! 0 0 0 0 TMP 0 0 1 0 VTMP 0 0 2 0 POT 0 0 3 0 EPOT 0 0 4 0 TMAX 0 0 5 0 TMIN….. 0 0 193 1 TTRAD…..

table_information:• 0 for the parameter from the WMO table and 1 for the parameter from teh NCEP local table.• WMO parameters have values ranging from 0 to 191,• NCEP local table parameters have values from 192 to 254,• the parameter value 255 is a missing or undefined.

NCEP_mnemonic:locally defined at NCEP

Grib2 templates---- grib2_all_tables_module Krishna and Boi created grib2_all_tables_module - data types and subroutines to match the mnemonic

names with the actual values defined by grib2 tabletype subcenters

character(len=20) :: subcenkey

integer :: subcenval

end type subcenters

type(subcenters), dimension(MAXSUBCEN) :: tablec

data tablec(1) /subcenters('ncep_reanl',1)/

data tablec(2) /subcenters('ncep_ensem',2)/

data tablec(3) /subcenters('ncep_nco',3)/

data tablec(4) /subcenters('ncep_emc',4)/

data tablec(5) /subcenters('ncep_hpc',5)/

data tablec(6) /subcenters('ncep_opc',6)/

data tablec(7) /subcenters('ncep_cpc',7)/

Grib2 templates---- grib2_all_tables_module

contains

subroutine get_g2_subcenters(key,value,ierr)subroutine get_g2_sigreftime(key,value,ierr)subroutine get_g2_versionno(key,value,ierr)subroutine get_g2_loctabversno(key,value,ierr)….subroutine get_g2_fixedsurfacetypes

Grib2 templates---- grib2_all_tables_module

• And all the templates for each grb2 sectionsubroutine g2sec0(idisc,listsec0)

subroutine g2sec1(origin_key,subcen_key,vers_key,lvers_key, sigreftime_key,refyear_val, refmon_val,refday_val,refhour_val, refmin_val,refsec_val,prodstatus_key,typeofdata_key,listsec1)

subroutine g2sec4_temp0(icatg,iparm,typ_gen_proc_key, & gen_proc_or_mod_key,hrs_obs_cutoff,min_obs_cutoff, & unit_of_time_key,fcst_time,lvl_type1,scale_fac1, & scaled_val1,lvl_type2,scale_fac2,scaled_val2, & ipdstmpl0)

subroutine g2sec4_temp8(icatg,iparm,typ_gen_proc_key,gen_proc_or_mod_key, & hrs_obs_cutoff,min_obs_cutoff,unit_of_time_key, & fcst_time,lvl_type1,scale_fac1,scaled_val1,lvl_type2, & scale_fac2,scaled_val2,year_intvl, & mon_intvl,day_intvl,hour_intvl,min_intvl,sec_intvl, & num_time_range,stat_miss_val,type_of_stat_proc, & type_of_time_inc,stat_unit_time_key, & leng_time_range_stat,stat_unit_time_key_succ, & time_inc_betwn_succ_fld,ipdstmpl8)

subroutine g2sec5_temp0(dec_scale_fac,type_of_field,ifield5)

subroutine g2sec5_temp40(dec_scale_fac,bin_scale_fac,tlnumbits, & type_of_field, type_of_compression,ifield5,len,fld)

• A library is created from grib2_all_tables_module.f and process_4dot2.c, it is linked to post executable along with g2 lib.

•Grib2_module.f is created to utilize these templates.

Grib2 templates---- grib2_all_tables_module

Add grib2 option in post

• Grib2 option is passed into post from scripts

• All post fields are computed and saved in a data array, field information related to the post control file is save in fld_info

• Gribit2 is called to parallel generate grib2 messages and to parallel write out grib2 message

Parallel generating and writing grib2 message

Each processor holds partial domain of data for all the output data fields in post computation; the data is then redistributed to all processors through mpi_alltoallv so that each processor holds whole domain data, but only with certain number of data fields.

All processors generate grib2 messages simultaneously

All processors exchange the information of their total grib2 message length

All processors will write out simultaneously through mpi_write_all

MPI_ALLTOALLV

MPI_ALLTOALLV

gengrb2msg

gengrb2msg

gengrb2msg

MPI IO

MPI IO

MPI IO

Parallel generating and writing grib2 message

PE1 PE1 PE1

PE3

PE2 PE2 PE2

PE3 PE3

Grb2 file

Partial domain data with all the output fields

Whole domain data with certain number of output fields

Grib2 messages for certain fields

• contains subroutines to parallel generate grib2 messages using grib2 templates and to parallel write out grib2 file

Public num_pset,nrecout

type(paramset_t),save :: pset,

subroutine grib_info_init(first_grbtbl)

subroutine grib_info_finalize

subroutine gribit2(post_fname)

Private:

subroutine gengrb2msg(idisc,icatg, iparm,nprm,nlvl,ntrange,tinvstat,datafld1,cgrib,lengrib)

subroutine getgds(ldfgrd,len3,ifield3len,igds,ifield3) use gridspec_mod, only: DXVAL,DYVAL,CENLAT,CENLON,LATSTART, LONSTART, & LATLAST, LONLAST,MAPTYPE,STANDLON,latstartv,cenlatv,lonstartv, cenlonv, TRUELAT1, TRUELAT2

Grib2_module.f

POST Script

cat > itag <<EOFnmm_b_history_nemsio.0${fhr}binarynemsio${YY}-${MM}-${DD}_${HH}:00:00NMMEOF

cat > itag <<EOFnmm_b_history_nemsio.0${fhr}binarynemsiogrib2${YY}-${MM}-${DD}_${HH}:00:00NMMEOF

* POST main program will detect if there is output file format specified, if the variable is not “grib2” or “grib1”, it will set output format as “grib1”

• 2. Post control filecp namrr_cntrl.parm.03 nmmb_cntrl.parm

cp /climate/noscrub/wx20wa/grib2/xml/post/postcntrl_nam.xml postcntrl.xml

• 1. The output file format is passed from post script

Results

NMMB Disk space Wall time Total Memory

grib1 950MB 275 740MB

parallel grb2(JPEG)

296MB 31% 220 80% 1168MB 157%

GFS master files from NEMS GFS (1152*576, 664 fields)

B grid 8KM output from NEMS NMMB (954*835, 1098fields)

GFS Disk space Wall time Total Memory

grib1 605MB 72 0.85G

parallel grb2 (JPEG)

195MB 32% 60 83% 1.1G 129%