04/12/23 HDF and HDF-EOS Workshop XI 1

Moving applications to HDF5 1.8.0

04/12/23 HDF and HDF-EOS Workshop XI 2

Outline

• Quick overview of new features• Move or not to move?• HDF5 APIs versioning

HDF5 library configuration HDF5 application configuration

• Examples

04/12/23 HDF and HDF-EOS Workshop XI 3

Goal of the tutorial

• Raise awareness about forward/backward compatibility issues with the new release

• Help with transition to the new release• Get feedback from you before official

release is done

04/12/23 HDF and HDF-EOS Workshop XI 4

Current status of the HDF5 1.8.0 release

• HDF5 1.8.0-beta4 was released on October 22, 2007

• Stable file format and APIs• Has been successfully tested with NetCDF-4 and

HDF-EOS5 on many supported platforms• Tasks left before the full release

Bugs fixing Incorporating contributed features Performance tuning (more will follow) Documentation Thorough backward/forward compatibility testing

for APIs and files format

04/12/23 HDF and HDF-EOS Workshop XI 5

Quick overview of new features in HDF5 1.8.0

and their affect on current applications

04/12/23 HDF and HDF-EOS Workshop XI 6

HDF5 1.8.0 themes

• Better handling of HDF5 metadata• Reduced HDF5 file overhead• Efficient metadata caching• Improved internal structures for better

handling of variable-length data • Features to support HDF and NetCDF

harmonization• New and improved tools • Numerous bug fixes and improvements• Result: More than 200 new APIs and

revised file format

04/12/23 HDF and HDF-EOS Workshop XI 7

Quick overview of the new features

• Complete list can be found at http://www.hdfgroup.uiuc.edu/HDF5/doc_1.8pre/WhatsNew180.html

• Two types of features• Features that did not require file format and

APIs changes• Features that did require file format

changes

04/12/23 HDF and HDF-EOS Workshop XI 8

Quick overview of the new features

• Features that did not require file format changes and changes to existing APIs• Improved metadata caching• New filters (scale+offset, n-bit, arithmetic

transformation)• Improved error handling mechanism• Support for dimension scales, packet tables and

other high-level APIs• Affect on applications:

Applications can be extended (calls added) to use new features

There is no need to modify existing callsCreated files are compatible with the earlier

versions of software

04/12/23 HDF and HDF-EOS Workshop XI 9

Quick overview of the new features

• Features that did require file format changes and changes to existing APIs• Creation order of the objects (groups, datasets,

attributes)• Support for UTF-8 encoding for strings that describe

links to the objects and names of the attributes• Configurable compact or indexed storage for groups• Shared object headers, external links• New file format features to reduce files overhead• Affect on applications:

Applications have to be modified to use new featuresCreated files are NOT compatible with the earlier

versions of software

04/12/23 HDF and HDF-EOS Workshop XI 10

To move or not to move?

04/12/23 HDF and HDF-EOS Workshop XI 11

Why applications should move to HDF5 1.8.0?

• My application doesn’t need any new features, why should I consider moving to HDF5 1.8.0?• Take advantage of improved library

Bug fixes (e.g. corrupted files)Better metadata cache performanceSmaller memory footprintSupport for big attributes, new compression filters,

datatype conversion, etc.Availability of new platforms like Windows 64-bitBetter support for different Fortran compilersImproved toolsFirst step in taking advantage of the new features in

the future

04/12/23 HDF and HDF-EOS Workshop XI 12

Why applications may not move to HDF5 1.8.0?

• May be expensive• Moving to the new release may require

Source code modificationBuilding scripts modification

• No new features are desired

04/12/23 HDF and HDF-EOS Workshop XI 13

Making decision….

• I would like to use new features, how much work it would be?

• I would like to use new library but I cannot modify my application, can I do it?

The rest of the talk is about the ways to move applications to the new release without doing much work

04/12/23 HDF and HDF-EOS Workshop XI 14

Introduction to HDF5 APIs versioning

04/12/23 HDF and HDF-EOS Workshop XI 15

Example: Four ways to create a group

• Old way

H5Gcreate ( loc_id, “New/My old group”, 0 )

H5Gcreate1( loc_id, “New/My old group”, 0 ) No new features can be invoked

• New wayH5Gcreate ( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id)

H5Gcreate2( loc_id, “New/My new group”, lcpl_id, gcpl_id, gapl_id) New features can be invoked

Creation order Unicode names Compact storage Intermediate group creation

04/12/23 HDF and HDF-EOS Workshop XI 16

New APIs in HDF5 1.8.0

• Examples of the new APIs H5Acreate1, H5Aopen1 H5Acreate2, H5Aopen2 H5Gcreate1, H5Gopen1 H5Gcreate2, H5Gopen2, H5G_link_hard H5Rget_obj_type1 H5Rget_obj_type2

• New APIs have more parameters to set up creation and access properties for the objects

• Default values for new parameters H5P_DEFAULT will emulate old behaviorH5Gcreate ( loc_id, “New/My old group”, 0 )H5Gcreate ( loc_id, “New/My new group”, H5P_DEFAULT,H5P_DEFAULT, H5P_DEFAULT)

04/12/23 HDF and HDF-EOS Workshop XI 17

Removed APIs in HDF5 1.8.0

• Examples of the removed APIs H5Acreate, H5Aopen H5Gcreate, H5Gopen H5Rget_obj_type

• Still can be called by using API mapping and macrosH5Gcreate ( loc_id, “New/My old group”, 0 )

04/12/23 HDF and HDF-EOS Workshop XI 18

Deprecated APIs in HDF5 1.8.0

• Include both public and private APIs• Functionality is replaced by new functions• See files in the src directory

H5Adeprec.c H5Ddeprec.c H5Edeprec.c H5Gdeprec.c H5Rdeprec.c H5Tdeprec.c

• Examples of deprecated APIs H5Acreate1, H5Aopen_name H5Gcreate1, H5Gopen1, H5G_link_hard H5Rget_obj_type1

• For complete list see Changes.html file, “API Compatibility Macros in HDF5”

• Deprecated APIs will be removed with caution (in some cases will not be removed at all)

04/12/23 HDF and HDF-EOS Workshop XI 19

HDF5 Macros for API compatibility

Name: H5GcreateSignature: hid_t H5Gcreate( hid_t loc_id, const char *name,

size_t size_hint ) hid_t H5Gcreate( hid_t loc_id, const char *name,

hid_t lcpl_id, hid_t gcpl_id, hid_t gapl_id Purpose: Creates a new empty group and links it to a

location in the file.Description: H5Gcreate is a macro that is mapped to either

H5Gcreate1 or

H5Gcreate2, depending on the needs of the application.

04/12/23 HDF and HDF-EOS Workshop XI 20

HDF5 Macros for API compatibility

• Defined in the H5version.h fileExample of the H5Gcreate macro#ifdef H5_USE_16_API…#if !defined(H5Gcreate_vers)#define H5Gcreate_vers 1#endif /* !defined(H5Gcreate_vers) */…#if !defined(H5Gcreate_vers) || H5Gcreate_vers == 2#ifndef H5Gcreate_vers#define H5Gcreate_vers 2#endif /* H5Gcreate_vers */#define H5Gcreate H5Gcreate2#elif H5Gcreate_vers == 1#define H5Gcreate H5Gcreate1#else /* H5Gcreate_vers */#error "H5Gcreate_vers set to invalid value"#endif /* H5Gcreate_vers */

…

04/12/23 HDF and HDF-EOS Workshop XI 21

HDF5 Library configuration

• Mapping of APIs is set at configuration time

• Several configuration flags are provided to enable different mappings• Macro name is mapped to old API, e.g.

H5Gcreate is mapped to H5Gcreate1• Macro name is mapped to new API, e.g.

H5Gcreate is mapped to H5Gcreate2 (will have different parameters from 1.6 H5Gcreate)

• Disable old APIs completely, e.g. H5Gcreate1 is not available in the library

04/12/23 HDF and HDF-EOS Workshop XI 22

HDF5 Library configurationConfigure flag(global settings)

Public symbols are mapped to

Are deprecated symbols available?

(default)--with-default-api-version=v18

1.8(e.g. H5Gcreate is mapped to H5Gcreate2, old H5Gcreate is H5Greate1)

Yes

--disable-deprecated-symbols

1.8(e.g. H5Gcreate is mapped to H5Gcreate2, H5Gcreate1 is not available)

No

--with-default-api-version=v16

1.6(e.g. H5Gcreate is mapped to H5Gcreate1, H5Gcreate2 is available)

Yes

04/12/23 HDF and HDF-EOS Workshop XI 23

HDF5 Library configuration

• Library configuration information is available in the libhdf5.settings file

• Examples…….Default Version of Public Symbols: v18With Deprecated Public Symbols: Yes…….Default Version of Public Symbols: v16With Deprecated Public Symbols: Yes…….Default Version of Public Symbols: v18With Deprecated Public Symbols: No

04/12/23 HDF and HDF-EOS Workshop XI 24

HDF5 Application configuration options

Global mapping• HDF5 APIs mapping can be done by application• Assuming both deprecated and new symbols are

available in the library:h5cc my_program.c

Both H5Gcreate1, H5Gcreate2 and H5Gcreate may be used

h5cc -DH5_NO_DEPRECATED_SYMBOLS my_program.cOnly new symbols are available for application;

H5Gcreate is mapped to H5Gcreate2; application may use both H5Gcreate2 and H5Gcreate; cannot use H5Gcreate1

h5cc -DH5_USE_16_API my_program.cH5Gcreate is mapped to H5Gcreate1; all three

H5Gcreate1, H5Gcreate2 and H5Gcreate can be used

04/12/23 HDF and HDF-EOS Workshop XI 25

HDF5 Application configuration options

Per-function scale• Version and mapping can be set per function• Assuming both deprecated and new symbols are

available in the library:h5cc -D H5Gcreate_vers=1 -D H5Acreate_vers=2

my_program.c• Maps H5Gcreate to H5Gcreate1• Maps H5Acreate to H5Acreate2• both H5Gcreate1 and H5Gcreate2 may be used; the

same for H5Acreate1 and H5Acreate2

h5cc -D H5Gcreate_vers=2 my_program.c• Maps H5Gcreate to H5Gcreate2• Both H5Gcreate1 and H5Gcreate2 may be used

04/12/23 HDF and HDF-EOS Workshop XI 26

HDF5 Fortran and C++ applications

• HDF5 Fortran and C++ libraries use new APIs H5<name>2 using default parameters for new arguments

• Backward/forward compatible• New features are not available yet

04/12/23 HDF and HDF-EOS Workshop XI 27

Example: --with-default-api-version=v18

hid_t file_id, group_id; /* identifiers */ .../* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT);

/* Create several groups in a file */ grp1_id = H5Gcreate (file_id, ”New/larry", H5P_DEAFULT, gcpt, gapt); grp2_id = H5Gcreate1(file_id,"/larry",0);… grp3_id = H5Gcreate2(file_id,”New/larry", H5P_DEAFULT, gcpt, gapt);

04/12/23 HDF and HDF-EOS Workshop XI 28

Example: --with-default-api-version=v16

hid_t file_id, group_id; /* identifiers */ .../* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT);

/* Create several groups in a file */ grp1_id = H5Gcreate (file_id, "/larry",0);grp2_id = H5Gcreate1(file_id,"/larry1",0); grp3_id = H5Gcreate2(file_id,”New/larry", H5P_DEAFULT, gcpt, gapt);

04/12/23 HDF and HDF-EOS Workshop XI 29

Example: --disable-deprecated-symbols

hid_t file_id, group_id; /* identifiers */ .../* Open “file.h5” */ file_id = H5Fopen(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, H5P_DEFAULT);

/* Create several groups in a file */ grp1_id = H5Gcreate (file_id, ”New/larry", H5P_DEAFULT, gcpt, gapt);/* Compilation will fail */ grp2_id = H5Gcreate1(file_id,"/larry",0); grp3_id = H5Gcreate2(file_id,”New/larry", H5P_DEAFULT, gcpt, gapt);

04/12/23 HDF and HDF-EOS Workshop XI 30

Example: new version of the file format

hid_t file_id, group_id; /* identifiers */ .../* Open “file.h5” and create objects using new format*/

fapl = H5Pcreate(H5P_FILE_ACCESS);ret = H5Pset_latest_format(fapl, 1);fid = H5Fcreate(“file.h5”, H5F_ACC_RDWR, H5P_DEFAULT, fapl);

/* Compose group creation property list */

gcpl = H5Pcreate(H5P_GROUP_CREATE);

ret = H5Pset_link_creation_order(gcpl, (H5P_CRT_ORDER_TRACKED | H5P_CRT_ORDER_INDEXED));

/* Create a group */

gid = H5Gcreate(fid, GNAME, H5P_DEFAULT, gcpl, H5P_DEFAULT);

04/12/23 HDF and HDF-EOS Workshop XI 31

Example: h5dump --group=1 tordergr.h5

HDF5 "tordergr.h5" {GROUP "1" { GROUP "a" { GROUP "a1" { } GROUP "a2" { GROUP "a21" { } GROUP "a22" { } } } GROUP "b" { } GROUP "c" { }}}

04/12/23 HDF and HDF-EOS Workshop XI 32

Example: h5dump … --sort_by=creation_order … tordergr.h5

HDF5 "tordergr.h5" {GROUP "1" { GROUP "c" { } GROUP "b" { } GROUP "a" { GROUP "a1" { } GROUP "a2" { GROUP "a22" { } GROUP "a21" { } } }}}

04/12/23 HDF and HDF-EOS Workshop XI 33

Example: creation of intermediate groups

/* Create link creation property list */ lcr_id = H5Pcreate(H5P_LINK_CREATE);

/* Set flag for intermediate group creation */

status = H5Pset_create_intermediate_group(lcr_id,

TRUE);

grp_id = H5Gcreate (file_id, "/A/B/C", lcr_id,

H5P_DEFAULT, access_id);

Groups A and B will be created automatically

04/12/23 HDF and HDF-EOS Workshop XI 34

Thank you!

Questions ?