Embed Size (px)
DESCRIPTION
Easy access HDF files via Hyrax. Kent Yang The HDF Group. Outline. Why OPeNDAP ? Hyrax features Hyrax modules Demos Suggestions. Why OPeNDAP?. Check metadata remotely (in various forms) Obtain the subset of data easily and efficiently Hide the original data sources - PowerPoint PPT Presentation
Citation preview
www.hdfgroup.org
The HDF Group
1ESIP Summer Meeting
Easy access HDF files via Hyrax
Kent YangThe HDF Group
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 2
Outline
• Why OPeNDAP?• Hyrax features • Hyrax modules• Demos• Suggestions
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 3July 8 – 11, 2014
Why OPeNDAP?
• Check metadata remotely (in various forms)• Obtain the subset of data easily and efficiently• Hide the original data sources
• netCDF, HDF4, HDF5 or GeoTiFF, GRIB • Many popular earth science tools can visualize
and analyze the data via OPeNDAP• OPeNDAP output(including subsets) can be
downloaded as other formats
www.hdfgroup.orgESIP Summer Meeting 4July 8 – 11, 2014
Doesn’t helpGreat!
www.hdfgroup.orgESIP Summer Meeting 5July 8 – 11, 2014
• The visualization of HDF(5) data via Hyrax• What NASA users request the most
HDF(5) Hyrax modules
www.hdfgroup.orgESIP Summer Meeting 6
Helpful Hyrax feature – BES keys
• BES keys to dynamic change the program flow• Easy for users to choose what they want• Easy for developers to add/revise features
requested by users
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 7
Example on using BES keys
July 8 – 11, 2014
Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
Float CERES_Unfiltered_Radiances_all_toa_sw_rad[gmt_hour_index = 24][latitude =180][longitude= 360]
www.hdfgroup.orgESIP Summer Meeting 8
Hyrax modulation
• Core framework implemented in C++• Just need to provide DDS, DAS and Data to Hyrax
libdap and BES• A handler can be sophisticated • HDF handler examples
• Retrieve latitude and longitude for HDF-EOS grid• Interpolate the latitude and longitude for HDF-EOS
swath with dimension map• Make sure the dimensions, variable names and
attributes follow CF conventions
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 9July 8 – 11, 2014
Non-CF CF
Float CERES_Unfiltered_Radiances_all_toa_sw_rad[gmt_hour_index = 24][latitude =180][longitude= 360]
Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]Float /CERES_Unfiltered_Radiances/all_toa_sw_rad[24][180][360]
www.hdfgroup.orgESIP Summer Meeting 10
Demos – HDF-EOS2 (AIRS grid)
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 11
Demo 2 – HDF-EOS2 (MODIS swath)
July 8 – 11, 2014
Swath with Dimension Map
www.hdfgroup.orgESIP Summer Meeting 12
Demo 3 – HDF-EOS5 (OMI grid)
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 13
Hyrax modules that help the easy access
• Filenet CDF module• Can download DAP output as a netCDF-3 or netCDF-
4 classic files
• NcML module• Add/delete/revise metadata information of the data
source• Aggregate granules
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 14
Demo
July 8 – 11, 2014
• GPM level 1B(HDF5)
www.hdfgroup.orgESIP Summer Meeting 15
HDF(5) and file netCDF demo
• FilenetCDF module can work with the HDF handlers to download the HDF files or subset of the files as netCDF-3 or netCDF-4 classic files that follow the CF conventions
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 16
HDF(5) and NcML demo
• NcML module can be used with HDF handlers to provide the missing CF information
• AVHRR example
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 17
HDF(5), file netCDF and NcML demo
• HDF handlers, NcML module and file netCDF module can work together
• AVHRR example
July 8 – 11, 2014
www.hdfgroup.orgESIP Summer Meeting 18
Summary of modules to serve HDF files
July 8 – 11, 2014
Hyrax core
HDF(5) files
Main engineHDF(5) handlers
Supplement InfoNcML
DAP clients(IDV, Ferret, Panoply, IDL…...)
netCDFfiles
Alternative outputFile netCDF
www.hdfgroup.orgESIP Summer Meeting 19
Demo - Beyond the web service
• BES standalone program • Connect the HDF handlers, the file netCDF module
and the NcML module• End users can build Hyrax and use this program as a
command-line tool
• Can it be a standalone tool?
July 8 – 11, 2014
besstandalone -c /etc/bes/bes.conf -i in4-GPM-l1b > GPM-l1b.nc4
www.hdfgroup.orgESIP Summer Meeting 20
Future - Performance
• Improve the BES caching mechanism• Critical for large file access and aggregation service
July 8 – 11, 2014
www.hdfgroup.org
The HDF Group
21ESIP Summer Meeting
Earth Science GroupTed Habermann
Aleksandar JelenakH. Joe Lee
Joel PlutchakJohn Readey
Kent Yang
July 8 – 11, 2014
