Embed Size (px)
Citation preview
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XIV 1
Easy Access of NASA HDF data via OPeNDAP
Kent Yang and Joe LeeThe HDF Group
September 28,2010
www.hdfgroup.org
Easy
HDF/HDF-EOS Workshop XIV 2September 28,2010
www.hdfgroup.org
Outline
• Introduction to OPeNDAP
• HDF4/5-OPeNDAP Handlers
• Live Demo
HDF/HDF-EOS Workshop XIV 3September 28,2010
www.hdfgroup.org
OPeNDAP
• A software framework that allows simple access to remote data
• Client-server model• Servers make the local data accessible remotely• Clients make applications easily access to remote
served data• Widely used by Earth Science Community• URL: http://www.opendap.org/
September 28,2010 HDF/HDF-EOS Workshop XIV 4
www.hdfgroup.org
Access HDF via FTP
HDF/HDF-EOS Workshop XIV 5
Users
FTP Clients
IDL
FTP Servers
HDF Files
AIRS/OMI
FTP
Local HDF Files
Aqua/Aura
September 28,2010
www.hdfgroup.org
Client Library(libnc_dap)
Data Access Protocol (DAP) via Internet
Server(Hyrax)
Handler(hdf4/5_handler)
Remote Data(HDF4/5)
View Data (User)
OPeNDAP Concept
6
Visualization Tools
(gradsdap)
Typical Remote Access
FTP/HTTP
Visualization tools
(e.g., grads)
HDF/HDF-EOS Workshop XIVSeptember 28,2010
www.hdfgroup.org
Access HDF via OPeNDAP
HDF/HDF-EOS Workshop XIV 7
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF Files
Hyrax AIRS/OMI
DAP
libnc-dap
Visualization Tools
HDF HANDLER
Aqua/Aura
September 28,2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 8
OPeNDAP Data Types
• OPeNDAP provides three basic data objects: - data descriptor structure (.dds)
syntactic metadata – rigid
- data attribute structure (.das)
semantic metadata – flexible
- data (.dods) - the actual data in a binary structure
• Data are requested with a URL
September 28,2010
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 9
OPeNDAP Servers
September 28,2010
Data Data Data Data Data Data Data
Matlab
HDF4/5 JDBC
FreeFromFITS
CDF CEDAR
Data
netCDF
netCDF HDF4/5 Matlab
Data
DSP
DSP
Data
JGOFS
Tables SQL FITS CDFFlat
Binary CEDAR
Data
CODAR
Data
ESML
GeneralCODAR
www.hdfgroup.orgHDF/HDF-EOS Workshop XIV 10
OPeNDAP Clients
netCDF C netCDF Java
IDVFerret GrADS VisAD ncBrowse Matlab ExcelIDL Access
MatlabClient
IDLClient
September 28,2010
www.hdfgroup.org
Outline
• Introduction to OPeNDAP
• HDF4/5-OPeNDAP Handlers
• Live Demo
HDF/HDF-EOS Workshop XIV 11September 28,2010
www.hdfgroup.org
Enhanced HDF4 Handler
HDF/HDF-EOS Workshop XIV 12
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF4 Files
Hyrax
DAP
libnc-dap
Aura
MODIS Swathw/ Dim. Map
Enhanced HDF4 HANDLER
Visualization Tools
September 28,2010
www.hdfgroup.org
An example: Access HDF data via OPeNDAP
HDF/HDF-EOS Workshop XIV 13
Swath with Dimension Map
September 28,2010
www.hdfgroup.org
HDF-EOS2 Products Supported
• GES DISC• AIRS / MERRA / TOMS
• LAADS / LP DAAC / NSIDC• Many MODIS products
• NSIDC• AMSR_E / NISE
• LaRC• MISR / MOPITT / some CERES(TRMM)
HDF/HDF-EOS Workshop XIV 14September 28,2010
www.hdfgroup.org
Pure HDF4 Products Supported
• GES DISC• TRMM L1B, L2B Swath • TRMM L3 Grid 42B & 43B products
• OBPG (Ocean Color)• SeaWiFS / MODIST / MODISA• CZCS / OCTS
HDF/HDF-EOS Workshop XIV 15September 28,2010
www.hdfgroup.org
Pure HDF4 Products Supported
• CERES• CER_AVG• CER_ES4• CER_ISCCP-D2like-Day• CER_ISCCP-D2like-GEO• CER_SRBAVG• CER_SYN• CER_ZAVG
HDF/HDF-EOS Workshop XIV 16September 28,2010
www.hdfgroup.org
Limitations
• Reasons for Visualization Failures
• IDV / Panoply / Java Heap Memory
• Network / DAP (e.g., max HTTP header size)
• Can’t retrieve key information correctly (e.g.,
LAMAZ projection) from data
• Unable to harmonize with CF-conventions (e.g.,
FillValue inside latitude and longitude)
HDF/HDF-EOS Workshop XIV 17September 28,2010
www.hdfgroup.org
Limitations(continued)
• HDF-EOS2 + additional HDF4 objects• Additional HDF4 objects will be ignored.
• The rest of pure HDF4 products not listed in previous slides are not tested and may not be visualized by Java OPeNDAP Clients.
HDF/HDF-EOS Workshop XIV 18September 28,2010
www.hdfgroup.org
HDF-EOS5 products Supported
• OMI Swath, Grid• MLS/HIRDLS Swath• TES Grid
September 28,2010 HDF/HDF-EOS Workshop XIV 19
www.hdfgroup.org
Learning Curve of accessing HDF data
September 28,2010 HDF/HDF-EOS Workshop XIV 20
From the ESIP wiki page:http://wiki.esipfed.org/index.php/Making_Science_Data_Easier_to_Use_with_OPeNDAPMaking Science Data Easier to Use with OPeNDAP
NCL
www.hdfgroup.org
Live Demo
HDF/HDF-EOS Workshop XIV 21September 28,2010
www.hdfgroup.org
OPeNDAP HDF4/HDF5 handlers
HDF/HDF-EOS Workshop XIV 22September 28,2010
• Source codes and RPMs can be found under
http://opendap.org• Demo HDF5/HDF4 servers
http://hdfeos.org/zoo/hdf5_handler/
http://hdfeos.org/zoo/hdf5_handler/• Currently NASA GES DISC has installed both
OPeNDAP HDF4 and HDF5 handlers
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XIV 23
Thank you !
September 28,2010
www.hdfgroup.org
Acknowledgements
This work was supported by cooperative agreement number NNX08AO77A from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or recommendations expressed in this material are
those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space
Administration.
HDF/HDF-EOS Workshop XIV 24September 28,2010
www.hdfgroup.org
The HDF Group
Questions/comments?
HDF/HDF-EOS Workshop XIV 25September 28,2010
www.hdfgroup.org
Our Work
• Let Visualization Clients Access HDF-EOS2 and HDF4 files
HDF/HDF-EOS Workshop XIV 26September 28,2010
www.hdfgroup.org
HDF5-OPeNDAP Handler Update
• Support Aura HDF-EOS5 OMI Swath
• Access some files that follow HDF5
Dimension Scale Specification
• Support Aura HDF-EOS5 MLS
HDF/HDF-EOS Workshop XIV 27September 28,2010
