Embed Size (px)
DESCRIPTION
A number of powerful NetCDF-based tools exist for visualizing and analyzing remote sensing data, such as IDV, McIDAS-V, Panoply, Ferret and GrADS. As a result, the GES DISC has been deploying a number of different strategies to make its native HDF, HDF-EOS2 and HDF-EOS5 data available in usable forms to these NetCDF tools. The most elaborate of these is an online, on-the-fly Web Service to convert HDF and HDF-EOS2 data into NetCDF-CF1. The end result is highly effective in making the data usable, even Level 2 swath data, which can be visualized in IDV. Unfortunately, the code is relatively convoluted and inflexible due to the difficulty of locating values in the HDF input file to describe the vertical dimension in CF1. (This could be made easier if HDF dimscales could be written through the HDF-EOS2 interface.) A similar approach has been to serve certain datasets through a Web Coverage Server developed by George Mason University, which is subject to many of the same issues. A third approach has been to use OPeNDAP to make data available to NetCDF-based tools. The HDF5 handler has proven to be effective for Level 3 gridded data in this respect at very low cost. These tools can access remote data directly through OPeNDAP, or the HDF5 data can be downloaded as a local NetCDF file. However, this is limited to HDF5 and does not work for Level 2 swath data. Recently, we have begun deploying a fourth approach, the Lats4D script running within GrADS. Though slower than our custom C++ converter, it supports (in principle) conversion of any Level 3 data that can be read in through GrADS, allowing for dimension definition through GrADS control files. Experimenting with Lats4D also led to the serendipitous discovery that NetCDF-based tools based on the netcdf-java library (e.g., IDV and Panoply) can actually read certain HDF files. If this capability could be generalized, the expense of netCDF conversion could be eliminated. Alas, the hierarchies written by HDF-EOS block this capability; we welcome ideas and contributions from the HDF and NetCDF communities to devising a solution to this impasse.
Citation preview
HDF Workshop 5 Nov 2009
PRACTICAL METHODS FOR MAKING HDF DATA USABLE IN
NETCDF-BASED TOOLS
Chris Lynnes and Fan Fang
Goddard Earth Sciences Data and Information Services Center
1Thursday, November 5, 2009
HDF Workshop 5 Nov 2009 2
IDVPanoply
Ferret GrADS
2Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
"And then a miracle occurs..." Unidata IDV
33Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
IDV with HDF-EOS5 “self-describing format”
4
4Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
IDV with HDF-EOS5 “self-describing format”
4
4Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Panoply w/HDF5
5
5Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Ferret and GrADS with HDF-EOS5
6
*Can be opened if user defines dimensions and uses xdfopen
6Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Solution #1: Convert to netCDF
• On-the-fly conversion to netCDF-3• Custom C++ code• Acquiring the URL executes the conversion and
downloads the file
7
7Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Panoply w/converted AIRS L3
8
8Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
IDV w/Converted AIRS L2
9
Surface Air Temperature
9Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
But...
• Custom C++ code is expensive to maintain• Z dimension scale values are hard to find in AIRS
data– vdatas w/non-standard names– external documents– therefore: hard-coded in C++ class– if only we could attach standard Z dimscales to the SDS
• New class for each dataset
10
10Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Solution #2: Lats4D Conversion• Essentially a GrADS script from Arlindo DaSilva, NASA/GSFC/
GMAO• Same on-the-fly mechanism as NetCDF converter• If you can read it into GrADS, you can convert it (usually)
– Dimensions can be defined in GrADS control file– Limited to L3 and L4 gridded products– Handles both HDF-EOS and HDF
• Also supports subsetting
11
11Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
TRMM Monthly (May 2009) in PanoplyLats4D Conversion + Subset
12
12Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Solution #3: OGC Web Coverage Service
• OGC WCS with netCDF/CF-1 profile– Supports subsetting– But: “prefers” interpolation to user’s resolution– Requests are difficult for users to submit– Works better for machine-to-machine interfaces
• DataFed• DLR
– Using GMU-developed WCS Server
13
13Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
AIRS L3 via WCS in Panoply
14
wget -O airs_wcs.nc 'http://acdisc.sci.gsfc.nasa.gov/daac-bin/wcsAIRSL3?VERSION=1.0.0&SERVICE=WCS&REQUEST=getCoverage&FORMAT=NetCDF&COVERAGE=AIRX3STD:ascending:TotCldLiqH2O_A&BBOX=-179.5,-89.5,179.5,89.5&TIME=2007-04-07/2007-04-07/P1D&CRS=CRS:84&RESX=1.0&RESY=1.0'
14Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Solution #4: OPeNDAP• HDF handlers
– Released HDF5 version supports L3 grid– New HDF4 and HDF5 version supports
• HDF4• L2 swath
15
15Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
OPeNDAP Usability Results
16
IDV Panoply Ferret GrADS CommentsAIRS L3
MODIS L3OMI L3
TOMS L3AIRS L2
HIRDLS L2MLS L2
MODIS L2OMI L2
✓ ✓ ✓ ✓ exc. AIRX3C2M✓ ✓ ✓ latest version only✓ ✓ ✓ ✓✓ ✓ ✓ ✓✓ ✓ ✓ x
xx
✓ ✓ ✓ x so far: MOD04, MOD07✓ ✓ x exc. OMIBRO, OMHCO,
OMOCLO, OMUVB
16Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
OMI L3 NearUV AAOD via OPeNDAP in Panoply
17
17Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
(but watch out for fast rendering in IDV)
18
18Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
IDV w/fast rendering off
19
19Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Panoply w/MODIS L2 via new OPeNDAP HDF4 handler
20
20Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Panoply w/AIRS L2 via new OPeNDAP HDF4 Handler
• Cool..but who projected the data and how?21
21Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Solution #5: GrADS Data Server• DAP protocol, but data read by GrADS
– Sort of like an OPeNDAP with a GrADS handler– If you can read it in in GrADS...
• Includes time aggregation– (Not always a good thing)
22
22Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
MERRA via GDS in Panoply
23
23Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Solution #6?: netcdf libraryMERRA HDF Subset from Lats4D in Panoply
24
24Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
netcdf libraries
25
• netcdf-java library can read some HDF files– But not HDF-EOS...
• Tried adding lat/long dimscales to an HDF-EOS2 grid, to no avail
– Except for an MLS L3 HDF-EOS5• (not a standard product)
25Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Panoply, using netcdf-java w/ MLS L3, HDF-EOS5
26
26Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Can we make netcdf libraries read HDF-EOS?
• Might require joint effort between UCAR, THG and tool developers to develop and debug...
• OR, solution might emerge from HDF-EOS5 / netCDF-4 Augmentation effort
• ...BUT– could make a lot of conversions unnecessary– would make data usage much easier for users
27
27Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Trade Space
28
Pros ConsCustom C++ Conversion
Lats4D Conversion
WCS
OPeNDAP
GrADS Data Server
netcdf library
L2 swath + L3/L4 gridFast
CostlyData size expansion
Easy to addHDF + HDF-EOS
L3/L4 grid onlySlow
Subsetting InterpolationCostly
L2 swath + L3/L4 grid EfficientInexpensive
Reprojects L2 swath
HDF + HDF-EOS L3/L4 onlySlow
Enables many toolsObsoletes conversion
HDF-EOS
28Thursday, November 5, 2009
HDF Workshop 5 Nov 2009
Future of HDF-netCDF Tool Interoperability: Key Challenges
• Level 1b data (calibrated radiance)– How should we fit the channel/wavelength dimension
into CF1?• Level 2 in native swath coordinates• netCDF library direct access to HDF-EOS• Non-self-describing implementations of HDF
– e.g., NPP
29
29Thursday, November 5, 2009
![NetCDF-4 Performance Report - The HDF Group · NetCDF-4 Performance Report Choonghwan Lee ... Introduction NetCDF-4 [1] ... netcdf4 large netcdf3 large netcdf4 small](https://img.pdfslide.net/doc/110x75/5b509b227f8b9a5a6f8ed256/netcdf-4-performance-report-the-hdf-group-netcdf-4-performance-report-choonghwan.jpg)
