Upload
jocelyn-cameron
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
1/14/2009 25th IIPS Conference 1
Challenges to Archive and Access NASA HDF-EOS Data in the long
Term
MuQun Yang (The HDF Group)
Choonghwan Lee (The HDF Group)
Ruth Duerr (NSIDC )
Prerequisite
• METS(Metadata Encoding & Transmission Standard)• Standard for encoding structural metadata
• ISO-19115• International Schema for describing geographic
information
• File-level Metadata• Metadata about the individual file or granule
• Dataset-level Metadata• Metadata that applies to each and every
granule/file in the whole data set(product)
1/14/2009 25th IIPS Conference 2
1/14/2009 25th IIPS Conference 3
HDF5 Archive Information Package
Data file HDF5
METS
Primary Schema Extension Schema
|<mets>|---<dmdSec>----------------<MODS>|---<amdSec>--------------|--<techMD>| |--<rightsMD>| |--<sourceMD>|----<fileGrp>|----<structMap>
http://www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF5_AIP_White_Paper.pdf
HDF5 AIP Components
Metadata file
1/14/2009 25th IIPS Conference 44
NOAA SDS Program
NetCDF4/HDF5-data
NetCDF4 / HDF5 Data
METS
NSIDC/ ECS
HDF4-data
NCDC:CLASS
ISO-19115
HDF5-AIP
H4toH5
ECS to ISO-19115
NSIDC/ECS
Metadata
CDM/NetCDF4
ECS to METS
Enhanced H4toH5 conversion tool
• Convert HDF-EOS2 data to NetCDF4-compliant HDF5 data
• Official release (2.0) can be found at http://hdfgroup.org/h4toh5/
1/14/2009 25th IIPS Conference 5
$ ./h4toh5 –eos –nc4 input.he2 output.nc4$ ./h4toh5 –eos –nc4 input.he2 output.nc4
Challenges to do the conversion
• Retrieve geo-location information from HDF-EOS2 data
• Conform to NetCDF4 data model in the existing H4toH5 conversion tool
• ……
1/14/2009 25th IIPS Conference 6
• Grid lacks geolocation fields• Use predefined projections
• Geographic• Sinusoidal• Polar stereographic• …
• New converter creates geolocation fields• HDF-EOS2 API GDij2ll()
1/14/2009 25th IIPS Conference 7
Challenges: Handle EOS - Grid
Data [4][12]Lon[12]Data [4][8]Lon[4][8]
Geographic
Sinusoidal
• The size of geolocation fields can be different from data fields
• New converter has to handle geolocation fields correctly
1/14/2009 25th IIPS Conference 8
Challenges: Handle EOS - Swath
• Follow CF conventions• Create two variables: NewLongitude and
NewLatitude• Add to the data field an attribute coordinates=“NewLongitude NewLatitude”
• Keep the original Latitude and Longitude
1/14/2009 25th IIPS Conference 9
Challenges in conforming to NetCDF4
Data field has three columnsLongitude field has two columnsNew longitude has three columns
1/14/2009 25th IIPS Conference 10
Now some examples to show NetCDF4 files converted from EOS2
1/14/2009 25th IIPS Conference 11
A netCDF-4 file converted from EOS2 data at NSIDC
1/14/2009 25th IIPS Conference 12
A netCDF-4 file converted from EOS2 data at NSIDC
Deliverables and future work
• Deliverables1. Enhanced HDF4 to HDF5 conversion tool
http://hdfgroup.org/h4toh5/2. A validation tool to verify the correctness of the
conversion
Will be released soon!
1/14/2009 25th IIPS Conference 13
1/14/2009 25th IIPS Conference 14
Acknowledgement
This work was supported under NOAA Scientific Stewardship Program grant number NA07OAR4310286. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NOAA.