Upload
unity
View
28
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Improving the usability of HDF-EOS2 data. Kent Yang, Joe Lee, Choonghwan Lee The HDF Group March 31 st , 2009. EOSDIS Evolution 2015 Vision Tenets. Feb 3, 2005. Purpose of the talk. Identify problems to access HDF-EOS2 data Share our plan to help improve the EOS2 usability. - PowerPoint PPT Presentation
Citation preview
Improving the usability of HDF-EOS2 data
Kent Yang, Joe Lee, Choonghwan Lee
The HDF Group
March 31st, 2009
04/21/23 Annual briefing to ESDIS 1
04/21/23 Annual briefing to ESDIS 2
EOSDIS Evolution 2015 Vision TenetsVision Tenet Vision 2015 Goals
Archive Management NASA will ensure safe stewardship of the data through its lifetime. The EOS archive holdings are regularly peer reviewed for scientific merit.
EOS Data Interoperability
Multiple data and metadata streams can be seamlessly combined. Research and value added communities use EOS data interoperably with other
relevant data and systems. Processing and data are mobile.
Future Data Access and Processing
Data access latency is no longer an impediment. Physical location of data storage is irrelevant. Finding data is based on common search engines. Services invoked by machine-machine interfaces. Custom processing provides only the data needed, the way needed. Open interfaces and best practice standard protocols universally employed.
Data Pedigree Mechanisms to collect and preserve the pedigree of derived data products are readily
available.
Cost Control Data systems evolve into components that allow a fine-grained control over cost
drivers.
User Community Support
Expert knowledge is readily accessible to enable researchers to understand and use the data.
Community feedback directly to those responsible for a given system element.
IT Currency Access to all EOS data through services at least as rich as any contemporary science
information system.
Feb 3, 2005
Purpose of the talk
• Identify problems to access HDF-EOS2 data• Share our plan to help improve the EOS2
usability
04/21/23 3Annual briefing to ESDIS
The success of EOS mission
04/21/23 Annual briefing to ESDIS 4
MODIS HDF-EOS data to detect sand storm from Gobi chinaCourtesy NASA Earth Observatory
The success of EOS mission
04/21/23 Annual briefing to ESDIS 5
MODIS HDF-EOS data to detect fires in AfricaCourtesy NASA
Identify problems : technical part
04/21/23 Annual briefing to ESDIS 6
Beauty of HDF-EOS2
• Very close to physical model• Convenient to handle geo-location projections• Files are generally compact
No needs to store latitude and longitude for grid Reduce the size of latitude and longitude for
swath
04/21/23 7Annual briefing to ESDIS
04/21/23 Annual briefing to ESDIS 8
A compact HDF-EOS2 Grid
• MODIS (MYD10A2) Grid file• Two 2400 X 2400 grid data fields with gzip
compression• File size: 75 KB• If lat/lon are kept in the file
• File size may become 44 MB,
500 times bigger than the original file size
October 15-18, 2008 8Workshop Name (edit footer to change)
A compact HDF-EOS2 swath
• Small file size – Swath
04/21/23 9
Annual briefing
to ESDIS
cross
alon
gcross
alon
g
coarse_cross
coar
se_a
long
with dimension maps
CloudCover(along, cross)Latitude(along, cross)Longitude(along, cross)
CloudCover(along, cross)Latitude(along, cross)Longitude(along, cross)
CloudCover(along, cross)Latitude(coarse_along, coarse_cross)Longitude(coarse_along, coarse_cross)
CloudCover(along, cross)Latitude(coarse_along, coarse_cross)Longitude(coarse_along, coarse_cross)
with dimension maps
w/o dimension maps
along 406 4060
cross 271 2708
Longitude 0.5 MB(406 X 271 X 4)
42 MB(4060 X 2708 X 4)
There are issues to access some HDF-EOS2 files
Aims to improve the usability
04/21/23 Annual briefing to ESDIS 10
04/21/23 Annual briefing to ESDIS 11
First of all, there should be more communications among providers, tool developers, users and data storage software developers.
Focus on HDF-EOS2 and HDF4 in the next few slides.
Schematic of Grid
04/21/23 Annual briefing to ESDIS 12
YD
imXDim
StructMetadata.0 =“… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0)… Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS:Float32 Ozone(YDim, XDim)
StructMetadata.0 =“… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0)… Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS:Float32 Ozone(YDim, XDim)
Issue 1: lack of strict checking in HDF-EOS2 library
04/21/23 Annual briefing to ESDIS 13
Lack of strict checking in HDF-EOS2 library
• XDim and YDim exist and have values .e.g, OMI (OMUVBL3) Generally, XDim and YDim do not exist because
they are implied. XDim and YDim do not need to have values
because they can be calculated. Values are just redundant.
04/21/23 14
Annual briefing
to ESDIS
• Inconsistency in the number of elements HDF-EOS2 informs and HDF-EOS2 actually reads e.g., MODIS (MAC021S0, MAC02QS0,
MAC02QS1, MAC021S1) We believe that this data field had four elements
once and it was appended later.
Lack of strict checking in HDF-EOS2 library
04/21/23 15
Annual briefing
to ESDIS
StructMetadata … Band_1KM_RefSB[4] …
StructMetadata … Band_1KM_RefSB[4] …
Data FieldBand_1KM_RefSB
4 elements
Data FieldBand_1KM_RefSB
4 elements
Data FieldBand_1KM_RefSB
15 elements
Data FieldBand_1KM_RefSB
15 elementsconsistent
INCONSISTENT!INCONSISTENT!
Lack of strict checking in HDF-EOS2 library
• Wrong grid parameters e.g., MISR (MI3DAER) Projection is defined by code, projection
parameters, etc. For Lambert Azimuth projection, the fifth
parameter specifies “longitude of center of projection”.
Its value is entered in DDDMMMSSS.SS format. -92970000.00
04/21/23 16Annual briefing to ESDIS
degrees -92degrees -92 seconds 0seconds 0minutes 970?minutes 970?
Issue 2: lack of standard way to handle some geolocation
fields
04/21/23 Annual briefing to ESDIS 17
Schematic of Swath
04/21/23 Annual briefing to ESDIS 18
cross
alon
g
StructMetadata.0 =“ … DimensionName=”along”
size= 11DimensionName=”cross” size= 3 …” Data fields: Float32 CloudCover(along,cross)Geolocation fields: Float32 lat(along,cross) Float32 lon(along,cross)
No standard way to handle geolocation fields
• No standard way to figure out the vertical coordinate in an HDF-EOS2 file A data field
• TAirStd[GeoTrack][GeoXTrack] StdPressureLev is a dimension Additional data field supplies values, probably
• pressStd How can a user know for sure that the field
pressStd provides the vertical coordinate information?
04/21/23 19Annual briefing to ESDIS
[StdPressureLev][StdPressureLev]
[StdPressureLev][StdPressureLev]
[StdPressureLev][StdPressureLev]
Issue 3: No enforcement for the use of standard names
04/21/23 Annual briefing to ESDIS 20
Schematic of Grid
04/21/23 Annual briefing to ESDIS 21
YD
imXDim
StructMetadata.0 =“… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0)… Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS:Float32 Ozone(YDim, XDim)
StructMetadata.0 =“… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0)… Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS:Float32 Ozone(YDim, XDim)
No enforcement for the use of standard names
• Use of non-standard dimensions in grid e.g., MOPITT (MOP03) To be interpreted properly, each data field must
make use of the two predefined dimensions: XDim and YDim
However, none of data fields in that file refers to XDim or YDim. Instead, they refer to nlon and nlat, which are not standard way.
04/21/23 22Annual briefing to ESDIS
No enforcement for the use of standard names
• Object names are not restrictive e.g., MODIS (MAC03S0) Name of a data field called
• Land/SeaMask A dimension called
• nscans*10 These names may not be safe in other APIs.
Especially, HDF5 does not allow slash(/).
04/21/23 23Annual briefing to ESDIS
Other bad practices
• Extreme long attributes• Chunk size much bigger than dimensional size
for fixed size SDS
• These are not enforced by HDF4 library
04/21/23 24Annual briefing to ESDIS
Other technical difficulties
• Hybrid HDF-EOS2 files Some data cannot be accessed by HDF-EOS2
library
• Need to help users access these files
04/21/23 25Annual briefing to ESDIS
Identify problems : service part
04/21/23 Annual briefing to ESDIS 26
A message from DAAC User Survey
04/21/23 27Annual briefing to ESDIS
I am a student and I need to read MODIS HDF-EOS data. But it is hard for me to extract the data. I need a fortran program to extract the data. I wish DAAC can provide this.
DAAC tool sites
• Provide most information users need
04/21/23 28Annual briefing to ESDIS
What’s left?
• Some tools are out of date• On some sites, no C/fortran examples to access
the data• Information on the tool page out of date
04/21/23 29Annual briefing to ESDIS
What’s the problem?
• A small percentage of EOS2 users needs help. Access “difficult” files Compensate tools/examples for DAACs
04/21/23 30Annual briefing to ESDIS
The HDF Group’s plan to DAACs
• Compensate fortran/C example code for DAACs• C/fortran examples for hybrid files and “difficult”
files• Evaluate/update tools listed at hdfeos.org• Evaluate tools at DAACs• EOS-forum
Web forum? EOS2 user and developer mailing lists?
04/21/23 31Annual briefing to ESDIS
Thank You!
04/21/23 Annual briefing to ESDIS 32
Acknowledgements
This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A.
Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.
04/21/23 33Annual briefing to ESDIS
Questions/comments?
04/21/23 Annual briefing to ESDIS 34