Upload
moana
View
22
Download
2
Tags:
Embed Size (px)
DESCRIPTION
HDF-EOS Maintenance, Current Development and Tools. Abe Taaheri, Raytheon Information Systems Larry Klein, RS Information Systems HDF-EOS Workshop X, November 2006. HDF-EOS Support. Maintenance support is funded on EMD Evolution through May 2008 Current staff - PowerPoint PPT Presentation
Citation preview
HDF-EOS Maintenance, Current Development
and Tools
Abe Taaheri, Raytheon Information SystemsLarry Klein, RS Information Systems
HDF-EOS Workshop X, November 2006
Page 2
HDF-EOS Support
• Maintenance support is funded on EMD Evolution through May 2008
• Current staff - One full time developer for maintaining: SDP/MTD Toolkit, HDF-EOS2 & 5, HDF-EOS plug-in for HDFView, HEG, other Toolkit/HDF-EOS related software/maintenance:
- Users Guides- Web sites- User help
- Training another developer for HEG maintenance
Page 3
ECS Support of HDF-EOS
• Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline.
• OS's: Solaris (8, 9, 10), Irix6.5, HP 11, AIX, DEC, Windows NT/2000/XP, Linux (including 64-bit Opteron and Itanium), Mac OS X, Cygwin (MTD Toolkit, HDF-EOS2 & 5)
• Adding Cygwin support for SDP Toolkit
• Possibly supporting Intel based Mac for next release
• Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++
• Support for SGI 7.4.2 besides SGI 7.3.1
Page 4
Software Releases
SDP Toolkit/HDF-EOS:• Last release – March 2006
- TOOLKIT 5.2.14, HDF-EOS 2.14, HDF-EOS5.1.10- with HDF4.2r1, HDF5-1.6.5-post5, Szip2.0
• Next release – Early 2007- TOOLKIT 5.2.15, HDF-EOS 2.15, HDF-EOS5.1.11- with new releases of HDF4, and HDF5
MTD Toolkit: Last Release – Nov. 2006
HDF-EOS Plug-in for HDFView:• Last release – May 2006
– Version 2.3 • Next release – ?
– After HDFView2.4 release by The HDF Group possibly in early 2007
HEG:• Last release – March 2006
– Version 2.7• Next release – Early 2007
Page 5
Downloadable Applications
• HE5View* (HDF5 - based browser)
• EOSView* (HDF4 - based browser)
• HDFView Plug-in, Java-based browser (HDF4, HDF5, HDF-EOS 2 and 5 access)
• heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents)
– Compliments HDF4 -> HDF5 conversion tool
– Not tested on all products
• HEG (HDF-EOS2 to GeoTIFF converter, subsetting, reprojection, stitching, sub-sampling, etc.)
* Dropped support for OS updates since July 2006
Page 6
SDP/MTD Toolkit
Science Data Production (SDP) Toolkit is a collection of tools used by
- Science software developers who produce code to process instrument data.
- HDF-EOS users who need to• Access metadata in HDF-EOS files• Perform Time/Date conversion• Access Digital Elevation Model Data• Access Ancillary Data, such as Digital
Chart of the World database (DCW), Olson World Ecosystem files, DEM data files, etc.
Page 7
SDP Toolkit (cont.)
• Tools provided in (SDP, MTD) Toolkit:- AA (Ancillary Data Access)- CBP (CELESTIAL Body Position)- CSC (Coordinate System Conversion)- CUC (Constant and Unit Conversions)- DEM (Digital Elevation Model Access)- EPH (Ephemeris Data Access)- GCT (Geo Coordinate Transformation)- IO (Input Output (File I/O))- MEM (Memory Management)- MET (Metadata Access)- PC (Process Control)- SMF (Status Message File (Error/Status))- TD (Time Date Conversion)
Page 8
What is HDF-EOS ?
• HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures
–By providing a single interface to data structures common to earth science
–By providing a container for EOS inventory, archive and product specific metadata
–By establishing a consistent relationship between geolocation and science data.
Page 9
HDF-EOS 2
• HDF4 - based, storage format for EOS standard products.
• Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments
• Support for Grid/Point/Swath structures
• HDF4 and HDF-EOS 2 will be supported by THG/NASA for the fore-seeable future.
Page 10
HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF-EOS2 with a different user interface.
–First released in 2000.
–Format used by EOS Aura instruments
• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible.
–Supports same data structures
–Added prefix ‘HE5_’ to HDF-EOS 2 functions.
Page 11
HDF-EOS Grid Structure
• Usage - Data which is organizedby regular geographic spacing, specified by projection parameters.
• Structure–Any number of 2-D to 8-D data arrays per structure–Geolocation information contained in projection formula,
coupled by structural metadata.–Any number of Grid structures per file allowed.
Page 12
HDF-EOS Swath Structure
• Data which is organized by time, or other track parameter.
• Spacing can be irregular.
• Structure–Geolocation information stored
explicitly in Geolocation Field (2-D array)
–Data stored in 2-D or 3-D arrays–Time stored in 1-D or 2-D array,
TAI is standard. –Geolocation/science data
connected by structural metadata
Page 13
HDF-EOS Point Structure
• Data is specified temporally and/or spatially, but with no particular organization
• Structure–Tables used to store science
data at a particular Lat/Long/Height
– Up to eight levels of data allowed. Structural metadata specifies relationship between levels.
Station Lat Lon Chicago 41.49 -87.37 Los Angeles 34.03 -118.14 Washington 38.50 -77.00 Miami 25.45 -80.11
Time Temp(C) 0800 -3 0900 -2 1000 -1 0800 20 0900 21 1000 22 1100 24 1000 6 1100 8 1200 9 1300 11 1400 12 0600 15 0700 16
Page 14
Zonal Average Structure
• Generalized array structure with no geolocation linkage
• Data can be organized by time or track parameter
• Data spacing can be irregular• Structure
– Data stored in multidimensional arrays– Time stored in 1-D or 2-D array
Page 15
Last Release:
HDF-EOS5: POINT object fixes of:
- linkage between levels- updating levels - writing Forward and Backward Linkage data- char field inquiry
Added a routine to return list of aliases for a given field Used HDF5’s H5T_C_S1 datatype for string type Added a few more Fortran wrappers Modified symbol “H5E_NONE_MAJOR” to other symbols
for better description of errors Ported to Solaris 9, Solaris 10, and Cygwin C++ fixes for SGI 7.4.2 compiler support
Major Development/Fixes
Page 16
Fixed subsetting problem in HDF-EOS5 Grid, resulting from faulty conversion of DMS degree to Decimal degree.
Setting LD_LIBRARY_PATH for SZIP/HDF5 shared libraries in installation and environment setting scripts
TOOLKIT/HDF-EOS2: Ported to Solaris 9 and Solaris 10 Keeping more significant digits after the decimal point in
projection parameters if needed. chkeph utility fixes for reading binary ephemeris files
(Problem in reading EPH files with different endiannes than the machine ) SGI 7.4.2 compiler support orbsim fixes for Linux to create files similar to those in
UNIX platforms
Major Development/Fixes (cont.)
Page 17
Next Release: Default installation of SZIP Choice for installing Toolkit/HDF-EOS in 32 or 64 bit
modes in 64-bit Linux platforms New HDF4/HDF5 Release support Lat/Lon conversions to pixel number in DEM tools in
Linux for getting results similar to those in Unix platforms
Future Plans: −Support for Cygwin−Support for HDF4.2r2 and HDF5-1.8−Performance improvement, if possible, for reading HDF
format ephemeris/attitude files in Toolkit−Performance improvement in HDF-EOS5 by redesigning
some functions
Future Development/Fixes
Page 18
HDFView
• HDFView: A Java-based browser from THG for HDF4 and HDF5 – based data.
• HDF-EOS plug-in extends HDFView to browse HDF-EOS data in both versions.
• The extension is a plug-in module and does not change THG HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains.
• Identical look and feel when displaying HDF and HDF-EOS objects.
• Users can access a complete breakdown of HDF-EOS objects. e.g. grid information, projection info., dimensions, datafields, and attributes are displayed.
• Current Support: SUN, SGI, LINUX, WINDOWS platforms
Page 19
HDFView (Cont.)
• Release 2.2 October 2005
- A separate HDF-EOS plug-in: Users install HDFView 2.2 and integrates HDF-EOS plug-in into it.
- Ability to do "Save as" for HDF-EOS5:
modify individual values in grid or swath data.
Delete entire grids or swaths.
Save the result.
- Separate help window for HDF-EOS objects
• Release 2.3 January 2006−Display Point Objects
−Scientific notation in Table data
−Geolocating pixels for swath and grid images
Page 20
HDFView (Cont.)
• Next Release: Version 2.4–Automatic installation of plug-in into HDFView running self-
extracting archives.–Display image for selected pixels in the viewed tables–Geolocating cells in the data field tables for swath and grid –Mac OSX porting–Better display of Plug-in User’s Guide–New icons for activating Lat/Lon display for Swath & Grid
- Future Plans:- Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes.- Ability to convert HDF-EOS2 to HDF-EOS5 - Ability to create new grids/swaths
Page 21
Swath, Grid, and Point Objects
Page 22
AMSR-E Grid, Northern Hemisphere, with selected palette
Page 23
AMSR-E and OMI Point objects
Page 24
AMSR-E Swath
New Features in Next Release Version 2.4
Page 25
Aster Swath, with Image menu
Page 26
HEG: Utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats.
• Two versions available:–A downloadable desktop version –Access through NASA archive online storage (Data Pools)
– Implemented also in prototypes for Stand-alone Data Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs
HDF-EOS to GeoTIFF Converter (HEG)
Page 27
HEG Functionality
• Conversion:–HDF-EOS to a single-band or multi-band GeoTIFF’s–HDF-EOS Swath to HDF-EOS Grid and Swath–HDF-EOS Swath or Grid to GeoTIFF–HDF-EOS Swath or Grid to generic Binary (with
metadata file)• Subsetting (spatial, field, band)• Stitching (mosaicing) + subsetting + Reprojecting• Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN)• Subsampling (Subsample stacks)
• Running from Command Line (batch jobs) or with Java-based GUI
Page 28
HEG Functionality
• Metadata preservation/creation.• Currently supports MODIS, MISR, ASTER, AIRS and AMSR-
E products on TERRA and AQUA (>110 products), • Allows file selection from users local storage• Operable on Sun, SGI, Win, Linux, MAC
• Integrated into ECS Data Pool– Reduces the transfer time of HDF-EOS data sets (if subset is
requested)– Provides the end-user with the exact file required by their application
• User Interface– Portable, written in Java.– Not dependent on COTS (eg. IDL).
Page 29
HEG Releases– Support for:
AMSR-E L3 GRID products (in Polar Stereographic and EASE-Grid)AMSR-E L3 GRID products (in Geographic projection)MISR L3 products (in Geographic projection).AMSR-E L2 SWATH products AIRS L3 products (in Geographic projection)ASTER DEM products: Correction for missing data in the internal
geolocation fields of the ASTER DEMs.MOD43C1, MOD43C2, MOD43C3, MCD43C1, MCD43C2, MCD43C3, MOD12C1, MYD12C1: Corner points of GRID in DD instead of DMS units. MOD11_L2: fillvalues (-999.0) in the Lat/Lon geolocations while (-999.9) in internal metadata.
– Added Albers Projection– New "Subsetter" functionality for Parameters (aka Fields). This results in:
• Output to HDF-EOS SWATH from input HDF-EOS SWATH• No alteration of input data because of projection operation.
Page 30
Mt. Etna Eruption: MISR L1B Swaths: Stitched
Page 31
MODIS L1B: La Plata Tornado
Reprojected, Co-Registered with ASTER
Page 32
2004 Hurricane Francis, MODIS
MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5
Page 33
AST_L1B, 2003 California FiresGeo. projection, Multi-Band GeoTIFF,
(3 bands: Visibile Near Infrared (VNIR))
Two Swaths Stitched …..
….. and Subsetted
Page 34
MODIS Land Products: Reprojected, Co-registered, Processed in ArcInfo/ArcMap
Vegetation Index Leaf Area Index Surface Reflectance
Page 35
Availability
• Access to libraries and applications: TOOLKITTOOLKIT:: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html
HDFViewHDFView::http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/HDFView_hdfeos_plugin.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/HDFView_hdfeos_plugin.html
HEGHEG: : http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html
• Email–[email protected]–[email protected]