Upload
elinor-warren
View
212
Download
0
Embed Size (px)
Citation preview
HDF & HDF-EOS Workshop VIIIHDF & HDF-EOS Workshop VIII
2004 October 26-28 Aurora, CO2004 October 26-28 Aurora, CO
Bruce Beaumont, Matt Smith, Bruce Beaumont, Matt Smith, Helen Conover, Sara GravesHelen Conover, Sara Graves
Subsetting at UAH
Why Subset?Why Subset?
• Goal: to provide a science data user with only the data they need as quickly as possible.
• Benefits science data users and data centers:- Reduces analysis time by reducing amount of data- Reduces time for data delivery- Reduces resources (network, personnel, media, etc.)
• Steps:- Locate spatial / temporal / spectral area of interest- Extract- Re-assemble for distribution/use
What is HSE?What is HSE?HHEW EW SSubsetting ubsetting EEnginengine
A new packaging option for the HDF-EOS Subsetter designed for users that want subsetting, but do not want to host the full HEW installation or even the standalone HEW back-end.
What is HEW?What is HEW?
• HDF-EOS Web-based Subsetter– Prototype software designed to be dataset-
independent (HDF-EOS)– Funded by NASA/ESDIS for EOS-DIS– To be used within ECS (EOS-DIS Core System)
– Original Front-end/GUI (optional)• Uses HTML forms and JavaScript
– Original Back-end• Needed subset criteria and HDF-EOS data
• Performed subsetting as a “batch” job
What are HSE’s capabilities?What are HSE’s capabilities?
• Versions available for HDF-EOS 2 and HDF-EOS 5.
• Subsets multiple files in one call.
• Subsets properly-formatted HDF-EOS grid and swath objects.
• Subsets spatially by latitude/longitude or row/column.
• Subsets swaths temporally by date/time range.
• Subsets swaths by full or partial (subscan) lines.
• Subsets spectrally by HDF-EOS field.
• Subsamples along any dimension by stride (repeating interval) or by discrete index list. (see chart)
• Copies all file and field attributes to the output file and updates HDF-EOS “core” metadata when possible.
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
Stride Subsampling on a Dimension•STRIDE = 2
Indexed Subsampling on a Dimension•INDEXES = (1, 3, 4, 5, 7)
Selected
Selected
Existing HEW Back-End Existing HEW Back-End ArchitectureArchitecture
• Front end supplies subsetting criteria file
• Subsetter writes messages to status file
• Subsetter writes messages to log file
• Subsetter sends e-mail to end-user
Subsetting criteria (ODL)
HEW_ DISS
End-user
Input file(s)
Output file(s) Log file
Status file
Many sites do not want to create ODL files or the hassle of dealing with message or log files or e-mail.
New HSE New HSE ArchitectureArchitecture
• User application code builds subsetting criteria structure
• Subsetting engine function calls user’s functions for status and log messages
• No e-mail is sent
User Application Code
HEW Subsetting Engine function Input
file(s)
Output file(s)
HSE_ LogMsg callback
HSE_ StatusMsg callback
Subsetting criteria Return code
All subsetting functionality is contained within a callable function
SPOTSPOT
• Subsettability “checker”– Displays content/structure
of HDF-EOS4 (or HE5) files– Examines files for subsettability by HSE– Simple command-line interface– Stand-alone operation– Available at subset.org
HEW integration with ECSHEW integration with ECS
ECS EDG System
EDG ECS
Subsetter Input data
Output data
End user
Order submission
(HTML)
Data order and reply
Subset ODL and reply
Subsetting System
Output data (Reingested)
1
2
3 4
5
6
7
Product Availability MatrixProduct Availability Matrix
Product
HDF-EOS 2
(HDF4)
HDF-EOS 5
(HDF5)
HSE SGI Sun Linux N/A
HSE-5 N/A SGI Sun Linux
HEWBE SGI Sun Linux Planned
HEW SGI Sun Linux N/A
SPOT SGI Sun Linux
Currently Available/Planned Currently Available/Planned Subsetting ApplicationsSubsetting Applications
• HDF-EOS Subsetting Engine (HDF-EOS, HE5)Status Deployments– Complete System available GHRC
– Subsetting Engine Only available GSFC
– SPOT - Subsettability Checker available many
– HSE Integrated with ECS Data Order System available NSIDC, EDC
– HSE Integrated with AMSR-E Processing available AMSR-E SIPS
– Subsetting as a Web Service planned (ECHO)
• Customized Subsetting– MODIS tools available Science teams
– Coarse-grain SSM/I Subsetter available GHRC
• General Purpose Customizable Subsetting– Subsetting Tool using ESML in work (various)
http://subset.orghttp://subset.org