Embed Size (px)
DESCRIPTION
Update on NASA Wish List. Mike Folk, Elena Pourmal , Kent Yang, Joe Lee The HDF Group Briefing to ESDIS April 4, 2013. Overview. NASA Wish List is a set of tasks requested during last briefing . Total 48 items in the list. Focus: 36 High Priority Items - PowerPoint PPT Presentation
Citation preview
www.hdfgroup.org
The HDF Group
Update on NASA Wish List
Mike Folk, Elena Pourmal, Kent Yang, Joe Lee The HDF Group
Briefing to ESDISApril 4, 2013
4/4/2013 HDF Briefing to NASA 1
www.hdfgroup.org
Overview
• NASA Wish List is a set of tasks requested during last briefing.
• Total 48 items in the list.• Focus: 36 High Priority Items
• Two Divisions• General Maintenance, QA, and Support
(GMQS)• ESDIS Support, Operation Users Support, and
Studies (EOS)
4/4/2013 HDF Briefing to NASA 2
www.hdfgroup.org
GMQS Division
Accountable: Elena PourmalCategories:• HDF4 and HDF5 Libraries• General Tools (HDFView, h5dump, etc.)• General Support (Releases)
4/4/2013 HDF Briefing to NASA 3
www.hdfgroup.org
EOS Division
Accountable: Kent YangCategories:• ESDIS Software (including OPeNDAP)• Promotion• Studies
4/4/2013 HDF Briefing to NASA 4
www.hdfgroup.org
GMQS > Libraries
HDF4 Library• Item 12: Modify a compressed HDF4 data array.
HDF5 Library• Item 34: Add “parallel” gzip.
4/4/2013 HDF Briefing to NASA 5
www.hdfgroup.org
GMQS > Libraries > Item 12
Item Modify a compressed HDF4 data array
Action Reviewed and confirmed that HDF4 library cannot overwrite a part of the dataset (all or nothing). Created an example code that HDFView can use to resolve user’s request.
Original comments
HDF Java user reported this problem for his quality screen application.
Status In Progress
Comments • Update HDFView (version 2.10). • Do more investigation in HDF4. Fix bugs.• Provide recommendations to HDF4 library
users on how modify compressed datasets.
4/4/2013 HDF Briefing to NASA 6
www.hdfgroup.org
GMQS > Libraries > Item 34
Item Add parallel gzip
Action Reviewed parallel gzip code. It cannot be used with HDF5 as is. Talked to other stakeholders interested in the feature.
Original comments
GES DISC (?) looked at parallel gzip to improve performance.
Status In Progress
Comments Work on the design document for the feature.Come up with the estimate to do the work.Talk to all stakeholders to raise funds.
4/4/2013 HDF Briefing to NASA 7
www.hdfgroup.org
GMQS >Tools
h5dump• Item 9: Clean ASCII output from h5dump
HDFView• Item15: HDFView and netCDF-4 files with VL attributes• Item 4,18, 35: HDF-EOS plug-in and HDFView• Item 17, 22-27: New features in HDFView
4/4/2013 HDF Briefing to NASA 8
www.hdfgroup.org
GMQS > Tools > Item 9
Item Clean ASCII output for h5dumpAction RFC is sent to James Johnson and Chris
Lynnes. The fix is made and ready for 1.8.11 release.
Original comments
Goal is to easily provide clean ASCII output, without attribute info., etc.
Status DoneComments
4/4/2013 HDF Briefing to NASA 9
www.hdfgroup.org
GMQS > Tools > Item 15
Item NetCDF-4 files with string type attributes give variable length of size of one.
Action Fixed and changes committed to SVN repository.
Original comments
There doesn’t seem to be a way to create variable length strings.
Status DoneComments It’ll be in HDFView 2.10, which is available
upon request.
4/4/2013 HDF Briefing to NASA 10
www.hdfgroup.org
GMQS > Tools > Item 4
Item Obtain permission to redistribute HDF-EOS plugin with HDFView
Action Sent a letter to Dan and confirmed with Lou.
Original comments
Lou will look into getting permission. Lou may need to ask Dan for a letter of approval.
Status DoneComments We can release HDF-EOS plug-in with
HDFView in our website.
4/4/2013 HDF Briefing to NASA 11
www.hdfgroup.org
GMQS > Tools > Item 18
Item Synchronize EOS plugin with HDFViewAction We plan to come up with the RFC document
that discusses the 4 solutions and we will solicit recommendations.
Original commentsStatus Planned
Comments See the next slide for 4 solutions.
4/4/2013 HDF Briefing to NASA 12
www.hdfgroup.org
GMQS > Tools > Item 18
4/4/2013 HDF Briefing to NASA 13
1. Add the plug-in to the HDFView installer but have an option to install the plug-in or no.
2. Have a separate HDFView installer that packed with the plug-in and will install the plug-in automatically.
3. Have a Add plug-in in HDFView so that users can add the plug-in anytime.
4. Ask Abe to distribute HDFView with the plug-in (like #2) and we will provide a link to his release of HDFView.
www.hdfgroup.org
GMQS > Tools > Item 35
Item HDF-EOS Plug-in FunctionalityAction Discuss the identified features with James
Johnson and other ESDIS stakeholders (RFC)
Original comments
James asked if HDFView can have some functionality like Panoply.
Status Gathered requirements and identified list of features
Comments The common feature between HDF-EOS Plug-in and Panoply is the ability to display data on map.
4/4/2013 HDF Briefing to NASA 14
www.hdfgroup.org
GMQS > Tools > Item 25
Item New Features in HDFView - OPeNDAPAction We’re working on RFC.
Original comments
You’ll probably get more requests for this in the future as far as MODIS goes.
Status In Progress
Comments Joe had discussion with Peter on OPeNDAP.
4/4/2013 HDF Briefing to NASA 15
www.hdfgroup.org
GMQS > Tools > Item 17, 24, 27, 23
Item New Features in HDFView:Dimension Scales, DOI, Graphics, GeoTIFF
Action Dimension Scales and DOI: AprilGraphics (fill values, etc.): MayGeoTIFF: June
Original commentsStatus Scheduled RFC
Comments GeoTIFF work depends on Item 8, which will be discussed in EOS division later.
4/4/2013 HDF Briefing to NASA 16
www.hdfgroup.org
GMQS > Tools > Item 22, 26
Item New Features in HDFView:CF Conventions, New web service queries
Action The survey report was completed and published.Original comments
Peter will do this as part of writing up results of survey.
DAACS are starting to provide services for WCS, OpenSearch, etc. but there are very few clients that support them.
Status No plan
Comments http://hdfgroup.org/hdf-java-html/hdfview/hdfview_surevey_report_2013.pdf
4/4/2013 HDF Briefing to NASA 17
www.hdfgroup.org
GMQS: General Support Issues
• Item 1: Supported OS and compilers• Item 3: Cygwin support is needed by HDF-EOS• Item 20: Tools of value to users
4/4/2013 HDF Briefing to NASA 18
www.hdfgroup.org
GMQS > General Support > Item 1
Item Need to know what OS and compilers are supported.
Action • OS and compilers are planned for each maintenance release.
• An HDF developer is assigned at the time of release planning for support issues on a particular platform.
Original comments
Status In Progress
Comments See our wiki page in the next slide.
4/4/2013 HDF Briefing to NASA 19
www.hdfgroup.org
GMQS > General Support > Item 1
4/4/2013 HDF Briefing to NASA 20
www.hdfgroup.org
GMQS > General Support > Item 3
Item Cygwin support is needed by HDF-EOS.Action Cygwin is tested daily for HDF4 and HDF5.
Original comments
Status Done
Comments
4/4/2013 HDF Briefing to NASA 21
www.hdfgroup.org
GMQS > General Support > Item 20
Item Tools of value to users
Action • We updated and reorganized tools pages* to help users to find the right tool.
• We’ve been working with DAACs to raise awareness of tools through telecons and direct contacts.
Original comments
Earthdata.nasa.gov describes tools available to DAACs. What we do here should be integrated with that and inform them.
Status In Progress
Comments *http://www.hdfgroup.org/products/hdf5_tools/toolsbycat.html
4/4/2013 HDF Briefing to NASA 22
www.hdfgroup.org
10 Minute Break
4/4/2013 HDF Briefing to NASA 23
www.hdfgroup.org
EOS > ESDIS Software
OPeNDAP HDF5 Handler• Item 19:Support NetCDF-4 model in HDF5 handler (Done)• Item 32:HDF5 handler maintenanceOPeNDAP HDF4 Handler• Item 28:HDF4 handler maintenanceOPeNDAP fileout_netcdf Handler• Item 5: Fix fileout_netcdf Handler. (Done) • Item 42:Dump as NetCDF-4 for fileout_netcdf
4/4/2013 HDF Briefing to NASA 24
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 25
EOS > ESDIS Software > Item 19
Item Make the HDF5 handler support the netCDF-4 data model
Action Kent added the support to HDF5 OPeNDAP handler in December
Original comments
GES DISC had netCDF-4 files that would like to be served by Hyrax
Status DoneComments A RPM was created and was sent to GES
DISC in Jan. Everything works fine.
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 26
EOS > ESDIS Software > Item 32
Item HDF5 OPeNDAP handler maintenance
Action Improve code quality and tests for HDF5 OPeNDAP handler (default option only)
Original commentsStatus Will get started on June 1st, 2013Comments
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 27
EOS > ESDIS Software > Item 28
Item HDF4 OPeNDAP handler maintenance
Action Improve the code quality and other maintenance
Original comments
Need to improve the code readability, error handling etc.
Status In ProgressComments • We replaced the new[ ] with vector
according to the suggestions of OPeNDAP team.
• We added support in handling MEaSUREs VIP product and the version 6 of MODIS geo-location files.
www.hdfgroup.org
EOS > ESDIS Software > Item 5
Item OPeNDAP’s file netCDF out module is not working in Hyrax 1.8
Action Investigate the problem: Then fix the bug or make recommendation.
Original commenats
Chris Lynnes and James Johnson: This is the first priority.
Status Done
Comments GES DISC, LP DAAC and MODAPS all installed this version and they reported the file netCDF dumper works fine.
4/4/2013 HDF Briefing to NASA 28
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 29
EOS > ESDIS Software > Item 42
Item netCDF-4 (as opposed to netCDF-3) as a file dumper output possibility for OPeNDAP.
Action Make this item a high priority based on the conversation with GES DISC (Previous priority: medium high)
Original commentsStatus Will work on this in JulyComments Using netCDF-4 classic(with the
compression) as the file dumper output becomes more important.
www.hdfgroup.org
EOS > ESDIS Software
H4CF Conversion Toolkit• Item 21:H4CF Conversion Toolkit
HDF-EOS5 to NetCDF-4 Tool• Item 29:Retire HDF-EOS5 to NetCDF-4 conversion tool (Done)
4/4/2013 HDF Briefing to NASA 30
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 31
EOS > ESDIS Software > Item 21Item Improve the H4CF conversion toolkitAction Upgrade the H4CF conversion toolkit to make it
consistent with the way how the HDF4 OPeNDAP handler handled the NASA HDF-EOS2 and HDF4 data.
Original comments
• Will update naming conventions, fix critical bugs, improve features for popular products such as MODIS.
• Will implement a test suite and update the configuration process.
Status In Progress. ( Release plan: June 15)Comments • Good progress upgrading the toolkit to follow CF
conventions as the HDF4 OPeNDAP handler. • Implemented comprehensive test suite• Updated configuration and conversion to netCDF3
and netCDF4-classic. • MISR data & coding style still need to be improved.
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 32
EOS > ESDIS Software > Item 29
Item Retire "HDF-EOS 5 to netCDF-4" conversion tool
Action Made an announcement and waited for replies
Original comments
HDF-EOS5 augmentation tool is a superset of this tool.
Status DoneComments • Announced on Dec. 5th, 2012.
• No response by Feb. 15th, 2013, so we announced tool’s official retirement on Feb. 15th, 2013.
www.hdfgroup.org
EOS > Operation Users Support
Guide Data Producers with HDF Best Practices• Item 11: Document issues for NASA HDF4/HDF-EOS2 products
Make HDF-EOS Easy to Build• Item 30:CMake for HDF-EOS• Item 31:Autoconfiguration for the EOS Toolkit
4/4/2013 HDF Briefing to NASA 33
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 34
EOS > Op. Users Support > Item 11
Item Publish the document describing issues for NASA HDF4 (EOS2) products
Action Write up the issues in a white paper. Make available on website. Push to all relevant stakeholders.
Original comments
Needs to be a clear and concise document. "Guide to good use of HDF." "Best practices," something like that. Perhaps create a "best practices" section on the web site.
Status In ProgressComments We are working on the outline of the white
paper now.
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 35
EOS > Op. Users Support > Item 30
Item CMake for HDF-EOS
ActionOriginal comments
Clarify with Abe what it means.
Status Not started yet, assigned the person to work on this
Comments Communicated with Abe several times.
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 36
EOS > Op. Users Support > Item 31
Item Autoconfiguration for the ECS toolkit.
Action Meet with Abe and look at all of his tools in the toolkit (including libraries) and create a roadmap for implementing autoconfiguration.
Original commentsStatus Not started yet, assigned the person to work
on thisComments Communicated with Abe several times.
www.hdfgroup.org
EOS > Promotion
HDF-EOS website• Item 16: Hard to find zoo link (Done)• Item 2: Promote example pages on HDF-EOS web site• Item13: Publicize HDF-EOS website
4/4/2013 HDF Briefing to NASA 37
www.hdfgroup.org4/4/2013 HDF Briefing to NASA 38
EOS > Promotion > Item 16
Item Hard to find zoo link at hdfeos.org
Action Improve the ability to find example pagesOriginal commentsStatus DoneComments Put the zoo link at the top section in
examples page with a description.
Put the zoo link in the front page’s DID YOU KNOW tip section.
www.hdfgroup.org
EOS > Promotion > Item 2
4/4/2013 HDF Briefing to NASA 39
Item Promote example pages on HDF-EOS web siteAction With Lou's help, contact Earth data pages about
putting pointers to these pages. Contact DAAC user services working groups. Work with ESDIS to submit an article to Earth Observer.
Original commentsStatus In ProgressComments 1) A link* to the comprehensive page at hdfeos.org
has been added to earthdata.nasa.gov. 2) We worked with ESDIS to submit an article about
the comprehensive example to Earth Observer.
*http://earthdata.nasa.gov/user-resources/tutorials-and-examples
www.hdfgroup.org
EOS > Promotion > Item 13
4/4/2013 HDF Briefing to NASA 40
Item Publicizing HDF-EOS website
Action Write brochure about website. Get help distributing brochure at conferences such as AGU. Contact ESDIS staff about this.
Original commentsStatus In ProgressComments We contacted with ESDIS staff and we are
planning to create a NASA brochure about the HDF-EOS comprehensive example page.
www.hdfgroup.org
EOS > Studies
Joe• Item 6: Open Source Apache Incubator pilot projectKent• Item 8: GeoTIFF investigation• Item 33: A specification for CF metadata for HDF5 and HDF5 is neededMike• Item10: DOI support in HDF4 and HDF5
4/4/2013 HDF Briefing to NASA 41
www.hdfgroup.org
EOS > Studies > Item 6
4/4/2013 HDF Briefing to NASA 42
Item Open Source Apache Incubator Pilot ProjectAction Legal issues like NOSA license were reviewed.
A new project proposal* for HDF to GeoTIFF converter is being written.
Original commentsStatus In Progress
Comments • We are recruiting team members and sponsors!• Champion: Paul Ramirez (NASA JPL)• Mentor: Chris Mattman (NASA JPL)
*http://wiki.apache.org/incubator/HotdoGProposal
www.hdfgroup.org
EOS > Studies > Item 8
4/4/2013 HDF Briefing to NASA 43
Item GeoTIFF investigations
Action Write white paper, jointly with Abe, evaluating GeoTIFF support needs and opportunities.
Original comments
Need to determine which products are important for GeoTIFF support.
Status In Progress
Comments • Collected GeoTiFF information from several data centers and Abe.
• Held a phone discussion with Abe about the paper.
• Started on outline of white paper.
www.hdfgroup.org
EOS > Studies > Item 33
4/4/2013 HDF Briefing to NASA 44
Item A specification for CF metadata for HDF4 and HDF5
Action Write a paper on the current status of work on CF conventions, then decide what is best.
Original comments
• We need to better understand who is doing what on this topic. Ted Habermann is working on it. There will be a discussion on it at the ESDSWG meeting.
• Would like the spec to be made available. Would be good for ESDIS to publish as a tech note/standard.
Status In Progress
Comments We participated in the telecon of ESDS HDF5 and provided material and suggestions about CF and interoperability to the WG chairs.
www.hdfgroup.org
EOS > Studies > Item 10
4/4/2013 HDF Briefing to NASA 45
Item Write paper on DOI support in HDF5Action White paper completed, ready for review.
Original commentsStatus Done
Comments Highlights:• In EOS, DOIs assigned at
program/mission/collection/product level• DOI API and BNF grammar proposed• Implementation estimated at 100 hoursNext steps?
www.hdfgroup.org
Thank You!
• Questions? • Comments?
4/4/2013 HDF Briefing to NASA 46
www.hdfgroup.org
Acknowledgements
4/4/2013 HDF Briefing to NASA
This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.
47
