Upload
marvin-thornton
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
VSO Status Update
HDMC Meeting
9 June 2008
Vision
To allow solar physicists to identify and search for data even if they don't know it exists.
Make it easier for them to get to the data whether they knew where it was before or not.
Why VSO?
Why not CDAWeb?Will they take > 2TB/day of data?
Why not FTP and VSPO:Do you want to scan > 70k files per day to
determine which ones meet your needs?Every time you perform a search?
(these numbers are just for SDO)
How to accomplish:
Standardized API for querying & requesting data using a consistent model at each provider
System to allow UIs to query a single API and have the queries distributed & responses aggregated.
Current Status
Data Providers : 14
Observatories : 26
Instruments : 55
Data Sets : 180?... dating back to 1915
Current Status (SPASE terms)
Repositories : 19 Observatories : 26 Observatory Groups : 3 Instruments : 66+ Instrument Groups : ~10 Numerical Data : ~180 products Display Data : (not tracked directly) Granules : ? millions
Data Sets Planned:
In progress: Hinode/SOT Huairou NRL EIT & LASCO TIMED/SEE
Planned: Hinode/EIS SMART SDO... and catalogs
SPASE use:
um ... no?
Current Interfaces / APIs
VSO SOAP API (finding & ordering): Web-based interface
3 custom VSO-derived interfaces (NSO, SAO, SSC) IDL-based interface
Dominic Zarro's IDL commands in SSW Perl command line interface AstroGrid
Registry API Catalog 'Display' API for HelioViewer
Interfaces / APIs
In progress: IDL client for Registry API More complex Catalogs API(s)
Planned: Cart API
and UI in IDL Catalog APIs for HELM
access via IDL SPASE compliant registry API
or at least SPASE descriptions of holdings
Data Coverage:
Incomplete.Measurement method:
People are still asking us to add data sets.
Goals:
Prepare for the flood of SDO data (and other recent missions: STEREO,
Hinode )
Plans for the Future:
Catalogs
Summary Rows
Looking to the work of others
Empowering new searches!
We’re doing some work under the hood!
Moving beyond time
• An event or feature driven approach
• Science catalogs• LASCO CME, GOES X-ray Flare, NOAA Active Region Number, Coronal
Holes, Bright points etc.
• Science queriesOne of the drivers for this
• SDO• So much data need to move to feature / event driven approach,• But useful now …
Give me all of these!
Actually I really want all XRT jets observed on the limb with coincident SOTand EIS data.
Would have made things much easier if I could haveperformed this query!
Science Queries
How do we know what queries to support?• We get asked!
• “It would be nice if … / I’d really like to be able to …”
Coincident observations• Observations of same features in multiple instruments within
specified time window.
Cadence• Find an hours worth of observations where there are images every
minute. (Helioseismology)
DEM Datasets• Find XRT datasets suitable for DEM analysis. (Closely spaced
observations, number of different filters/wavelengths). EIS data would be a bonus!
Science Queries
Investigating EIT dimming regions as proxies for CMEs
• Combine GOES Flare List + LASCO CME list to generate list of on disk flares without associated CMEs. Compare list to catalog of EIT dimming regions. Do spatial comparison. Generate list of EIT dimming regions whose properties are observed to be associated with CME list.
• Currently waiting on EIT dimming region catalog! Can do the first part to at least generate list of EIT images that should be analyzed.
• Support driven by science questions!
Conclusions
• We are moving from just data discovery in a certain time range, to finding datasets of interest that fit a number of scientific criteria.
• Searches for features and events and in the process blurring the distinction between data and catalogs.
• SDO will require these new approaches. • But it’s going to return a LOT of data.
Plans : Summary Rows
VSO data providers can return a record summarizing multiple data granules Data Providers can define their own groupings, UIs can
decide to expand them. Summary products still work like regular products
Data providers serve a tarball or data cube in response to a data transfer request
UIs that don’t understand summary products can still retrieve the collection
Changing over to a new JavaScript toolkit to provide AJAX summary row expansion
Plans : Look to Others
Catalogs Bobby Candey's HELM project (VxO funded) Peter Fox's SESDI project (NASA funded) Related IVOA & astronomy standards
Presentation / UI Use existing JavaScript toolkits
ExtJS and GWT... still requires customization to serve our needs, but gives
head start
Plans : Look to Others (2)
Massive amounts of data:How libraries handle interlibrary loanHow libraries handle 'de-duplicating' of
records Functional Requirements of Bibliographic
Records… is going to require significant changes at the data
providers.
The Future of VSO
• We are moving from just data discovery in a certain time range, to finding datasets of interest that fit a number of scientific criteria.
• Searches for features and events and in the process blurring the distinction between data and catalogs.
• SDO will require these new approaches.
http://virtualsolar.org/