Click here to load reader

Lehrstuhl Informatik III: Datenbanksysteme Astrometric Matching - E-Science Workflow 1 Lehrstuhl Informatik III: 1 Datenbanksysteme 1 Fakultät für Informatik

  • View
    216

  • Download
    3

Embed Size (px)

Text of Lehrstuhl Informatik III: Datenbanksysteme Astrometric Matching - E-Science Workflow 1 Lehrstuhl...

  • Slide 1

Lehrstuhl Informatik III: Datenbanksysteme Astrometric Matching - E-Science Workflow 1 Lehrstuhl Informatik III: 1 Datenbanksysteme 1 Fakultt fr Informatik 1 Technische Universitt Mnchen Max-Planck-Institut 2 fr Astrophysik Max-Planck-Institut 3 fr extraterrestrische 3 Physik Full paper at the: 2nd IEEE International Conference on e-Science and Grid Computing, Dec. 4-6, 2006, Amsterdam Slide 2 Lehrstuhl Informatik III: Datenbanksysteme 2 Astrometric Matching - E-Science Workflow The SED Scenario 1.Catalog query 2.Astrometric (spatial) matching 3.Assembly of raw photometry 4.Photometric transformation 5.SED classification Slide 3 Lehrstuhl Informatik III: Datenbanksysteme 3 Astrometric Matching - E-Science Workflow Astrometric (Spatial) Matching Current solutions load all data into main memory Uses a lot of memory Infeasible if memory size is insufficient process all data at once and deliver the complete result at the end Inefficient No results until all processing has completed Slide 4 Lehrstuhl Informatik III: Datenbanksysteme 4 Astrometric Matching - E-Science Workflow Our Contributions In-network processing Early filtering Parallelization Pipelining (data streaming) Load-balancing Mobile user-defined operators Dynamic integration Extensible framework Legacy applications Slide 5 Lehrstuhl Informatik III: Datenbanksysteme 5 Astrometric Matching - E-Science Workflow The StarGlobe Architecture Slide 6 Lehrstuhl Informatik III: Datenbanksysteme 6 Astrometric Matching - E-Science Workflow Mobile User-Defined Operators Infrastructure provided by StarGlobe encapsulated operators provided by community Load user-defined operators from function provider servers in the network Common interface for integrating external operators Flexibility Slide 7 Lehrstuhl Informatik III: Datenbanksysteme 7 Astrometric Matching - E-Science Workflow Communication between Stream Processor and Stream Iterator Slide 8 Lehrstuhl Informatik III: Datenbanksysteme 8 Astrometric Matching - E-Science Workflow Astrophysical Example Workflow Slide 9 Lehrstuhl Informatik III: Datenbanksysteme 9 Astrometric Matching - E-Science Workflow Astrophysical Example Workflow (Setup) Slide 10 Lehrstuhl Informatik III: Datenbanksysteme 10 Astrometric Matching - E-Science Workflow Distributed Query Evaluation Plan Slide 11 Lehrstuhl Informatik III: Datenbanksysteme 11 Astrometric Matching - E-Science Workflow Conclusion StarGlobe Prototype Handling large data volumes efficiently Early filtering, parallelization, pipelining Returning first results early on Pipelining (data streaming) Flexible support of domain-specific application logic Mobile user-defined operators Results also applicable to other domains