Embed Size (px)
Citation preview
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow 1
Astrometric Matching - E-Science Workflow
Lehrstuhl Informatik III:1Datenbanksysteme1Fakultät für Informatik1Technische Universität München
Max-Planck-Institut2für Astrophysik
Max-Planck-Institut3für extraterrestrische3Physik
Full paper at the:
2nd IEEE International Conference on e-Science and Grid Computing, Dec. 4-6, 2006, Amsterdam
2
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
The SED Scenario
1. Catalog query
2. Astrometric (spatial) matching
3. Assembly of raw photometry
4. Photometric transformation
5. SED classification
3
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
Astrometric (Spatial) Matching
Current solutions … … load all data into main memory
Uses a lot of memory Infeasible if memory size is insufficient
… process all data at once and deliver the complete result at the end Inefficient No results until all processing has completed
4
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
Our Contributions
In-network processing Early filtering Parallelization Pipelining (data streaming) Load-balancing
Mobile user-defined operators Dynamic integration Extensible framework Legacy applications
5
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
The StarGlobe Architecture
Super-Peer BackboneQuery 1
Stream 0
Publish
Subscribe
6
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
Mobile User-Defined Operators
Infrastructure provided by StarGlobe encapsulated operators provided by community
Load user-defined operators from function provider servers in the network
Common interface for integrating external operators
Flexibility
7
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
Communication between Stream Processor and Stream Iterator
8
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
Astrophysical Example Workflow
Input ListRASS-BSC
2MASS FIRST USNOB1
NVSS GSC-2
SED assembly
9
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
Astrophysical Example Workflow (Setup)
Input ListRASS-BSC
2MASS FIRST USNOB1
NVSS GSC-2
SED assembly
10
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
Distributed Query Evaluation Plan
χ²filter-0
join-0
enrichσ-1
transform-1
stream-1
enrichσ-0
transform-0
stream-0
11
Lehrstuhl Informatik III: Datenbanksysteme
Astrometric Matching - E-Science Workflow
Conclusion
StarGlobe Prototype Handling large data volumes efficiently
Early filtering, parallelization, pipelining Returning first results early on
Pipelining (data streaming) Flexible support of domain-specific application logic
Mobile user-defined operators
Results also applicable to other domains
