Grid-based Data Stream Processing in e-Science€¦ · Lehrstuhl Informatik III: Datenbanksysteme Grid-based Data Stream Processing in e-Science 1 Grid-based Data Stream Processing

Lehrstuhl Informatik III: Datenbanksysteme

Grid-based Data Stream Processing in e-Science 1

Grid-basedData Stream Processingin e-Science

Richard Kuntschke1, Tobias Scholl1, Sebastian Huber1,Alfons Kemper1, Angelika Reiser1,Hans-Martin Adorf2, Gerard Lemson3, and Wolfgang Voges3

1Lehrstuhl Informatik III:1Datenbanksysteme1Fakultät für Informatik1Technische Universität München

2Max-Planck-Institut2für Astrophysik

3Max-Planck-Institut3für extraterrestrische3Physik

2


Grid-based Data Stream Processing in e-Science

Important Challenges in e-Science

In general:Large and exponentially growing amounts of dataDistributed data archivesNo unique identifiersUncertainty

In astrophysics:Spectral Energy Distributions (SEDs)

Used to classify celestial objects (active galactic nuclei, brown dwarfs, neutron stars, ...)Generation requires spatial (astrometric) matching

3



Spatial (Astrometric) Matching

Current solutions …… load all data into main memory

Uses a lot of memoryInfeasible if memory size is insufficient

… process all data at once and deliver thecomplete result at the end

InefficientNo results until all processing has completed

4



Our Contributions

StarGlobeGrid-based P2P DataStream Management System implemented on top of GlobusIn-network processing

Early filteringParallelizationPipeliningLoad-balancing

Mobile user-definedoperators

Astrophysical ExampleWorkflow

Astrometric matchingPerformance evaluation

5



The StarGlobe Architecture

Super-Peer BackboneQuery 1

Stream 0

Publish

Subscribe

filter

transform

Loadmobile operators

Fct-Provider

filter

transform

Stream 1

Publish

Query 2

Subscribe

6



Traditional Approach: Bring Data to Code

union

NN_10

T... ... ...... ... ...... ... ...

Data-Prov. BT

... ... ...

... ... ...

... ... ...

Data-Prov. CT

... ... ...

... ... ...

... ... ...

Data-Prov. DT

... ... ...

... ... ...

... ... ...

Data-Prov. A

7



New Approach: Bring Code to Data

T... ... ...... ... ...... ... ...

Data-Prov. Ascan

NN_10

T... ... ...... ... ...... ... ...

Data-Prov. Bscan

NN_10

T... ... ...... ... ...... ... ...

Data-Prov. Cscan

NN_10

T... ... ...... ... ...... ... ...

Data-Prov. Dscan

NN_10

union

NN_10Fct-Provider

NN_10

8



Mobile User-Defined Operators

Load user-defined operators from functionprovider servers in the network

Common interface for integrating externaloperators

Push-based iterator

Flexibility

9



StreamIterator Interface

open(Config, StreamWriter)Configuration parametersWriter for result stream

next(StreamIteratorEvent)Next element in input streamWriting output to result stream usingStreamWriter.write()

close()

10



Communication betweenStreamProcessor and StreamIterator

StreamIterator

StreamHandler 1

StreamHandler 2

StreamHandler n

StreamWriter

StreamProcessor

XML InputStream 1

XML InputStream 2

XML InputStream n

XML OutputStream

Item 1 Item 2 Item n Result Item

11



Astrophysical Example Workflow

peer-10

peer-9 peer-8

peer-6 peer-4 peer-5peer-7

peer-2peer-1peer-0 peer-3

Input ListRASS-BSC

2MASS FIRST USNOB1

NVSS GSC-2

SED assembly

12



Distributed Query Evaluation Planplan-10

at peer-10

plan-8at peer-8

plan-5at peer-5enrichσ-5

transform-5

stream-5

χ²filter-2

join-2


transform-4

stream-4

plan-9at peer-9χ²filter-3

join-3

plan-7at peer-7


transform-2

stream-2


transform-3

stream-3

χ²filter-1

join-1

plan-6at peer-6χ²filter-0

join-0


transform-1

stream-1

χ²filter-4

join-4

display


transform-0

stream-0

13



Distributed Query Evaluation Plan

χ²filter-0

join-0

enrichσ-1

transform-1

stream-1

enrichσ-0

transform-0

stream-0

14



Distributed Query Evaluation Plan

15



Evaluation of Early Filtering

16



Conclusion

Synergies between research in computer scienceand other scientific disciplines, e.g., astrophysics

StarGlobeHandling large data volumes efficiently

Early filtering, parallelization, pipeliningReturning first results early on

PipeliningFlexible support of domain-specific application logic

Mobile user-defined operators

Results also applicable to other domains

Documents

Grid-based Data Stream Processing in e-Science€¦ · Lehrstuhl Informatik III: Datenbanksysteme Grid-based Data Stream Processing in e-Science 1 Grid-based Data Stream Processing