SAMOA: A Platform for Mining Big Data Streams (Apache BigData North America 2016)

1

SAMOA: A Platform for Mining Big Data Streams

Nicolas KourtellisAssociate Researcher

Telefonica I+D, Barcelona@kourtellis

@ApacheSAMOA

SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016

2

What is Big Data?Search queriesFacebook postsEmailsTweetsPhoto sharesClicks on ads…


3

How BIG is your data?Volume (+ Variety)

Too large for RAM of single commodity serverVelocity

Too fast for CPU of single commodity server


4

What is the Streaming Paradigm?High amount of data, high speed of arrivalUpdated models at “real” timePotentially infinite sequence of dataChange over time (concept drift)


5

Mining Big Data StreamsApproximation algorithms:

Single pass, one data item at a timeSub-linear space and time per data itemSmall error with high probability

A platform solution:Support different algorithms & processing enginesDistributedScalable


6

What is SAMOA?Scalable Advanced Massive Online AnalysisA platform for mining big data streams

Framework for developing new distributed stream mining algorithms

Framework for deploying algorithms on new distributed stream processing engines


7

Taxonomy


8

SAMOA ArchitectureMachine LearningAlgorithms

Distributed StreamProcessing Engines Flink


9

Why is SAMOA important?Program once, run everywhere

Reuse existing infrastructureAvoid deploy cycles

No system downtimeNo complex backup/update processNo need to select update frequency


10

ML Developer API


11

ML Developer API


12

Deployment


13

Easy to get!


14

Easy to get!


15

Easy to get!


16

Easy to test!bin/samoa storm target/SAMOA-Storm-0.3.0-SNAPSHOT.jar"PrequentialEvaluation-d /tmp/dump.csv-i 1000000 -f 100000-l (classifiers.trees.VerticalHoeffdingTree -p 4 -k)-s (generators.RandomTreeGenerator –r 1 -c 2 -o 10 -u 10)"


17

Case study: Decision TreesVHT: Vertical Hoeffding Tree*


Task parallelism

*VHT: Vertical Hoeffding Tree. N. Kourtellis, G. De Francisci Morales, A. Bifet, A. Mordupo. IEEE BigData 2016.

18

Case study: VHT


Horizontal Parallelism

19

Case study: VHT


Vertical Parallelism

20

Benefits of Vertical ParallelismHigh number of attributes:

high level parallelism (e.g., documents)vs. task parallelism:

obvious parallelism observedvs. horizontal parallelism:

reduced memory usage (no model replication)parallelized split computation


21

Vertical Hoeffding Tree


22

Preliminary results: Dense instancesRandom decision treeMixed categorical and numerical attributes

10-10, 100-100, 1k-1k, 10k-10kInstances: 1,000,0002 balanced classes10 different seeded runsTest every 100k instancesMOA HT vs. Local VHT vs. Storm cluster VHT


23

Results: Accuracy


24

Results: Accuracy


25

Results: Accuracy Evolution


26

Results: Speedup


27

Results: Speedup


28

Preliminary results: Artificial TweetsZipf skew: 1.5Bag of words: 100, 1000, 10000 (attributes)Size of tweet: ~15 words Instances: 1,000,000Class: positive or negative

Gaussian random variable10 different seeded runsTest every 100k instancesMOA HT vs. Local VHT vs. Storm cluster VHT


29

Results: Accuracy


30

Results: Accuracy


31

Results: Accuracy Evolution


32

Results: Speedup


33

Results: Speedup


34

Is SAMOA for you?Are you dealing with:

Big fast data?Possibly endless streams of data?Evolving data?

Do you need updated models at real time?Do you want to test an algorithm on different DSPEs?


35

SAMOA Team

Albert Bifet

GianmarcoDe Francisci Morales

Nicolas Kourtellis

Matthieu Morel

Arinto Murdopo

Olivier Van Laere


36

Status Apache Incubator

Released version 0.3.0 in July Execution Engines

Input: Local FS HDFS Avro Kafka [pending]

Heron?


Apache Beam?

37

Algorithms in SAMOAExisting:

Vertical Hoeffding Tree (classification) CluStream (clustering) Adaptive Model Rules (regression)

Pending: Distributed Naïve Bayes Stochastic Gradient Descent Adaptive + Boosting VHT Parallelized Gradient Boosted Decision Tree PARMA (frequent pattern mining) …

Check Samoa Roadmap for more

Looking for contributors!


38

SAMOA: A Platform for Mining Big Data Streams

@ApacheSAMOAhttp://samoa.incubator.apache.org/

https://github.com/apache/incubator-samoa

Nicolas Kourtellis@kourtellis

[email protected]


http://samoa.incubator.apache.org/

http://samoa.incubator.apache.org/



mailto:[email protected]

Science

SAMOA: A Platform for Mining Big Data Streams (Apache BigData North America 2016)