1
SAMOA: A Platform for Mining Big Data Streams
Nicolas KourtellisAssociate Researcher
Telefonica I+D, Barcelona@kourtellis
@ApacheSAMOA
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
2
What is Big Data?Search queriesFacebook postsEmailsTweetsPhoto sharesClicks on ads…
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
3
How BIG is your data?Volume (+ Variety)
Too large for RAM of single commodity serverVelocity
Too fast for CPU of single commodity server
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
4
What is the Streaming Paradigm?High amount of data, high speed of arrivalUpdated models at “real” timePotentially infinite sequence of dataChange over time (concept drift)
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
5
Mining Big Data StreamsApproximation algorithms:
Single pass, one data item at a timeSub-linear space and time per data itemSmall error with high probability
A platform solution:Support different algorithms & processing enginesDistributedScalable
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
6
What is SAMOA?Scalable Advanced Massive Online AnalysisA platform for mining big data streams
Framework for developing new distributed stream mining algorithms
Framework for deploying algorithms on new distributed stream processing engines
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
7
Taxonomy
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
8
SAMOA ArchitectureMachine LearningAlgorithms
Distributed StreamProcessing Engines Flink
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
9
Why is SAMOA important?Program once, run everywhere
Reuse existing infrastructureAvoid deploy cycles
No system downtimeNo complex backup/update processNo need to select update frequency
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
10
ML Developer API
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
11
ML Developer API
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
12
Deployment
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
13
Easy to get!
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
14
Easy to get!
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
15
Easy to get!
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
16
Easy to test!bin/samoa storm target/SAMOA-Storm-0.3.0-SNAPSHOT.jar"PrequentialEvaluation-d /tmp/dump.csv-i 1000000 -f 100000-l (classifiers.trees.VerticalHoeffdingTree -p 4 -k)-s (generators.RandomTreeGenerator –r 1 -c 2 -o 10 -u 10)"
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
17
Case study: Decision TreesVHT: Vertical Hoeffding Tree*
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
Task parallelism
*VHT: Vertical Hoeffding Tree. N. Kourtellis, G. De Francisci Morales, A. Bifet, A. Mordupo. IEEE BigData 2016.
18
Case study: VHT
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
Horizontal Parallelism
19
Case study: VHT
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
Vertical Parallelism
20
Benefits of Vertical ParallelismHigh number of attributes:
high level parallelism (e.g., documents)vs. task parallelism:
obvious parallelism observedvs. horizontal parallelism:
reduced memory usage (no model replication)parallelized split computation
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
21
Vertical Hoeffding Tree
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
22
Preliminary results: Dense instancesRandom decision treeMixed categorical and numerical attributes
10-10, 100-100, 1k-1k, 10k-10kInstances: 1,000,0002 balanced classes10 different seeded runsTest every 100k instancesMOA HT vs. Local VHT vs. Storm cluster VHT
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
23
Results: Accuracy
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
24
Results: Accuracy
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
25
Results: Accuracy Evolution
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
26
Results: Speedup
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
27
Results: Speedup
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
28
Preliminary results: Artificial TweetsZipf skew: 1.5Bag of words: 100, 1000, 10000 (attributes)Size of tweet: ~15 words Instances: 1,000,000Class: positive or negative
Gaussian random variable10 different seeded runsTest every 100k instancesMOA HT vs. Local VHT vs. Storm cluster VHT
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
29
Results: Accuracy
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
30
Results: Accuracy
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
31
Results: Accuracy Evolution
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
32
Results: Speedup
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
33
Results: Speedup
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
34
Is SAMOA for you?Are you dealing with:
Big fast data?Possibly endless streams of data?Evolving data?
Do you need updated models at real time?Do you want to test an algorithm on different DSPEs?
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
35
SAMOA Team
Albert Bifet
GianmarcoDe Francisci Morales
Nicolas Kourtellis
Matthieu Morel
Arinto Murdopo
Olivier Van Laere
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
36
Status Apache Incubator
Released version 0.3.0 in July Execution Engines
Input: Local FS HDFS Avro Kafka [pending]
Heron?
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
Apache Beam?
37
Algorithms in SAMOAExisting:
Vertical Hoeffding Tree (classification) CluStream (clustering) Adaptive Model Rules (regression)
Pending: Distributed Naïve Bayes Stochastic Gradient Descent Adaptive + Boosting VHT Parallelized Gradient Boosted Decision Tree PARMA (frequent pattern mining) …
Check Samoa Roadmap for more
Looking for contributors!
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016
38
SAMOA: A Platform for Mining Big Data Streams
@ApacheSAMOAhttp://samoa.incubator.apache.org/
https://github.com/apache/incubator-samoa
Nicolas Kourtellis@kourtellis
SAMOA: Scalable Advanced Machine Online Analysis, ApacheCon BigData, NA, 2016