Department of Computer Science, University of Waikato, New Zealand
Geoffrey Holmes, Bernhard Pfahringer and Richard Kirkby
Traditional machine learning model
Data stream requirements
Scaling up existing methods
Network caching CHIC architecture Experimental Design Results Discussion Further Work
Cache Hierarchy Inspired
Compression: A Novel Architecture for Data Streams
2
Traditional Machine Learning Create train/test splits of the data (possibly via
cross-validation) Load ALL the training data into main memory Compute a model from the training data (may
involve multiple passes) Load all the test data into main memory Compute accuracy measures from the test data
3
Consequences Data is processed one instance at a time Very few incremental methods – none used
seriously in practice. Many existing techniques don’t scale Machine Learning is perceived as a small to
medium data set field of study. Larger data sets are tackled through sampling or
building several models on separate portions and combining their predictions
4
Data Streams Takes the “stream” view, source maybe finite or
infinite Concept of train/test less well defined, could train
for a while then test for a while – what is the definition of “a while”?
What ever you do you can be sure that ALL the data will NOT fit in main memory
5
Data Stream Constraints Cannot store instances (not all anyway) Cannot use more than available memory – no
swapping to disk Cannot dwell too long over an instance - must
keep up with the incoming rate of instances Cannot wait to make predictions – need to be
ready to make predictions at any time
6
Scaling up existing methods Could learn models using existing methods in
batches and then merge models Could merge instances (meta-instances) Could use a cache model where we keep a set of
models and update the cache in time – eg use least recently used, least frequently used type strategies
Could do the above but use performance measures to decide the make-up of the cache.
7
Caching in Data Communications
Web proxy caches provide a good model for what we need to satisfy stream constraints
Real caches are hierarchical (Squid) The hierarchy provides a mechanism for sharing
the load and increasing the likelihood of a page hit When full a cache needs a replacement policy To replicate this system we need to design a
hierarchy, fill it (with models) and implement a model replacement policy
8
General CHIC Architecture Idea: Build a hierarchy of levels (N) as follows:
Level Zero: data buffer from stream Level One: Build models from data at level zero Level Two to N-1: Fill with “best” models from lower
levels Level N: Adopt models from level N-1 but also
discard models so that new ones can be entered For prediction use all models in hierarchy and vote
9
Features Can use any machine learning algorithm to build
models Implements a form of batch-incremental learning Replacement policy can be performance based As with the web cache CHIC fills up initially and
then keeps the best performing models If a variety of models is used at the lower levels
then it is possible to adapt to the source of the data.
10
Experimental Design Try to demonstrate adaptation to data source
Learn a mixture of models at levels one and two and let the performance based promotion mechanism take over
Evaluate: two issues Need performance measure (model
promotion/deletion) Overall performance of hierarchy (adopt a first test
then train approach)
11
Data Sources Random Naïve Bayes
Weight attribute labels and class values (here we use 5 classes, 5 nominal and 5 numeric attributes)
Random Tree Choose a depth and randomly assign splitter nodes, here 5
nodes deep, leaves starting from depth 3, as above number of attributes/classes.
Random RBF Random set of centers for each class, center is weighted and
has a standard deviation – here 50 centers, 10 numeric attributes and 5 classes
Real data Forest covertype (UCI repository), 7 classes, 500K instances.
12
Specific CHIC Architecture Six levels with 16 models per level Data buffer of size 1000 instances First level uses four algorithms to generate
models Naïve Bayes(N), C4.5(J), linear SVM(L), RBF
Kernel SVM(R)
13
Example Read 1000 instances into buffer, build 4 models, repeat on
next 3 buffers – leads to level one full Read next 1000 and build 4 more models Using the buffer data as test data evaluate all models at
level 1 – promote best 4 (groupwise) to level 2 and replace the worst 4 with the new ones.
Continue in this manner (at 12,000 instances level 2 will be full)
Note: Levels 1&2 always have 4 models from the 4 groups From level 2 ONLY promote the best (adapt to source)
14
Example Contd Four models are promoted to level 3 the 4 worst
deleted freeing 8 spaces. Levels 3, 4 & 5 work on the same basis. Level 6
simply deletes the worst 4 models to free up space.
At prediction time all models have an equal vote and the classification decision is arrived at by majority.
15
Resultsafter 1000 instances: after 2000 instances:N---J---L---R--- NN--JJ--LL--RR-- after 3000 instances: after 4000 instances:NNN-JJJ-LLL-RRR- NNNNJJJJLLLLRRRR after 5000 instances: after 6000 instances:N-NNJJ-JL-LLR-RR NNNNJJJJLLLLRRRRNJLR------------ NJLR------------ after 7000 instances: after 8000 instances:NNN-J-JJL-LLR-RR NNNNJJJJLLLLRRRRNJLRNJLR-------- NJLRNJLR--------
16
Continuedafter 9000 instances: after 10000 instances:NN-N-JJJLL-L-RRR NNNNJJJJLLLLRRRRNJLRNJLRNJLR---- NJLRNJLRNJLR---- after 11000 instances: after 12000 instances:-NNN-JJJLLL-RR-R NNNNJJJJLLLLRRRRNJLRNJLRNJLRNJLR NJLRNJLRNJLRNJLR after 13000 instances:NN-NJJ-JLL-LRR-RNNLJNLLRN-L-N-L-JJJJ------------
17
Model Adaptation to Sourcerandom tree random naive Bayes
N-NN-JJJLL-LR-RR -NNNJJ-JLLL-R-RR
NNNNLNNLJLRNNJLR NJJJLLLLRNJJJLLR
JJJJJNNLJJ---JL- NNNNNJJNJJ-NN---
JJJJJJJJJJJJJJJJ NNNNNNNNNNNNNNNN
JJJJ--JJ-JJJJJ-J NNNNNNNN---NNNN-
JJJJJJJJJJJJJJJJ NNNNNNNNNNNNNNNN
18
Continuedrandom RBF covertype
NNN--JJJLL-LRR-R N-NN-JJJ-LLLR-RR
NNJJLRNJNNJJNJLR NNJJJRJLNRJ---R
RRJRJJRRRRJRRJJJ LJJLLJLJLJL--N--
RRRRRRRRRRRRRRRR LLLLLLLLLLLLJLLJ
RRRRRRRRRRRRRRRR LLLLLLLLJL-J-L--
RRRRRRRRRRRRRRRR LLLLLLLLLLLLLLLL
19
Learning Curve – Random Tree
20
Learning Curve - CoverType
21
Conclusion Novel architecture for data streams Operates much like a web cache (hierarchy and
replacement policy) Provides scaling-up mechanism for arbitrary
classifiers (batch-incremental) Can be adapted to clustering, regression,
association rule learning Thousands of options still to explore!