Ensemble  Methods    

Adapted  from  slides  by  Todd  Holloway  h8p://abeau<fulwww.com/2007/11/23/ensemble-­‐machine-­‐learning-­‐tutorial/  

Outline  

•  The  NeHlix  Prize  – Success  of  ensemble  methods  in  the  NeHlix  Prize  

•  Why  Ensemble  Methods  Work  •  Algorithms  

– Bagging  – Random  forests  – AdaBoost  

Defini<on  

 Ensemble  Classifica.on  Aggrega<on  of  predic<ons  of  mul<ple  classifiers  with  the  goal  of  improving  accuracy.  

Teaser:    How  good  are  ensemble  methods?  

Let’s  look  at  the  Ne,lix  Prize  Compe55on…  

•  Supervised  learning  task  –  Training  data  is  a  set  of  users  and  ra<ngs  (1,2,3,4,5  stars)  those  users  have  given  to  movies.  

–  Construct  a  classifier  that  given  a  user  and  an  unrated  movie,  correctly  classifies  that  movie  as  either  1,  2,  3,  4,  or  5  stars  

•  $1  million  prize  for  a  10%  improvement  over  NeHlix’s  current  movie  recommender/classifier      (MSE  =  0.9514)  

Began  October  2006  

Just  three  weeks  a`er  it  began,  at  least  40  teams  had  bested  the  NeHlix  classifier.      

 Top  teams  showed  about  5%  improvement.  

from  h8p://www.research.a8.com/~volinsky/neHlix/  

However,  improvement  slowed…  

Today,  the  top  team  has  posted  a  8.5%  improvement.    Ensemble  methods  are  the  best  performers…  

“Thanks  to  Paul  Harrison's  collabora<on,  a  simple  mix  of  our  solu<ons  improved  our  result  from  6.31  to  6.75”  

Rookies  

“My  approach  is  to  combine  the  results  of  many  methods  (also  two-­‐way  interac<ons  between  them)  using  linear  regression  on  the  test  set.  The  best  method  in  my  ensemble  is  regularized  SVD  with  biases,  post  processed  with  kernel  ridge  regression”  

Arek  Paterek  

h8p://rainbow.mimuw.edu.pl/~ap/ap_kdd.pdf  

“When  the  predic<ons  of  mul.ple  RBM  models  and  mul.ple  SVD  models  are  linearly  combined,  we  achieve  an  error  rate  that  is  well  over  6%  be8er  than  the  score  of  NeHlix’s  own  system.”  

U  of  Toronto  

h8p://www.cs.toronto.edu/~rsalakhu/papers/rbmcf.pdf  

Gravity  

home.mit.bme.hu/~gtakacs/download/gravity.pdf    

“Our  common  team  blends  the  result  of  team  Gravity  and  team  Dinosaur  Planet.”    Might  have  guessed  from  the  name…  

When  Gravity  and    Dinosaurs  Unite  

And,  yes,  the  top  team  which  is  from  AT&T…    “Our  final  solu.on  (RMSE=0.8712)  consists  of  blending  107  individual  results.  “  

BellKor  /  KorBell  

The winner was an ensemble of ensembles (including BellKor).

Some  Intui<ons  on  Why  Ensemble  Methods  Work…  

Intui<ons  

 •  U<lity  of  combining  diverse,  independent  opinions  in  human  decision-­‐making  – Protec<ve  Mechanism  (e.g.  stock  porHolio  diversity)  

•  Viola.on  of  Ockham’s  Razor  –  Iden<fying  the  best  model  requires  iden<fying  the  proper  "model  complexity"  

See  Domingos,  P.  Occam’s  two  razors:  the  sharp  and  the  blunt.    KDD.    1998.  

Majority  vote  Suppose  we  have  5  completely  independent  classifiers…  –  If  accuracy  is  70%  for  each  

•  10  (.7^3)(.3^2)+5(.7^4)(.3)+(.7^5)    •  83.7%  majority  vote  accuracy  

– 101  such  classifiers  •  99.9%  majority  vote  accuracy  

Intui<ons  

Strategies  

Bagging    – Use  different  samples  of  observa<ons  and/or  predictors  (features)  of  the  examples  to  generate  diverse  classifiers  

– Aggregate  classifiers:  average  in  regression,  majority  vote  in  classifica<on  

Boos<ng  – Make  examples  currently  misclassified  more  important  (or  less,  in  some  cases)  

 

Bagging  (Construc<ng  for  Diversity)  

1. Use  random  samples  of  the  examples  to  construct  the  classifiers  

2. Use  random  feature  sets  to  construct  the  classifiers  •  Random  Decision  Forests  

•  Bagging:  Bootstrap  Aggrega<on  

Leo  Breiman  

•  Bootstrap:  consider  the  following  situa<on:  – A  random  sample                                                                                from  unknown  probability  distribu<on  F  

– We  wish  to  es<mate  parameter    – We  build  es<mate    – What  is  the  s.d.  of        ?    

x = (x1, . . . , xN )

θ = t(F )θ = s(x)

θ

Examples: 1) estimate mean and sd of expected prediction error 2) estimate point-wise confidence bands in smoothing

•  Bootstrap:  –  It  is  completely  automa<c  – Requires  no  theore<cal  calcula<ons  – Not  based  on  asympto<c  results  – Available  regardless  of  how  complicated  the  es<mator    θ

•  Bootstrap  algorithm:  1.  Select  B  independent  bootstrap  samples                  

each  consis<ng  of  N  data  values  drawing  with  replacement  from  

2.  Evaluate  the  bootstrap  replica<on  corresponding  to  each  bootstrap  sample  

3.  Es<mate  the  standard  error  of          using  the  sample  standard  error  of  the  B  es<mates    

     

x∗1, . . . ,x∗B

x

θ∗(b) = s(x∗b), b = 1, . . . , B

θ

•  Bagging:  use  bootstrap  to  improve  predic5ons  1.  Create  bootstrap  samples,  es<mate  model  from  

each  bootstrap  sample  2.  Aggregate  predic<ons  (average  if  regression,  

majority  vote  if  classifica<on)  

•  This  works  best  when  perturbing  the  training  set  can  cause  significant  changes  in  the  es<mated  model  

•  For  instance,  for  least-­‐squares,  can  show  variance  is  decreased  while  bias  is  unchanged  

Random  forests  

•  At  every  level,  choose  a  random  subset  of  the  variables  (predictors,  not  examples)  and  choose  the  best  split  among  those  a8ributes  

Breiman, L. (2001). Random forests. Machine learning, 45(1), 5-32.

Random  forests  

•  Let  the  number  of  training  points  be  M,  and  the  number  of  variables  in  the  classifier  be  N.  

For  each  tree,  1.  Choose  a  training  set  by  choosing  N  <mes  with  

replacement  from  all  N  available  training  cases.    2.  For  each  node,  randomly  choose  m  variables  on  

which  to  base  the  decision  at  that  node.    

Calculate  the  best  split  based  on  these.  

Random  forests  

•  Grow  each  tree  as  deep  as  possible  no  pruning!  

•  Out-­‐of-­‐bag  data  can  be  used  to  es<mate  cross-­‐valida<on  error  – For  each  training  point,  get  predic<on  from  averaging  trees  where  point  is  not  included  in  bootstrap  sample  

•  Variable  importance  measures  are  easy  to  calculate  

Boos<ng  1.  Create  a  sequence  of  classifiers,  giving  higher  influence  to  

more  accurate  classifiers  2.  At  each  itera<on,  make  examples  currently  misclassified  

more  important  (get  larger  weight  in  the  construc<on  of  the  next  classifier).      

3.  Then  combine  classifiers  by  weighted  vote  (weight  given  by  classifier  accuracy)  

AdaBoost  Algorithm  1.  Ini<alize  Weights:  each  case  gets  the  same  weight:    

2.  Construct  a  classifier  using  current  weights.    Compute  its  error:  

3.  Get  classifier  influence,  and  update  example  weights  

4.  Goto  step  2…  

wi = 1/N, i = 1, . . . , N

εm =�

i wi × I{yi �= gm(xi)}�i wi

αm = log�

1− εm

εm

�wi ← wi × exp {αmI{yi �= gm(xi)}}

Final  predic<on  is  weighted  vote,  with  weight      αm

AdaBoost  Algorithm  

Dra

ft:D

oN

otD

istrib

ute

152 a course in machine learning

Algorithm 31 AdaBoost(W , D, K)1: d(0) ← � 1

N , 1N , . . . , 1

N � // Initialize uniform importance to each example2: for k = 1 . . . K do3: f (k) ← W(D, d(k-1)) // Train kth classifier on weighted data4: yn ← f (k)(xn), ∀n // Make predictions on training data5: �(k) ← ∑n d(k-1)

n [yn �= yn] // Compute weighted training error

6: α(k) ← 12 log

�1−�(k)

�(k)

�// Compute “adaptive” parameter

7: d(k)n ← 1

Z d(k-1)n exp[−α(k)ynyn], ∀n // Re-weight examples and normalize

8: end for9: return f (x) = sgn

�∑k α(k) f (k)(x)

�// Return (weighted) voted classifier

questions that you got right, you pay less attention to. Those that yougot wrong, you study more. Then you take the exam again and repeatthis process. You continually down-weight the importance of questionsyou routinely answer correctly and up-weight the importance of ques-tions you routinely answer incorrectly. After going over the exammultiple times, you hope to have mastered everything.

The precise AdaBoost training algorithm is shown in Algorithm 11.2.The basic functioning of the algorithm is to maintain a weight dis-tribution d, over data points. A weak learner, f (k) is trained on thisweighted data. (Note that we implicitly assume that our weak learnercan accept weighted training data, a relatively mild assumption thatis nearly always true.) The (weighted) error rate of f (k) is used to de-termine the adaptive parameter α, which controls how “important” f (k)

is. As long as the weak learner does, indeed, achieve < 50% error,then α will be greater than zero. As the error drops to zero, α growswithout bound. What happens if the weak learn-

ing assumption is violated and � isequal to 50%? What if it is worsethan 50%? What does this mean, inpractice?

After the adaptive parameter is computed, the weight distibutionis updated for the next iteration. As desired, examples that are cor-rectly classified (for which ynyn = +1) have their weight decreasedmultiplicatively. Examples that are incorrectly classified (ynyn = −1)have their weight increased multiplicatively. The Z term is a nom-ralization constant to ensure that the sum of d is one (i.e., d can beinterpreted as a distribution). The final classifier returned by Ad-aBoost is a weighted vote of the individual classifiers, with weightsgiven by the adaptive parameters.

To better understand why α is defined as it is, suppose that ourweak learner simply returns a constant function that returns the(weighted) majority class. So if the total weight of positive exam-ples exceeds that of negative examples, f (x) = +1 for all x; otherwisef (x) = −1 for all x. To make the problem moderately interesting,suppose that in the original training set, there are 80 positive ex-amples and 20 negative examples. In this case, f (1)(x) = +1. It’sweighted error rate will be �(1) = 0.2 because it gets every negative

Classifica.ons  (colors)  and    Weights  (size)  aYer  1  itera+on  Of  AdaBoost  

3  itera+ons  20  itera+ons  

from  Elder,  John.    From  Trees  to  Forests  and  Rule  Sets  -­‐  A  Unified  Overview  of  Ensemble  Methods.    2007.  

AdaBoost  

•  Advantages  – Very  li8le  code  – Reduces  variance  

•  Disadvantages  – Sensi<ve  to  noise  and  outliers.    Why?    

How  was  the  NeHlix  prize  won?  

Gradient boosted decision trees… Details: http://www.netflixprize.com/assets/GrandPrize2009_BPC_BellKor.pdf

Sources  •  David  Mease.    Sta<s<cal  Aspects  of  Data  Mining.    Lecture.  

h8p://video.google.com/videoplay?docid=-­‐4669216290304603251&q=stats+202+engEDU&total=13&start=0&num=10&so=0&type=search&plindex=8  

•  Die8erich,  T.  G.    Ensemble  Learning.  In  The  Handbook  of  Brain  Theory  and  Neural  Networks,  Second  edi<on,  (M.A.  Arbib,  Ed.),  Cambridge,  MA:  The  MIT  Press,  2002.  h8p://www.cs.orst.edu/~tgd/publica<ons/hbtnn-­‐ensemble-­‐learning.ps.gz  

•  Elder,  John  and  Seni  Giovanni.    From  Trees  to  Forests  and  Rule  Sets  -­‐  A  Unified  Overview  of  Ensemble  Methods.    KDD  2007  h8p://Tutorial.  videolectures.net/kdd07_elder_`fr/  

•  NeHlix  Prize.  h8p://www.neHlixprize.com/  •  Christopher  M.  Bishop.    Neural  Networks  for  Pa8ern  Recogni<on.    Oxford  University  Press.    1995.