18
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ���

��

Additive Logistic Regression

a Statistical View of Boosting

Jerome Friedman� Trevor Hastie� Rob Tibshirani

Stanford University

Thanks to Bogdan Popescu for helpful and very lively

discussions on the history of boosting� and for help in

preparing that part of this talk

Email� trevor�stat�stanford�edu

Ftp� stat�stanford�edu� pub�hastie

WWW� http���www�stat�stanford�edu��trevor

These transparencies are available via ftp�

ftp���stat�stanford�edu�pub�hastie�boost���ps

Stanford University December �� ���� Boosting� ���

��

Classi�cation Problem

0

00

0

00

0

0

00

0

0

0

0

0

0

0

0

0 0 0

00

0

00

00

00

0

0

00

00

00

0 0

0

0

00

0

0

0 0

00

0

0

0

000

0

0 0

00

00

00 00 0000

0

0

0

0

0

0

0

00

00

0

0

00

00

0

0

0

000

00

00

0

000 00

0

0

0

0

0

00

00

0

00

0

00

00

0

0

0

0

0

0

0

0

00

00

0

0

0

0

00

0

0

0 00

0

0 00

00

00

0

0

0

0

0 0

0

0

000

0

0

0

0

00

0

0

000

0

00

0

0

0

0 00

0

00 00 0

0 0

000

00

0

0000 0

0

0

0

0

00 0

000

0

0

0

0

00

0

00

0

0

0

0

0

0

0

0 00

0

00

00

0

0

0

0 0

00

00

0 0

0

0

0

00

0

00

0

0

000

0

0

0

00

0

00 00

0

00

0

0

0

0

0

0

0

00

0

0

0

0

00

000

0

0

00

00

0

00

0

0

0 0

0

0

0

0

0

0

00

0

0

0

0

0

00

00

0

0

00

00 0

0

0

0

0 0

0

0

00

000

0

0 0

00

00

0 0

0 0

0

0

0

0

00

0

0

0

0

000

0

00

0 0

0

0

00

00

0 0

0

0

0

0

0

0

0

0

0

0 00

00

0

00

0 0

00

00

1

1

1

11

1

1

1

11

1

111

1

1

11

1

1

1

1

1

1

1

1

1 1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1

1

1

1

11

1

1

1

11

1

1

1

1

1

11

1

1

11

1

1 11

1

1

1

1

1

1

1

1

1

1

1

11 1

1

1

1

1

1

1

1

1

11

1

1

1

1

11

1

11

1

1

11

1

1

11

11

11

1

11

11

1

1

1

11

1

1

1

1

11

1

1

1

1

11

1

1

1

1

1

1

1

1

11

1

111

1

1

1

11

11

11

1

1

1

11

1

1

1

1

1

1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1 1

11

1

1

1

1

1

1

11

111

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

11

1

11

11

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1

1

1 1

1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1

1

1

1

1

11

11

1

1

1

1

11

11

1

1

1

1

1

11

1

1

1

11

1

1

1 1

1

1

1

1

1

1

1

1

11

1

1

1 1

1

1

11

1

1

1

1

1 1

1

11

1

1

1

1

1

1

1

1

11

1

11

1

Data �X�Y � � Rp � f�� �g�

X is predictor� feature� Y is class label� response�

�X�Y � have joint probability distribution D�

Goal� Based on N training pairs �Xi� Yi� drawn

from D produce a classier C�X� � f�� �g

Goal� choose C to have low generalization error

R� C� � PD� C�X� �� Y �

� ED��� C�X��Y

Page 2: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ���

��

Deterministic Concepts

0

0

00

0 0

0

0 0

0

0

0

0

0

0

00

0

0

0

0

0

0

0

0

0

0

0

0

00

0

0

0

000

0

0

0

0

0

0

0

0

0

00

00 0

0

00

0

0

00

0

0

0

0

0

0

0

0

0

0

0

0

0

0 0

00

00

0

0

0

0

0

0

00

0

0

0

0

00

0

0

000

0

0

00

0

0

0

0 0

00

0

0 00

0

0

0

00

00 0

0

0

00

0

0

0

0

0

00

0

00

00 0

0

000

0

0

0

00

0 00

0

0

0

0

0

0

0

0

0

00

0

0 0

0

0

0

00

0

00

00

0

0

00

0

0

0

0

00

0 0

0

00

0

0

0

0 0

0

0

0

0

00

0

000

000

0

00 0

0

0

0

0 00

0

0

0

00

00

0

0

0

0

0

00

0

00

0

00

0

0

00

00

0

0 0

0

0

0

0

0

0

00

0

00

00

00

0

0

00

0

00

0

0

00

0

00

0 0

0

0

0

0

0

0

0

0

0

0

0

00

00

0

0

0

0

0

00

0

0 00

00

0

0

0

0

0

00

00

0

0

00

0

00

00

0

0

00

00

0

000 0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0 00

0

0

00

00 00

00

00

00

0

0

0

0

0

0

0

0 0

0

0

0

0

00

0

00

0

000

00

00

0

0

0

1

1

1

1

1

1

11

11

11

1

1

1

1

1

1

1

11

1

11

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1 1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

11

11

1

1 1

1

1

1

1

1

1

1

11

1

1

1

1

11

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1 1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1 1

1

1

1

1

1 1

11

1

1

1

1

1

1

1

1

1

1

1

1

1 1

1

1

1

1

1

1

1

1

1

1

1

11

1 1

1

1

1

1

1

1

11

1

1

1

1

1

1

1

1

1

1

1 1

1

11

1

1

1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1 1

1

1

1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1 11

1

1

11

11

1

11

1

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1

1

11

1

11

1

1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

1 11 1

1

1

1

1

1

1

1

11

1

1

1

1

11

11

1

1

11

1

1 1

1

1

1

1

1

1

11

1

1

1

1

11

1

1

X � Rp has distribution D�

C�X� is deterministic function � concept class�

Goal� Based on N training pairs

�Xi� Yi � C�Xi�� drawn from D produce a

classier C�X� � f�� �g

Goal� choose C to have low generalization error

R� C� � PD� C�X� �� C�X��

� ED��� C�X��C�X

Stanford University December �� ���� Boosting� ��

��

Classi�cation Trees

x.2<-1.06711x.2>-1.06711

94/200

1

0/34

1

x.2<1.14988x.2>1.14988

72/166

0

x.1<1.13632x.1>1.13632

40/134

0

x.1<-0.900735x.1>-0.900735

23/117

0

x.1<-1.1668x.1>-1.1668

5/26

1

0/12

1

x.1<-1.07831x.1>-1.07831

5/14

1

1/5

1

4/9

1

x.2<-0.823968x.2>-0.823968

2/91

0

2/8

0

0/83

0

0/17

1

0/32

1

Page 3: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ���

��

Decision Boundary� Tree

1

1

11

1

11

1

1

1

11

1

1

1

1

11

1

1

1

1

1

11

1

11

1

11

1

1

1

1

1

1 1

1

1

1

1

1

1

1

1

1

1

1

1

1

1 1

1

1

1

1

1

1

1

111

1

1

1 1

1

1

1

1

11 1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

11

1

1

1

1

1

1

0

0

0

0

0

0

0

00

0

0

0

0

0

0

00

0

0

0

0

0

0

0

0

0

0

0

0

000

0

00

00

0

0

0

00

0

0

0

0

0

0

00

00

0

0

0

0

0

00

0

0

0

0

00

0 0

0

0

0

0

0 0

00

0

0

0

0

0

0

00

0

00

0

0

0 0

0

0

0

00

0

0

0

00

0

When the nested spheres are in R��� CARTTM

produces a rather noisy and inaccurate rule

C�X�� with error rates around ����

Stanford University December �� ���� Boosting� ���

��

Bagging and Boosting

Classication trees can be simple� but often

produce noisy �bushy� or weak �stunted�

classiers�

� Bagging �Breiman� ������ Fit many large

trees to bootstrap�resampled versions of the

training data� and classify by majority vote�

� Boosting �Freund � Shapire� ������ Fit many

large or small trees to reweighted versions of

the training data� Classify by weighted

majority vote�

In general Boosting � Bagging � Single Tree�

�AdaBoost � � � best o��the�shelf classier in the

world� � Leo Breiman� NIPS workshop� �����

Page 4: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ���

��

training sample

weighted sample

weighted sample

weighted sample

f1(x)

f2(x)

f3(x)

fB(x)

sign[Σαbfb(x)]

Final classifier

The weighting in boosting can be achieved by

weighted importance sampling�

Stanford University December �� ���� Boosting� ���

��

Bagging and Boosting

���� points from Nested Spheres in R��� Bayes

error rate is ���

Trees are grown Best First without pruning�

Leftmost iteration is a single tree�

Page 5: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ���

��

Decision Boundary� Boosting

1

1

11

1

11

1

1

1

11

1

1

1

1

11

1

1

1

1

1

11

1

11

1

11

1

1

1

1

1

1 1

1

1

1

1

1

1

1

1

1

1

1

1

1

1 1

1

1

1

1

1

1

1

111

1

1

1 1

1

1

1

1

11 1

1

1

1

1

1

1

1

1

1

1

1

11

1

1

1

1

11

1

1

1

1

1

1

0

0

0

0

0

0

0

00

0

0

0

0

0

0

00

0

0

0

0

0

0

0

0

0

0

0

0

000

0

00

00

0

0

0

00

0

0

0

0

0

0

00

00

0

0

0

0

0

00

0

0

0

0

00

0 0

0

0

0

0

0 0

00

0

0

0

0

0

0

00

0

00

0

0

0 0

0

0

0

00

0

0

0

00

0

Bagging and Boosting average many trees� and

produce smoother decision boundaries�

Stanford University December �� ���� Boosting� ����

��

AdaBoost �Freund � Schapire� ����

�� Start with weights wi � ��N �i � �� � � � � N �

yi � f��� �g�

�� Repeat for m � �� �� � � � �M �

�a� Estimate the weak learner fm�x� � f��� �g

from the training data with weights wi�

�b� Compute em � Ew���y �� fm�x� �

cm � log���� em��em��

�c� Set wi � wi exp�cm � ��yi �� fm�xi� � i �

�� �� � � �N � and renormalize so thatPi wi � ��

�� Output the majority weight classier

C�x� � sign�PM

m�� cmfm�x� �

Page 6: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

His

tory

of

Boost

ing

Boo

stin

g

Bag

ging

L. B

reim

an(1

996)

Atte

mpt

s to

Scha

pire

, Y. S

inge

r (1

998)

Scha

pire

, Fre

und,

P. B

artl

ett,

Lee

(19

97)

Con

cept

of

with

Exp

erim

ents

Impr

ovem

ents

expl

ain

why

Ada

boos

t wor

ks

Ada

boos

t

PA

C L

earn

ing

Mod

el

Scha

pire

& F

reun

d (1

995,

1997

)

appe

ars

Ada

boos

tis

bor

n

Gen

esis

of

Fre

und

& S

chap

ire

(199

6)B

reim

an(1

996,

199

7)R

. Qui

nlan

(19

96)

L.G

. Val

iant

(19

84)

Y. F

reun

d (1

995)

R.E

Sch

apir

e (1

990)

Fri

edm

an, H

asti

e, T

ibsh

iran

i (19

98)

Scha

pire

, Sin

ger,

Fre

und,

Iye

r (1

998)

Stanford University December �� ���� Boosting� ����

��

PAC Learning Model

X D� Instance Space

C � X � f�� �g Concept � C

h � X � f�� �g Hypothesis � H

error�h� � PD�C�X� �� h�X�

hC

X

Denition� Consider a concept class C dened

over a set X of length N � L is a learner

�algorithm� using hypothesis space H� C is PAC

learn�able by L using H if for all C � C� all

distributions D over X and all �� � � ��� �� ��

learner L will� with Pr � ��� ��� output an

h � H s�t� errorD�h� � in time polynomial in

���� ����N and size�C��

Such an L is called a strong Learner�

Page 7: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Boosting a Weak Learner

Weak learner L produces an h with error rate

� � � �� � �� �� � with Pr � ��� �� for any D�

L has access to continuous stream of training

data and a class oracle�

�� L learns h� on rst N training points�

�� L randomly lters the next batch of training

points� extracting N�� points correctly

classied by h�� N�� incorrectly classied�

and produces h��

�� L builds a third training set of N points for

which h� and h� disagree� and produces h��

�� L outputs h � Majority Vote�h�� h�� h��

THEOREM �Schapire� ������ �The Strength of

Weak Learnability�

errorD�h� ��� � ��� �

Stanford University December �� ���� Boosting� � ��

��

Boosting � Training Error

Nested spheres in R�� � Bayes error is ���

Boosting drives the training error to zero� Further

iterations continue to improve test error in many

examples�

Page 8: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Boosting Noisy Problems

Nested Gaussians in R�� � Bayes error is ����

Here the test error does increase� but quite slowly�

Stanford University December �� ���� Boosting� ����

��

Bagging and Boosting Smaller Trees

���� points in R��� Bayes error rate is ���

Each tree has �� terminal nodes� grown best�rst�

Bagging�Boosting gap is wider

Page 9: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Bagging and Boosting Stumps

���� points in R��� Bayes error rate is ���

Each tree has � terminal nodes� grown best�rst�

Bagging fails � boosting does best ever�

Stanford University December �� ���� Boosting� ����

��

Prediction games

Results of Freund and Schapire ������ and

Breiman �������

Idea�� Start with xed learners

f��x�� f��x� � � � fM �x��

� Play a two person game� player � picks

observation weights wi� player � picks learner

weights cm �i�e� he will use learner fm�x�

with probability cm��

� Player � tries to make the prediction problem

as hard as possible� while Player � does the

best he can on the weighted problem� We

judge di�culty by the approximate �margin��

a smooth version of misclassication loss�

Page 10: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

� Result� this is a zero�sum game� and the

minimax theorem gives the best strategy for

each player� Furthermore� AdaBoost

converges to this optimal strategy�

� However� link with statistical properties of

actual AdaBoost is tenuous�

�� why minimize hardest weighted problem

�makes test error smaller���

�� actual AdaBoost does not use a random

choice of learner�

�� actual AdaBoost nds the learners fm�x��

Stanford University December �� ���� Boosting� ����

��

Stagewise Additive Modeling

Boosting builds an additive model

F �x� �PM

m�� fm�x� and then C�x� � sign�F �x� �

We do things like that in statistics�

� GAMs� F �x� �P

j fj�xj�

� Basis expansions� F �x� �PM

m�� mhm�x�

Traditionally each of the terms fm�x� is di�erent

in nature� and they are t jointly �i�e� least

squares� maximum likelihood�

With Boosting� each term is equivalent in nature�

and they are t in a stagewise fashion�

Simple example� stagewise least�squares� Fix the

past M � � functions� and update the Mth using

a tree�min

fM�Tree�xE�Y �

M��Xm��

fm�x�� fM �x���

Page 11: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Boosting and Additive Models

� Discrete AdaBoost builds an additive model

F �x� �

MXm��

cmfm�x�

by stage�wise optimization of

J�F �x�� � Ee�y�F �x

� Given an imperfect FM���x�� the updates in

Discrete AdaBoost correspond to a Newton

step towards minimizing

J�FM���x��cMfM �x�� � Ee�y�FM���x�cMfM �x

over fM �x� � f��� �g� with step length cM �

� Ee�y�F �x is minimized at

F �x� ��

�log

P �y � �jx�

P �y � ��jx�

Hence Adaboost is tting an additive logistic

regression model�

Stanford University December �� ���� Boosting� ����

��

Details

fM �

fM �x� � arg min

g�x�f����gEw�y � g�x���

with weights w�x� y� � e�yFM���x�

cM �

cM � argminc

Ewe�cyfM �x

��

log�� e

e

with e � Ew���y ��fM �x� �

� Empirical version� at each stage� fM �x� is

estimated by the classication at the terminal

nodes of a tree� grown to appropriately

weighted versions of the training data�

� At the Mth stage of the Discrete AdaBoost

iterations� the weights are such that fM��

has weighted training error � ����

Page 12: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Real AdaBoost

�� Start with weights wi � ��N � i � �� �� � � � � N �

yi � f��� �g�

�� Repeat for m � �� �� � � � �M �

�a� Fit the class probability estimate

pm�x� � Pw�y � �jx� � ��� � using weights

wi on the training data�

�b� Set fm�x�� �� log

pm�x

��pm�x � R�

�c� Setwi � wi exp��yifm�xi� � i � �� �� � � �N �

and renormalize so thatP

i wi � ��

�� Output the classier sign�PM

m�� fm�x�

Stanford University December �� ���� Boosting� � ��

��

Real AdaBoost

� Real AdaBoost also builds an additive logistic

regression model�

logP �y � �jx�

P �y � ��jx��

MXm��

fm�x�

by stage�wise optimization of

J�F �x�� � Ee�y�F �x�

� Given an imperfect FM���x�� Real AdaBoost

minimizes

J�FM���x� � fM �x�� � Ee�y�FM���x�fM �x

over fM �x� � R� with solution

fM �x� ��

�log

Pw�y � ��

Pw�y � ���

where the weights w�x� y� � e�yFM���x�

� Empirical version� at each stage� Pw��jx� is

estimated by averages at the terminal nodes

of a tree� grown to appropriately weighted

versions of the training data�

Page 13: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Why J�F �x�� � Ee�yF �x��

� e�yF �x is a monotone� smooth upper bound

on misclassication loss at x�

� J�F � is an expected � statistic at its

minimum� and equivalent to the binomial

log�likelihood to second order�

� Stage�wise binomial maximum�likelihood

estimation of additive models based on trees

works as least as well�

Stanford University December �� ���� Boosting� ����

��

Stagewise Maximum Likelihood

Consider the model

FM �x� � logP �y� � �jx�

P �y� � �jx��

MXm��

fm�x�

or

P �y� � �jx� � p�x� �

eF �x

� � eF �x

The binomial log�likelihood is

��F �x�� � E�y� log�p�x�� � ��� y�� log��� p�x��

� E�y�F �x�� log�� � eF �x��

Stagewise Maximum Likelihood� Given an

imperfect FM���x�� maximize

��FM���x� � fM �x�� over fM �x� � R

The LogitBoost algorithm takes a single

Newton�Step at each stage�

Page 14: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

LogitBoost

�� Start with weights wi � ��N i � �� �� � � � � N �

F �x� � � and probability estimates p�xi� ��� �

�� Repeat for m � �� �� � � � �M �

�a� Compute the working response and

weights

zi �

y�i � p�xi�

p�xi���� p�xi��

wi � p�xi���� p�xi��

�b� Fit the function fm�x� by a weighted

least�squares regression of zi to xi using

weights wi� i�e� a weighted tree

�c� Update F �x�� F �x� � fm�x� and p�x�

�� Output the classier

sign�F �x� � sign�PM

m�� fm�x�

We also have a natural generalization of

LogitBoost for multiple classes�

Stanford University December �� ���� Boosting� ����

��

Additive Logistic Trees

Best�rst tree growing allows us to limit the size

of each tree� and hence the interaction order�

By collecting terms� we get

F �x� �

Xj

fj�xj� �X

j�k

fjk�xj � xk�

�X

j�k�lfjkl�xj � xk� xl� � ���

Coordinate functions for Additive Stumps Model�

Boosting uses stage�wise optimization� as opposed

to joint optimization �full least squares�

backtting�� � ���

Page 15: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Stanford University December �� ���� Boosting� ����

��

Page 16: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Larg

eR

eal

Exam

ple

�Sati

mage

Meth

od

Term

inal

Iteration

s

Nod

es

��

��

���

���

Satimage

CA

RT

err

or

���

Logit

Bo

ost

���

����

����

����

Real

Ad

aB

oost

���

����

����

����

Gentl

eA

daB

oost

���

����

����

����

Dis

cre

teA

daB

oost

���

����

�� �

����

Logit

Bo

ost

���

��

����

����

����

Real

Ad

aB

oost

���

��

����

����

����

Gentl

eA

daB

oost

���

��

����

����

����

Dis

cre

teA

daB

oost

���

��

����

����

����

����

trai

nin

g���

��te

st�

��fe

atu

res�

�cl

asse

s

Stanford University December �� ���� Boosting� ����

��

Larg

eR

eal

Exam

ple

�L

ett

er

Meth

od

Term

inal

Iteratio

ns

Fractio

n

Nod

es

��

��

���

���

Le

tt

er

CA

RT

err

or

���

Logit

Bo

ost

���

��

����

����

�� �

���

Real

Ad

aB

oost

���

����

����

����

���

Gentl

eA

daB

oost

���

����

����

�� �

��

Dis

cre

teA

daB

oost

���

��

����

����

����

���

Logit

Bo

ost

���

��

�� �

����

����

���

Real

Ad

aB

oost

���

��

�� �

����

����

���

Gentl

eA

daB

oost

���

��

�� �

����

����

���

Dis

cre

teA

daB

oost

���

��

�� �

����

����

���

����

�tr

ain

ing�

����

test

���

feat

ure

s��

�cl

asse

s

Page 17: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Weight Trimming

� At each iteration� observations with wi � t��� are

not used for training� t��� is �th quantile of

weight distribution� and � � ������ ���� Works

better for LogitBoost�

� LogitBoost has weights wi pi��� pi� which

are large near the decision boundary�

� AdaBoost has weights wi e�yiFM �xi� �recall

yi � f��� �g�� Large for misclassi�ed points�

� For multiple�class procedures� if the class�k logit

Fmk � � � log�N�� training stops for that class�

Stanford University December �� ���� Boosting� � ��

��

Summary and Closing Comments

� The introduction of Boosting by Schapire�

Freund� and colleagues has brought us an

exciting and important set of new ideas�

� Boosting ts additive logistic models� where

each component �base learner� is simple� The

complexity needed for the base learner

depends on the target function�

� Little connection between weighted boosting

and bagging� boosting is primarily a bias

reduction procedure� while the goal of

bagging is variance reduction�

� The distinction becomes blurred when

weighting is achieved �in boosting� by

importance sampling�

Page 18: Stanford Univ ersit y Decem ber Bo osting - perso.ens-lyon.frperso.ens-lyon.fr/.../_agarivie/Telecom/PESTO/boost.slides.pdf · Stanford Univ ersit y Decem ber Bo osting Additiv e

Stanford University December �� ���� Boosting� ����

��

Margins

• ••• •••••••••• • •••••••••••• •••••••••••••••••••••••••••••••••

••••••••••••••••••••••••••••••••••

••••••••••••••••••••••••••••••

•••••••••••••••••

••••••••••••••••••••••••••••••••••••••••••••

•••••••••••••••••••••••

••••••••••••••••••••••••••••••••••••••••••••••

••••••••••••••••••

•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••

•• •• ••• •••••••••••••••••••••••••••••••••••••••••••••••••••

•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••

• • • • •• •• • ••••• • ••••• •••• ••••••••••••••••••••••••••••••••••••••••••••••••••

••••••••••••••••••••••••••••••••••••••••••••••••••

•••••••••••••••••••••••••••••••••••••••••••••••••

•••••••••••••••••••••••••••••••••••••••••••••••••

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••

•••

margin�X� � M�X� � � PC�X � �

Freund � Schapire ���� �� Boosting generalizes

because it pushes the training margins well above

zero� while keeping the VC dimension under

control �also Vapnik� ������ With Pr � ��� ��

PTest�M�X� � �� � PTrain�M�X� � ��

�O�

�pN

�logN log jHj

�� � log ���� ��

Stanford University December �� ���� Boosting� ����

��

How does Boosting avoid overtting�

� As iterations proceed� impact of change is

localized�

� Parameters are not jointly optimized � stagewise

estimation slows down the learning process�

� Classi�ers are hurt less by over�tting �Cover and

Hart� ������

� Margin theory of Schapire and Freund� Vapnik�

Disputed by Breiman �������

� Jury is still out�