Naive Bayes Herni

7/25/2019 Naive Bayes Herni

1/20

COMP24111 Machine Learning

Nave Bayes Classifer

Ke Chen


2/20



2

Outline

Background

Probability Basics

Probabilistic Classifcation

Nave Bayes

Principle and Algorithms

Eample! Play "ennis

#elevant $ssues

%ummary


3/20


3

Background

"here are three methods to establish a classifer

a& 'odel a classifcation rule directly

Eamples! k(NN) decision trees) perceptron) %*'

b& 'odel the probability o+ class memberships given input data

Eample! perceptron ,ith the cross(entropy cost

c& 'ake a probabilistic model o+ data ,ithin each class

Eamples! naive Bayes) model based classifers

a& and b& are eamples o+ discriminative classifcation

c& is an eample o+ generativeclassifcation

b& and c& are both eamples o+ probabilisticclassifcation


4/20


4

Probability Basics

Prior) conditional and -oint probability +or random

variables

Prior probability!

Conditional probability!

.oint probability! #elationship!

$ndependence!

Bayesian #ule

)|,)( 121 XP(XX|XP 2

)()()()(

X

XX

PCPC|P|CP =

)(XP

))(),,( 22 ,XP(XPXX 11 == XX

)()|()()|() 2211122 XPXXPXPXXP,XP(X1 ==

)()()),()|(),()|( 212121212 XPXP,XP(XXPXXPXPXXP 1 ===

EvidencePriorLikelihoodPosterior

=


5/20


5

Probability Basics

/ui0! 1e have t,o si(sided dice2 1hen they are tolled) it couldend up ,ith the +ollo,ing occurance! 3A& dice 4 lands on side 567)

3B& dice 8 lands on side 547) and 3C& ",o dice sum to eight2 Ans,er

the +ollo,ing 9uestions!

?toequal),(Is8)

?),(7)?),(6)

?)|(5)

?)|(4)

?3)

?2)

?)()1

P(C)P(A)CAP

CAPBAP

ACP

BAP

P(C)

P(B)

AP

==

==

===


6/20


6


Establishing a probabilistic model +or classifcation

Discriminative model

),,,)( 1 n1L X(Xc,,cC|CP == XX

),,,( 21 nxxx =x

Discriminative

Probabilistic Classifier

1x 2x nx

)|(1 xcP )|(2 xcP )|( xLcP


7/20


7


Establishing a probabilistic model +or classifcation

3cont2&

Generative model

),,,)( 1 n1L X(Xc,,cCC|P == XX

Generative

Probabilistic Model

for Class 1

)|( 1cPx

1x 2x nx

Generative

Probabilistic Model

for Class 2

)|( 2cPx

1x 2x nx

Generative

Probabilistic Model

for Class L

)|(LcPx

1x 2x nx

),,,( 21 nxxx =x


8/20


8


'AP classifcation rule MAP! Maimum APosterior

Assign xto c*i+

:enerative classifcation ,ith the 'AP rule Apply Bayesian rule to convert them into posterior

probabilities

"hen apply the 'AP rule

Lc,,cccc|cCP|cCP ===>== 1** ,)()( xXxX

LicCPcC|P

P

cCPcC|P|cCP

ii

ii

i

,,2,1for)()(

)(

)()()(

====

=

======

xX

xX

xXxX


9/20


9

Nave Bayes

Bayes classifcation

;i


10/20

COMP24111 M hi L i

10

Nave Bayes

;inexampleswith)|(estimate)|(),1;,,1(featureeachofvaluefeatureeveryFor

;inexampleswith)(estimate)(

oft valueeach targeFor 1

S

S

ijkjijkj

jjjk

ii

Lii

cCxXPcCxXPN,kFjXx

cCPcCP

)c,,c(cc

======

==

=

Lnn ccccccPcaPcaPcPcaPcaP ,,,),()]|()|([)()]|()|([ 1*

1***

1 =>

),,(1 naa =X


11/20

COMP24111 M hi L i

11

Eample

Eample! Play "ennis


12/20

COMP24111 M hi L i

12

Eample

>earning Phase

Outlook Play=Yes Play=No

Sunny 2/9 3/5Overcast 4/9 0/5Rain 3/9 2/5

Temperature Play=Yes Play=No

Hot 2/9 2/5Mild 4/9 2/5Cool 3/9 1/5

Humidity Play=YesPlay=No

High 3/9 4/5Normal 6/9 1/5

Wind Play=Yes Play=No

Strong 3/9 3/5Weak 6/9 2/5

P(Play=Yes) =9/14P(Play=No) =5/14


13/20

Eample o+ the Nave BayesClassifer

The weather data, with counts and probabilities

outlook temperature humiit! "in! pla!

!e# no !e# no !e# no !e# no !e# no

#unn! 2 3 hot 2 2 high 3 4 $al#e 6 2 9 5

o%erca#t 4 0 mil 4 2 normal 6 1 true 3 3rain! 3 2 cool 3 1

#unn! 2&9 3&5 hot 2&9 2&5 high 3&9 4&5 $al#e 6&9 2&5 9&14 5&14

o%erca#t 4&9 0&5 mil 4&9 2&5 normal 6&9 1&5 true 3&9 3&5

rain! 3&9 2&5 cool 3&9 1&5

A new day

outlook temperature humiit! "in! pla!

#unn! cool high true '


14/20

( >ikelihood o+ yes

( >ikelihood o+ no

( "here+ore) the prediction is No

005!01"

#

#

#

#

#

$==

0$0%!01"

5

5

5

"

5

1

5

==


16/20

COMP24111 M hi L i

16


17/20

COMP24111 M hi L i

17

Nave Bayes Algorithm! Continuous(valued ?eatures

Numberless values +or a +eature

Conditional probability o+ten modeled ,ith the normal

distribution

>earning Phase!

@utput! normal distributions and

"est Phase! :iven an unkno,n instance $nstead o+ looking(up tables) calculate conditional probabilities ,ith all the

normal distributions achieved in the learning phrase

Apply the 'AP rule to make a decision

ijji

ijji

ji

jij

jiij

cC

cX

XcCXP

==

==

whichforexamplesofXvaluesfeatureofdeviationstandard:

Cwhichforexamplesofvaluesfeatureof(avearage)mean:

2

)(exp

2

1)|( 2

2

Ln ccCXX ,,),,,(for 11 ==XLn LicCP i ,,1)( ==

),,(1 naa =X


18/20

COMP24111 M hi L i

18

Nave Bayes Eample! Continuous(valued ?eatures

"emperature is naturally o+ continuous value2

Yes! 828) 426) 42) 842D) 824) 8F26) 882) 8624) 42

No! 8D26) 624) 4D2F) 82) 424

Estimate mean and variance +or each class

Learnin Phase! output t,o :aussian models +or

P3tempGC&

==

==N

n

n

N

n

n xN

xN 1

22

1

)(1

,1

09.7,88.23

35.2,64.21

==

==

NoNo

YesYes

=

=

=

=

$5!50

)&&!$(exp

$0#!'

1

0#!'$

)&&!$(exp

$0#!'

1)|(

0#!11)%"!$1(exp

$5!$1

5!$$)%"!$1(exp

$5!$1)|(

$

$

$

$

$

$

xxNoxP

xxYesxP


19/20COMP24111 Machine Learning19

#elevant $ssues

*iolation o+ $ndependence Assumption

?or many real ,orld tasks)

Nevertheless) nave Bayes ,orks surprisingly ,ell

any,ay=

Hero conditional probability Problem

$+ no eample contains the +eature value

$n this circumstance) during test

?or a remedy) conditional probabilities re(estimated ,ith

)|()|()|,,( 11 CXPCXPCXXP nn

0)|(,

====

ijkjjkj cCaXPaX0)|()|()|(1 = inijki cxPcaPcxP

)1examples,virtual""of(numberpriortoweight:

)ofvaluespossiblefor/1(usually,estimateprior:

whichforexamplestrainingofnumber:

Candwhichforexamplestrainingofnumber:

)|(

==

==

++

===

mm

Xttpp

cCn

caXn

mn

mpncCaXP

j

i

ijkjc

cijkj


20/20

20

%ummary

Nave Bayes! the conditional independence assumption

"raining is very easy and +astI -ust re9uiring considering each

attribute in each class separately

"est is straight+or,ardI -ust looking up tables or calculating

conditional probabilities ,ith estimated distributions

A popular generativemodel

Per+ormance competitive to most o+ state(o+(the(art classifers

even in presence o+ violating independence assumption

'any success+ul applications) e2g2) spam mail fltering

A good candidate o+ a base learner in ensemble learning

Apart +rom classifcation) nave Bayes can do moreJ

Documents

Naive Bayes Herni