PatReco: Bayes Classifier and Discriminant Functions

Alexandros Potamianos

Dept of ECE, Tech. Univ. of Crete

Fall 2009-2010

PatReco: Problem Solving

1. Data Collection2. Data Analysis3. Feature Selection4. Model Selection5. Model Training6. Classification7. Classifier Evaluation

Bayes Classifier

Classes: ω1, ω2 , … ωn

Sample: x = (x1, x2, … xd ) [d-Dimensional features]

Model: p(x|ω1), p(x|ω2), … p(x|ωn)

p(ω1), p(ω2), … p(ωn)

Bayes classifier (classify sample x to class ωω):

ωω = arg maxωi p(ωi|x)

= arg maxωi p(x|ωi) p(ωi)

Bayes Error

Classes: ω1, ω2 , … ωn

Sample: x = (x1, x2, … xd ) [d-Dimensional features]

Model: p(x|ω1), p(x|ω2), … p(x|ωn)

p(ω1), p(ω2), … p(ωn)

Decision regions: Ω1, Ω2, ... Ωn

Bayes error (probability of wrong classification):

P(error)P(error) = 1 - P(correct) =

= 1 - i Ωi p(x|ω1) p(ω1) dx

Discriminant Functions

Define class boundaries (instead of class characteristics)

Dualism: Parametric class description

Bayes classifier Decision boundary

Parametric Discriminant Functions

Normal Density

1D Multi-D

Full covariance Diagonal covariance Diagonal covariance + univariate

Mixture of Gaussians Usually diagonal covariance

Gaussian Discriminant Functions

Same variance ALL classes Hyper-planes

Different variance among classes Hyper-quadratics (hyper-parabolas, hyper-

ellipses etc.)

Hyper-Planes

When the covariance matrix is common across Gaussian classes The decision boundary is a hyper-plane that is

vertical to the line connecting the means of the Gaussian distributions

If the a-priori probabilities of classes are equal the hyper-planes cuts the line connecting the Gaussian means in the middle Euclidean classifier

Gaussian Discriminant Functions

Same variance ALL classes Hyper-planes

Different variance among classes Hyper-quadratics (hyper-parabolas, hyper-

ellipses etc.)

Hyper-Quadratics

When the Gaussian class variances are different the boundary can be hyper-plane, multiple hyper-planes, hyper-sphere, hyper-

parabola, hyper-elipsoid etc. The boundary in general in NOT vertical to the Gaussian

mean connecting line If the a-priori probabilities of classes are equal the

resulting classifier is a Mahalanobois classifier

Conclusions

Parametric statistical models describe class characteristics x

by modeling the observation probabilities p(x|class)

Discriminant functions describe class boundaries

parametrically

Parametric statistical models have an equivalent parametric

discriminant function

For Gaussian p(x|class) distributions the decision boundaries

are hyper-planes or hyper-quadratics