5. Maximum Likelihood –II

Prof. Yuille.

Stat 231. Fall 2004.

Topics

• Exponential Distributions, Sufficient Statistics, and MLE.

• Maximum Entropy Principle.

• Model Selection.

Exponential Distributions.

• Gaussians are a member of the class of exponential distribution.

• Parameters• Statistics

Sufficient Statistics.

• The are the sufficient statistics of the distribution.

• Knowledge of

is all we need to know about the data

The rest is irrelevant.

• Almost all distributions can be expressed as Exponentials – Gaussian, Poisson, etc.

Sufficient Statistics of Gaussian

• One-Dimensional Gaussian and samples

• Sufficient statistics are

• And

• These are sufficient to learn the parameters

of the distribution from data.

MLE for Gaussian

• To estimate the parameters – maximize

• Or equivalently, maximize:

• The sufficient statistics are chosen so that

Sufficient Statistics for Gaussian

• Distribution is of form:

• This is the same as a Gaussian with mean

• and variance

Exponential Models and MLE.

• MLE corresponds to maximizing

Equivalent to minimizing

Where

Exponential Models and MLE.

• This minimization is a convex optimization problem and hence has a unique solution. But finding this solution may be difficult.

• Algorithms such as Generalized Iterative Scaling are guaranteed to converge.

Maximum Entropy Principle.

• An alternative way to think of Exponential Distributions and MLE.

• Start with the Statistics, and then estimate the form and the parameters of the probability distribution.

• Using the Maximum Entropy principle.

Entropy

• The entropy of a distribution is

• Defined by Shannon as a measure of the information obtained by observing a sample from P(x).

Maximum Entropy Principle

• Maximum Entropy Principle. Select the distribution P(x) which maximizes the entropy subject to constraints.

• Lagrange multipliers

• The observed value of the statistics are

Maximum Entropy

• Minimize with respect to P(x). Gives the (exponential) form of the distribution:

• Maximizing with respect to the Lagrange parameters ensures that the constraints are satisfied:

•

Maximum Entropy.

• This gives the same result as MLE for Exponential Distributions.

• Maximum Entropy + Constraints = Exponential Distribution + MLE Parameter.

• The Max-Ent distribution which has the observed sufficient statistics is the exponential distribution with those statistics.

• Example: can obtain a Gaussian by performing Max-Ent on statistics

Minimax Principle.

• Construct a distribution incrementally by increasing the number of statistics

• The entropy of the Max-Ent distribution with M statistics is given by:

• Minimax Principle: select the statistics to minimize the entropy of the maximum entropy distribution. This relates to model selection.

Model Selection.

• Suppose we do not know which model generates the data.

• Two models

• Priors

• Model selection enables us to estimate which model is most likely to have generated the data

Model Selection.

• Calculate

• Compare with

• Observe that we must sum over all possible values of the model parameters

Model Selection & Minimax.

• The entropy of the Max-Ent distribution

• Is minus the probability of the data

• So the Minimax Principle is a form of model selection. But it estimates the parameters instead of summing them out.

Model Selection.

• Important Issue:

Suppose the model

has more parameters than

Then is more flexible and can fit a larger number of data models.

• But summing over the parameters • and penalizes this flexibility.• Gives “Occam’s Razor” favoring the simpler

model.

Model Selection.

• More advanced modeling requires performing model selection – where the models are complex.

• Beyond scope of this course.