Example

• 16,000 documents

• 100 topic

• Picked those with large p(w|z)

• Given a new document, compute and • words allocated to each topic• approximates p(zn|w)• See cases where these values are relatively

large• 4 topics found

n

New document?i n

ii

Unseen document (contd.)

• Bag of words - William Randolph Hearst Foundation assigned to different topics

Applications and empirical results

• Document modeling

• Document classification

• Collaborative filtering

Document modeling

• Task: density estimation, high likelihood to unseen document

• Measure of goodness: perplexity

• Monotonically decreases in the likelihood

The experiment

Articles Terms

Scientific abstracts

5,225 28,414

Newswire articles

16,333 23,075

The experiment (contd.)

• Preprocessed– stop words– appearing once

• 10% held for training

• Trained with the same stopping criteria

Results

Overfitting in Mixture of unigrams

• Peaked posterior in the training set

• Unseen document with unseen word

• Word will have very small probability

• Remedy: smoothing

Overfitting in pLSI

• Mixture of topics allowed• Marginalize over d to find p(w)• Restriction to having the same topic

proportions as training documents• “Folding in” ignore p(z|d) parameters and

refit p(z|dnew)

LDA

• Documents can have different proportions of topics

• No heuristics

Document classification

• Generative or discriminative• Choice of features in document

classification• LDA as dimensionality reduction

technique• as LDA features)w(

The experiment

• Binary classification• 8000 documents, 15,818 words• True label not known• 50 topic• Trained SVM on the LDA features• Compared with SVM on all word features• LDA reduced feature space by 99.6%

GRAIN vs NOT GRAIN

EARN vs NOT EARN

LDA in document classification

• Feature space reduced, performance improved

• Results need further investigation• Use for feature selection

Collaborative filtering

• Collection of users and movies they prefer• Trained on observed users• Task: given unobserved user and all

movies preferred but one, predict the held out movie

• Only users who positively rated 100 movies

• Trained on 89% of data

Some quantities required…• Probability of held out movie p(w|wobs)

– For mixture of unigrams and pLSI sum out topic variable

– For LDA sum out topic and Dirichlet variables (quantity efficient to compute)

Results

Further work

• Other approaches for inference and parameter estimation

• Embedded in another model

• Other types of data

• Partial exchangeability

Example – Visual words

• Document = image• Words = image features: bars, circles• Topics = face, airplane• Bag of words = no spatial relationship

between objects

Visual words

Identifying the visual words and topics

Conclusion

• Exchangeability, De Finetti Theorem• Dirichlet distribution Generative Bag of words Independence assumption in Dirichlet

distribution - correlated topics

Implementations

• In C (by one of the authors)– http://www.cs.princeton.edu/~blei/lda-c/

• In C and Matlab– http://chasen.org/~daiti-m/dist/lda/

References• Latent Dirichlet allocation, D. Blei, A. Ng, and M. Jordan.

In Journal of Machine Learning Research, 3:993-1022, 2003

• Discovering object categories in image collections. J. Sivic, B. C. Russell, A. A. Efros, A. Zisserman, W. T. Freeman. MIT AI Lab Memo AIM-2005-005, February, 2005

• Correlated topic models, David Blei and John Lafferty, Advances in Neural Information Processing Systems 18, 2005.