View
59
Download
2
Category
Tags:
Preview:
DESCRIPTION
Text Classification from Labeled and Unlabeled Documents using EM. Kamal Nigam Andrew Kachites Mccallum Sebastian Thrun Tom Mitchell Presented by Yuan Fang, Fengyuan Hu and Sandhya Prabhakaran. Job Hunting?. Roadmap. Part 1 – Text Classification - PowerPoint PPT Presentation
Citation preview
Text Classification from Labeled and Unlabeled Documents using EM
- Kamal Nigam- Andrew Kachites Mccallum- Sebastian Thrun- Tom Mitchell
Presented by
Yuan Fang, Fengyuan Hu and Sandhya Prabhakaran
Job Hunting?
Roadmap
Part 1 – Text Classification
Part 2 – Incorporating Unlabeled data with EM
Part 3 – Results and Recap
Part I Text Classification
Text Classification – the Definition
“Text classification systems categorize documents into one (or several) of a set of pre-defined topics of interest”
How Are Automatic Text Classifiers Created
Before: Manual construction of rule sets (Painful and time-consuming )
Present : Supervised learning to construct a classifier (efficient and successful)
What To Provide
An algorithm with an example set of documents for each class and allow it to find a representation or decision rule for classifying future documents automatically
This approach will : - give high-accuracy classifiers - be significantly less expensive
What Data is Available
Key difficulty : A large number of labeled training examples are required to learn accurately - What we need but don't have
One would obviously prefer algorithms that can provide accurate classifications after hand labeling only a dozen articles, rather than thousands
What other sources of information can reduce the need for labeled data?
Unlabeled data
How unlabeled data can be used to increase classification accuracy, especially when labeled data are scarce
An intuitive example
Goal And Merit
The goal – To demonstrate that supervised learning algorithms
can use a small number of labeled examples with a large number of unlabeled examples to create high-accuracy text classifiers
The merit – Unlabeled examples are much less expensive and
easily available
Parametric Generative Model Overview
Assumption : a statistical process generates the documents (words and class labels)
statistical process - parametric generative model
Incorporating Unlabeled Data withGenerative Models
Using EM to find high-probability parameters of the model given a combination of labeled and unlabeled data
Experimental evidence shows that using unlabeled data with EM can increase classification accuracy
Assumptions In the Model
(1) Documents are generated by a mixture of multinomials model, where each mixture component corresponds to a class (1 class to 1 component)
(2) The mixture components are multinomial distributions of individual words - the words are produced independently of each other given the class
Two Multisided Dies
Let there be |C| classes and a vocabulary of size |V|; each document d has |d| words in it.
First, we roll a biased |C|-sided die to determine the class of our document.
We roll the biased |V|-sided die that corresponds to the chosen class |d| times and write down the indicated words. These words form the generated document.
Parametric Generative Model
- parameters for the mixture model - mixture of components - mixture weights or class probabilities - document distribution of selected class
Equation (1)
Denotation
- the jth mixture component, as well as the jth class. - the class label for a particular document ( )
A document is considered to be an ordered list of word events,
We write for the word in position k of - a word in the vocabulary
- document length, chosen independently of the component, its own probability
Parametric Generative Model
Expanding the Equation (1) with document length and the words in the document. Equation (2)
The words of a document are generated independently of context
Equation (3)
Combining these last two equations gives the naive Bayes expression for the probability of a document given its class
Equation (4)
Model Parameters
Collection of word probabilities, each written
Document length is identically distributed, no need to be parameterized for classification
denoted as the mixture weights (class probabilities)
The complete collection of model parameters
Naive Bayes Text Classification
Using a collection of labeled documents for training
Finding the most probable parameters for the statistical
model introduced
Training A Naive Bayes Classifier With Labeled Data
Estimating the parameters of the generative model by using a set of labeled training data
(the estimate of the parameters is written )
Finding (MAP), the value of that is most probable given the evidence of the training data and a prior.
Training A Naive Bayes Classifier With Labeled Data
The word probability estimates are given by Equation (6)
Class probabilities
Equation (7)
Classifying New Documents with Naive Bayes
Equation (8)
If the task is to classify a test document into a single class, then the class with the highest posterior probability
is selected.
Part ⅡIncorporating Unlabeled Data with EM
The Problem
The case that given only labeled data is explained already. MAP– to maximize the posterior probability. Naïve Bayes– do classification of labeled data. Now the case is given both labeled and unlabeled data. Searching for a solution? – Here it is!
Revision of EM
Recall the EM knowledge in PMR – Might be painful, but helpful
Mixture Model Hidden variable – z to active the components
Revision of EM
EM applied to Gaussian Mixture Model Maximum Likelihood Estimation Parameters: µ andΣ E step: evaluate the responsibilities using current estimators/
parameters M step: re-estimate by using the maximum a posteriori para
meter Run the demo
Back to the paper
Back to the paper
Collection of labeled and unlabeled documents. MAP Try to maximize P(θ|D) Bayesian method -- P(θ|D) → P(θ) P(D| θ)
ul DDD
Back to the paper
Log likelihood Incomplete equation
Back to the paper
z – binary indicator variables which is set to be 1 if y = c, else zero.
Then problem of the incomplete log probability can be transferred to complete log probability of parameters.
Back to the paper
Methods used in the paper Basic EM Augmented EM
(1) Weighting the unlabeled data
(2) Multiple mixture components per class
Basic EM
Initialize the NB classifier using MAP parameter estimation, from only labeled dataset.
E step: estimate the component membership
by calculating its expected value generated by from only unlabeled data. M step: re-estimate the classifier for the whole data set,
using MAP, loop from E step: Look at to measure the improvement of the
parameters, decide when to stop the loop
),|( ij dcP
),|(maxarg 11 kk zDP
),|(1 kk DzEz
),|( zDlc
Restrictions of Basic EM
Assumptions/Restrictions: Large unlabeled data set, small labeled data set → if not
true, unlabeled data will hurt the accuracy. One-to-one correspondence of components and classes →
not so accurate because subtopics exist.
Augmented EM – weighting unlabeled data
Method: weakening the contribution of unlabeled data while the labeled set is already good enough for classification.
Equation:
))),|()|(log(
()),|()|(log(
))(log(),|(
1
1
ui
ld
Dd
C
jjijij
Dd
C
jjijij
c
cdPcPz
cdPcPz
PzDl
Augmented EM – weighting unlabeled data
λis decided by leave-one-out cross validation. is defined to tell whether it is labeled or unlabeled.
Modified MAP parameters:
)(i
Augmented EM -- multiple mixture components per class
Method: Relax the assumption that one-to-one correspondence of components and classes.
Many-to-one relationship between components and classes.
Augmented EM – multiple mixture components per class
How? Decide the number of components per class by again cross-
validation. Mapping from components to classes: }1,0{),|( ja ctP
The complete algorithm
Collections of labeled, unlabeled documents. Set λby cross-validation. Set the number of components per class. Randomly assign for mixture components. Initialize the parameters θ of NB classifier using MAP. Loop until complete log likelihood of labeled and unlabeled
data is satisfying enough. E step: estimate the component membership of each doc usi
ng θ M step: re-estimate θgiven the membership, still MAP.
ul DDD
),|( ij dcP
Comparison
Basic EM: performs well comparing with naïve bayes classifier alone, with large unlabeled dataset and small set of labeled data
EM-λ: can apparently improve the accuracy if the assumption above doesn’t fit.
Multiple Components: dramatically outperforms than basic EM.
Part III Results and Recap
Experimental Results
Empirical evidence that on combining labeled with unlabeled data using EM outperforms naive Bayes.
20 Newsgroups, WebKB, Reuters
Improvements in accuracy due to unlabeled data are dramatic, especially when the number of labeled data is low.
Augmented EM can increase performance even when basic EM performs poor due to large number of unlabeled data.
Data sets and Protocols
- 20 Newsgroup
20017 articles divided evenly among 20 different UseNet discussion groups.
Task - to classify an article into the one newsgroup to which it was posted.
Many categories fall into confusable clusters. Stop words are removed – 62258 unique words Word counts are normalized and scaled – each document
has constant length.
Data sets and Protocols
- WebKB
8145 Web pages gathered from university computer science departments.
Choosing 4199 pages covering categories: student, faculty, course and project.
Task - to classify a web page into one of the four categories.
Stemming and stoplist are not used. Vocabulary is limited to 300 most informative words using
leave-one-out cross validation.
Data sets and Protocols
- Reuters
12902 articles and 90 topic categories. Task - to build a binary classifier for each of the ten most
populous classes to identify the news topic. Words inside <TEXT> tags are used – REUTERS and &#
not used. Stoplist are used, but no stemming. Metrics are Recall and Precision instead of Accuracy.
Precision-Recall breakeven point
• Standard information retrieval measure
• Recall – number of correct positive predictions
number of positive examples
• Precision - number of correct positive predictions
number of positive predictions
Wall-clock timing
EM usually converges after 10 iterations
Less than 1 minute for the WebKB
Less than 15 minutes for 20 Newsgroups – huge vocabulary and more documents
EM with unlabeled data increases Accuracy
Figure 1:- Accuracy versus # of Labeled Documents. (20 Newsgroups)
Effect of varying the # of unlabeled documents
Figure 2:- Accuracy versus # of unlabeled documents. (20 Newsgroups)
EM algorithm in action
Figure 3:- ‘Course’ class for WebKB dataset
EM performance degradation
Figure 4:- As # of Labeled data increases, accuracy of classifier falls with more # of unlabeled data. Importance of weighting factor λ. (WebKB)
Effects of different EM
Figure 5:- Comparison between EM, CV EM-λ and EM-λ (WebKB)
Performance of EM on different # of mixture components
Figure 6:- Too few or too many mixture components result in poor performance. Unlabeled data is used. (Reuters)
Precision-Recall breakeven points
Figure 7:- Comparison between NB and EM on Reuters dataset
Related Work
EM is a well-known family of algorithms that works by treating unclassified data as incomplete.
According to Miller et al - EM on non-textual tasks using mixture of Gaussians – assumed unlabeled data to be sufficient to estimate parameter values.
Castelli and Cover - unlabeled data does not improve the classification results in the absence of labeled data.
EM can be combined with active-learning to improve performance – now only slightly more than half of labeled data was enough!
EM can be applied with other machine learning algorithms like SVM, kNN.
Punchwords
• Text classification
• Naive Bayes
• Expectation Maximisation Algorithm
• EM-λ
• Multiple Mixture models for subclass
• Leave-one-out cross validation
• Stemming and stoplist words
• Accuracy, Precision, Recall
Recap
A family of algorithms have been presented to address text classification using voluminous unlabeled data and scarce labeled data.
When data is consistent with the assumptions - Basic EM performs well.
When data is not consistent - 2 extensions hold valid
- EM-λ: controlling the contribution of unlabeled data.
- Multiple Mixture Components per Class: “many-to-one” constraint.
References
Using Unlabeled Data to Improve Text Classification – May 2001 at
www.kamalnigam.com/papers/thesis-nigam.pdf Netlab toolkit - www.ncrg.aston.ac.uk/netlab/ Validation Lecture – Intelligent Sensor Systems, RicardoG
utierrez-Osuna, Wright State University
Question Time!!
Route further questions to ...
Ryan - 0789317
Neo - 0785401
Sandhya - 0671562
Recommended