1

Learning to Classify Short and Sparse Text & Web withHidden Topics from Large-

scale Data CollectionsXuan-Hieu Phan Le-Minh Nguyen Susumu HoriguchiGSIS, Tohoku University GSIS, JAIST GSIS, Tohoku

University

WWW 2008

NLG Seminar 2008/12/31Reporter:Kai-Jie Ko

2

Motivation

Many classification tasks working with short segments of text & Web, such as search snippets, forum & chat messages, blog & news feeds, product reviews, and book & movie summaries, fail to achieve high accuracy due to the data sparseness

3

Previous works to overcome data sparsenessEmploy search engines to expand and

enrich the context of data

4

Previous works to overcome data sparsenessEmploy search engines to expand and

enrich the context of data

Time consuming!

5

Previous works to overcome data sparsenessTo utilize online data repositories, such as

Wikipedia or Open Directory Project,as external knowledge sources

6

Previous works to overcome data sparsenessTo utilize online data repositories, such as

Wikipedia or Open Directory Project,as external knowledge sources

Only used the user defined categories and concepts in those repositories, not general enough

7

General framework

8

(a)Choose an universal data

•Must large and rich enough to cover words, concepts that are related to the classification problem.•Wikipedia & MEDLINE are chosen in this paper.

9

(a)Choose an universal data

Use topic oriented keywords to crawl Wikipedia with maximum depth of hyperlink 4◦240MB◦71,968 documents◦882,376 paragraphs◦60,649 vocabulary◦30,492,305 words

10

(a)Choose an universal data

Ohsumed : a test collection of medical journal abstracts to assist IR research◦156MB◦233,442 abstracts

11

(b)Doing topic analysis for the universal dataset

12

(b)Doing topic analysis for the universal dataset

Using GibbsLDA++, a C/C++ implementation of LDA using Gibbs Sampling

The number of topics ranges from 10, 20 . . . to 100, 150, and 200

The hyperparameters alpha and beta were set to 0.5 and 0.1, respectively

13

Hidden topics analysis for Wikipedia data

14

Hidden topics analysis for the Ohsumed-MEDLINE data

15

(c)Building a moderate size labeled training dataset

•Words/terms in this dataset should be relevant to as many hidden topics as possible.

16

(d)Doing topic inference for training and future data

•To transform the original data into a set of topics

17

Sample Google search snippets

18

Snippets word co-occurence

This show the sparseness of web snippetsin that only small fraction of words are shared by the 2 or 3 different snippets

19

Shared topics among snippets after inferenceAfter doing inference and integration,

snippets are more related in semantic way

20

(e) Building the classifier

•Choose from different learning methods•Integrate hidden topics into the training, test, or future data according to the data representation of the chosen learning technique•Train the classifier on the integrated training data

21

Evaluation

Domain disambiguation for Web search results◦To classify Google search snippets into different

domains, such as Business, Computers, Health, etc.

Disease classification for medical abstracts◦Classifies each MEDLINE medical abstract into

one of five disease categories that are related to neoplasms, digestive system, etc.

22

Domain disambiguation for Web search results

Obtain Google snippet as training and testing data, the search phrase of the two data are totally exclusive

23

Domain disambiguation for Web search results

The result of doing 5-fold cross validation on the training data

Reduce 19% of error on average

24

Domain disambiguation for Web search results

25

Domain disambiguation for Web search results

26

Disease Classification for Medical Abstracts with MEDLINE Topics

The proposed method requires only 4500 training data to reachthe accuracy of the baseline which uses 22500 training data!

27

Conclusion

Advantages of proposed framework:◦A good method to classify sparse and previous

unseen data Utilizing the large universal dataset

◦Expanding the coverage of the classifier Topics coming from external data cover a lot of

terms/words that do not exist in training dataset◦Easy to implement

Only have to prepare a small set of labeled training example to attain high accuracy