Is Unlabeled Data Suitable for Multiclass SVM-based Web Page Classification?

Is Unlabeled Data Suitable for Multiclass SVM-basedWeb Page Classification?

Arkaitz Zubiaga, Vıctor Fresno, Raquel Martınez

Universidad Nacional de Educacion a Distancia

June 4, 2009

Text Classification

1 Text Classification

2 Motivation

3 Support Vector Machines

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

7 Compared Approaches: Multiclass SVM vs Multiclass S3VM

8 Experiments

9 Results

10 Conclusions and Outlook

11 Thank you

A. Zubiaga, V. Fresno, R. Martınez (UNED) Unlabeled Data for Multiclass SVM June 4, 2009 2 / 31

Text Classification

What is it?

We have a set of documents:

D = {d1, ..., d|D|}

With a set of predefined categories:

C = {c1, ..., c|C |}

Classification is known as:

〈dj , ci 〉 ∈ D × C

Motivation

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Motivation

Several studies for plain text classification (news), but a few for webpage classification.

Typical web page classification task:

Semi-supervised: not much labeled documents.Multiclass: taxonomy > 2.

(Joachims, 1999) proved the suitability of unlabeled data for binarytasks.

What about multiclass tasks?(Chapelle et al., 2006) did it over image datasets, but never fortext/web pages.

Support Vector Machines

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

It looks for a hyperplane to separate the classes

Margin maximization

It looks for a hyperplane to separate the classes

Margin maximization

Optimization function: min 12 ||ω||

2 + C ·∑n

i=1 ξdi

Subject to: yi (ω · xi + b) ≥ 1− ξi , ξi ≥ 0

It only handles binary and supervised problems by nature.

Multiclass SVM

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Multiclass SVM

Approaches to multiclass SVM:

Direct.Combining binary classfiers.

One-against-one.One-against-all.

Usually applied to supervised tasks, but hardly ever to semi-supervisedones.

Multiclass SVM

Multiclass SVM: Direct approach

The optimization function considers all the hyperplanes at the sametime.

n∑m=1

||wm||2 + Cl∑

∑m 6=yi

Subject to:

wyi · xi + byi ≥ wm · xi + bm + 2− ξmi , ξmi ≥ 0

Multiclass SVM

Multiclass SVM: One-against-one

It creates k·(k−1)2 binary classifiers

Multiclass SVM

sign(ωTij · x + bij) −→ Add a vote for the winning class between i and j

The class with more votes will be the output.

Multiclass SVM

Multiclass SVM: One-against-all

It creates k binary classifiers

Multiclass SVM

Ci = arg maxi=1,...,k

(ωi · x + bi )

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Semi-supervised SVM (S3VM)

Unlabeled documents are considered during the learning phase.

The optimization function results:

2· ||ω||2 + C ·

l∑i=1

ξdi + C ∗ ·u∑

ξ∗d

Convex optimization algorithms required.

Commonly used over binary taxonomies, but hardly ever with moreclasses.

Multiclass S3VM

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Multiclass S3VM

(Yajima and Kuo, 2006) present the following optimization function:

h∑i=1

βiT K−1βi + Cl∑

∑i 6=yj

max(0, 1− (βyj

j − βij ))2)

where β represents the product of a vector and a kernel matrix defined bythe author.

(Chapelle et al., 2006): direct approach by means of the ContinuationMethod.

2 steps:

(Qi et al., 2004) use Fuzzy C-Means to predict new unlabeleddocuments.(Xu and Schuurmans, 2005) rely on a clustering-based approach tolabel the unlabeled data.

Compared Approaches: Multiclass SVM vs Multiclass S3VM

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Compared Approaches: Multiclass SVM vs Multiclass S3VM

Multiclass SVM vs Multiclass S3VM

2-steps-SVM/1-step-SVM: Multiclass SVM.Does an intermediate step adding newly labeled data improveclassifier’s performance?

One-against-all-S3VM/One-against-all-SVM.

One-against-one-S3VM/One-agaisnt-one-SVM.Does unlabeled data help to improve binary combining classifier’sresults?

Experiments

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Experiments

Experiments settings

Datasets:

BankSearch: 10.000 web documents / 10 categories (4.000 for thetraining set).WebKB: 4.518 web documents / 6 categories (2.000 for the trainingset).Yahoo! Science: 788 web documents / 6 categories (200 for thetraining set).

Numerous labeled/unlabeled sets.

9 executions for each.

Representation: TF-IDF.

Software:

SVM-light (http://svmlight.joachims.org)SVM-multiclass

Evaluation by means of the accuracy (percent of correct predictions).

Results

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Results

Results: BankSearch

Results

Results: WebKB

Results

Results: Yahoo! Science

Results

Supervised multiclass approaches (2-steps-SVM & 1-step-SVM)outperform the rest.

Among binary combinations, one-against-all outperformsone-against-one.

Unlabeled data slightly helps for one-against-all.

1-step-SVM and 2-steps-SVM show similar results, except forWebKB, where the former wins.

It could be due to the homogeneous nature of the WebKB dataset.

Conclusions and Outlook

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Conclusions

Comparison of multiclass SVM and S3VM approaches for web pageclassification.

Direct and combining approaches.

Direct approaches outperform the rest.

Unlabeled data did not provide considerable improvements, and evenprovide worsenings in some cases.

Future Work

To add more multiclass S3VM approaches to the study.

To test with different SVM settings (kernel, parameters,...).

Thank you

2 Motivation

4 Multiclass SVM

5 S3VM

6 Multiclass S3VM

8 Experiments

9 Results

11 Thank you

Thank you

Is Unlabeled Data Suitable for Multiclass SVM-based Web Page Classification?

Technology

Multiclass Securities Guide Part I

Linear models and multiclass classification

SVM for Overlapping Classes - University at Buffalosrihari/CSE574/Chap7/7.2-SVM-Overlap.pdf · SVM for Overlapping Classes srihari@buffalo.edu . Machine Learning Srihari SVM Discussion

Multiclass Classification

Position-BasedFeatureSelectionforBodySensorsregardingDaily ... · our previous work [6], a multiclass support vector machine (SVM) algorithm demonstrated 97.3% accuracy when we used

Learning from Positive and Unlabeled Examples

Consistent Algorithms for Multiclass Classification with

MultiClass UL 400

Daniele Loiacono - Politecnico di Milanohome.deib.polimi.it/loiacono/uploads/Teaching/DMTM/DMTM Suppor… · Training SVM ν-SVM Multi-class SVM Regression. Daniele Loiacono SVM Training

Bootstrapping Information Extraction with Unlabeled Data

HEAVY TRAFFICCONVERGENCEOF ACONTROLLED, MULTICLASS …

Mesoscopic multiclass traffic flow modeling on multi-lane ... · PDF fileMesoscopic multiclass traﬃc ﬂow modeling on multi-lane sections ... G. Costeseque∗, A. Duret Meso Multiclass

Robust Top-k Multiclass SVM for Visual Category Recognitiony328yu/mypapers/kdd17.pdf · Xiaojun Chang, Yao-Liang Yu, and Yi Yang. 2017. Robust Top-k Multiclass SVM for Visual Category

Set 1 Unlabeled

The Pricing of Multiclass Commercial Mortgage-Backed ...efinance.org.cn/cn/fm/The Pricing of Multiclass Commercial Mortgage... · The Pricing of Multiclass Commercial Mortgage

LEUKO: Computer Decision Support System for Leukemia Diagnosis · 2012. 2. 25. · Leuko + SVM • SVM extension to multiclass task, applying tree-based strategy; • SVM and minimum

Classification of unlabeled data:

2014 Spring Fashion Lookbook from Unlabeled

Mengke Hu, David Cinciruk · Mengke Hu, David Cinciruk ASPITRG Group, ... anti-bio >antibio Stemming: ... (for some SVM algorithms even Multiclass is allowed)

PERFORMANCE OF MULTICLASS MARKOVIAN QUEUEING - MIT