An Introduction to Structural SVMs and its Application to Information Retrieval

Yisong YueCarnegie Mellon University

Supervised Learning• Find function from input space X to output space Y

such that the prediction error is low.

Microsoft announced today that they acquired Apple for the amount equal to the gross national product of Switzerland. Microsoft officials stated that they first wanted to buy Switzerland, but eventually were turned off by the mountains and the snowy winters…

GATACAACCTATCCCCGTATATATATTCTATGGGTATAGTATTAAATCAATACAACCTATCCCCGTATATATATTCTATGGGTATAGTATTAAATCAATACAACCTATCCCCGTATATATATTCTATGGGTATAGTATTAAATCAGATACAACCTATCCCCGTATATATATTCTATGGGTATAGTATTAAATCACATTTA

• Part-of-Speech Tagging– Given a sequence of words x, predict sequence of tags y.– Dependencies from tag-tag transitions in Markov model.

Similarly for other sequence labeling problems, e.g., RNA Intron/Exon Tagging.

The rain wet the catx Det NVDet Ny

Examples of Complex Output Spaces

• Natural Language Parsing– Given a sequence of words x, predict the parse tree y.– Dependencies from structural constraints, since y has to be a

The dog chased the catx

Det NV

• Information Retrieval– Given a query x, predict a ranking y.– Dependencies between results (e.g. avoid redundant hits)– Loss function over rankings (e.g. Average Precision)

SVMx 1. Kernel-Machines

2. SVM-Light3. Learning with Kernels4. SV Meppen Fan Club5. Service Master & Co.6. School of Volunteer Management7. SV Mattersburg Online…

• Multi-label Prediction• Protein Sequence Alignment• Noun Phrase Co-reference Clustering• Rankings in Information Retrieval• Inference in Graphical Models

• …and many more.

1st Order Sequence Labeling• Given:

– scoring function S(x, y1, y2)– input example x = (x1,…,xn)

• Finds sequence y = (y1,…,yn) to maximize

• Solved with dynamic programming (Viterbi)

“Hypothesis Function”

Some Formulation Restrictions

• Assume S is parameterized linearly by some weight vector w in RD.

• This means that

“Hypothesis Function”

Joint Feature Map

• From last slide:

• Joint feature map:

• Our hypothesis function:

ttt yyx ),,(),( 1xy

),(maxarg);( xyxy

“Linear Discriminant Function”

Structured Prediction Learning Problem

• Efficient Inference/Prediction

– Viterbi in sequence labeling– CKY Parser for parse trees– Belief Propagation for Markov random fields– Sorting for ranking

• Efficient Learning/Training – Learn parameters w from training data {xi,yi}i=1..N – Solution: use Structural SVM framework– Can also use Perceptrons, CRFs, MEMMs, M3Ns etc.

Conventional SVMs• Input: x (high dimensional point)• Target: y (either +1 or -1)• Prediction: sign(wTx)

• Training:

subject to:

• The sum of slacks upper bounds the 0/1 loss!

, 21minarg

iiT xwi 1)(y : i

Structural SVM• Let x denote a structured input (sentence)• Let y denote a structured output (POS tags)

• Standard objective function:

• Constraints are defined for each incorrect labeling y’ over each x.

iNCw 2

[Tsochantaridis et al., 2005]

Score(y(i)) Score(y’) Loss(y’) Slack

Interpreting Constraints

Suppose for incorrect y’:

iNCw 2

)'(75.0 yi

Score(y(i)) Score(y’) Loss(y’) Slack

[Tsochantaridis et al., 2005]

Adapting to Sequence Labeling• Minimize

subject to

• Sum of slacks upper bound loss.

ttt yyx ),,(),'( 1xy

iNCw 2

yy ttn '11)',( 1yy

Too many constraints!

Structural SVM Training

• The trick is to not enumerate all constraints.

• Suppose we only solve the SVM objective over a small subset of constraints (working set).– This is efficient– Equivalent to solving standard SVM.

• But some constraints from global set might be violated.

Structural SVM Training• STEP 1: Solve the SVM objective function using only working set of

constraints W.

• STEP 2: Using the model learned in STEP 1, find the most violated constraint from the global set of constraints.

• STEP 3: If the constraint returned in STEP 2 is violated by more than ε, add it to W.

• Repeat STEP 1-3 until no additional constraints are added. Return the most recent model that was trained in STEP 1.

STEP 1-3 is guaranteed to loop for at most O(1/epsilon) iterations. [Joachims et al., 2009]

*This is known as a “cutting plane” method.

Illustrative Example

Original SVM Problem• Exponential constraints• Most are dominated by a small set

of “important” constraints

Structural SVM Approach• Repeatedly finds the next most

violated constraint…• …until set of constraints is a good

approximation.

Finding Most Violated Constraint

• A constraint is violated when

• Finding most violated constraint reduces to

• Highly related to inference: “Loss augmented inference”

Sequence Labeling Revisited

• Finding most violated constraint…

… can be solved using Viterbi!

Structural SVM Recipe• Joint feature map

• Inference method

• Loss function

• Loss-augmented (most violated constraint)

),(maxarg xyy

),( xy

)'(),(maxarg'

Structural SVMs for Rankings

• Predicting rankings important in IR

• Must provide the four ingredients:– How to represent joint feature map Ψ?– How to perform inference?– What loss function to optimize for?– How to find most violated constraint?

Joint Feature Map for Ranking• Let x = (x1,…xn) denote candidate documents• Let yjk = {+1, -1} encode pairwise rank orders

• Feature map is pairwise feature difference of documents.

• Inference made by sorting on document scores wTxi

Multivariate Loss Functions• Information Retrieval focused on ranking-centric

performance measures– Normalized Discounted Cumulative Gain– Precision @ K– Mean Average Precision– Expected Reciprocal Rank

• These measures all depend on the entire ranking

Multivariate Loss Functions• Information Retrieval focused on ranking-centric

performance measures– Normalized Discounted Cumulative Gain– Precision @ K– Mean Average Precision– Expected Reciprocal Rank

• These measures all depend on the entire ranking

Mean Average Precision• Consider rank position of each relevant doc

– K1, K2, … KR

• Compute Precision@K for each K1, K2, … KR

• Average precision = average of P@K

• Ex: has AvgPrec of

• MAP is Average Precision across multiple queries/rankings

76.053

[Yue & Burges, 2007]

Structural SVM for MAP• Minimize

subject to

where ( yjk = {-1, +1} )

• Sum of slacks is smooth upper bound on MAP loss.

relj relk

ii xxyjk

)()()()( )(),( xy

iNCw 2

iiiTiiTi ww )'(),'(),( :' )()()()( yxyxyyy

)'(1)'( yy Avgprec

[Yue et al., SIGIR 2007]

Too Many Constraints!• For Average Precision, the true labeling is a ranking where the

relevant documents are all ranked in the front, e.g.,

• An incorrect labeling would be any other ranking, e.g.,

• This ranking has Average Precision of about 0.8 with (y’) ≈ 0.2

• Intractable number of rankings, thus an intractable number of constraints!

Observations• MAP is invariant on the order of documents within a relevance

class– Swapping two relevant or non-relevant documents does not change MAP.

• Joint SVM score is optimized by sorting by document score, wTxj

• Reduces to finding an interleaving between two sorted lists of documents

relj relk

jk xwxwy: :!'

)(')'(maxarg yy

• Start with perfect ranking• Consider swapping adjacent

relevant/non-relevant documents

relj relk

jk xwxwy: :!'

)(')'(maxarg yy

relevant/non-relevant documents• Find the best feasible ranking of

the non-relevant document

relj relk

jk xwxwy: :!'

)(')'(maxarg yy

relevant/non-relevant documents• Find the best feasible ranking of the

non-relevant document• Repeat for next non-relevant

document

relj relk

jk xwxwy: :!'

)(')'(maxarg yy

document• Never want to swap past previous

non-relevant document

relj relk

jk xwxwy: :!'

)(')'(maxarg yy

document• Never want to swap past previous

non-relevant document• Repeat until all non-relevant

documents have been considered

relj relk

jk xwxwy: :!'

)(')'(maxarg yy

Comparison with other SVM methods

TREC 9 Indri TREC 10 Indri TREC 9Submissions

TREC 10Submissions

TREC 9Submissions(without best)

TREC 10Submissions(without best)

Dataset

sion SVM-MAP

SVM-ROCSVM-ACCSVM-ACC2SVM-ACC3SVM-ACC4

Need for Diversity (in IR)• Ambiguous Queries

– Different information needs using same query– “Jaguar”– At least one relevant result for each information need

• Learning Queries– User interested in “a specific detail or entire breadth

of knowledge available” • [Swaminathan et al., 2008]

– Results with high information diversity

• Choose top 3 documents• Individual Relevance: D3 D4 D1• Greedy Coverage Solution: D3 D1 D5

Example

D1 D2 D3 Best

Iter 1 12 11 10 D1Iter 2

Marginal Benefit

V1 V2 V3 V4 V5D1 X X XD2 X X XD3 X X X X

Word Benefit

V1 1V2 2V3 3V4 4V5 5

Document Word Counts

Example

D1 D2 D3 Best

Iter 1 12 11 10 D1Iter 2 -- 2 3 D3

Marginal Benefit

V1 V2 V3 V4 V5D1 X X XD2 X X XD3 X X X X

Word Benefit

V1 1V2 2V3 3V4 4V5 5

Document Word Counts

Prior Work• Essential Pages [Swaminathan et al., 2008]

– Uses fixed function of word benefit– Depends on word frequency in candidate set

– - Local version of TF-IDF

– - Frequent words low weight– (not important for diversity)

– - Rare words low weight– (not representative)

Joint Feature Map• x = (x1,x2,…,xn) - candidate documents • y – subset of x • V(y) – union of words from documents in y.

• (v,x) – frequency features (e.g., >10%, >20%, etc).• Benefit of covering word v is then wT(v,x)

),(),(y

TT vww

[Yue & Joachims, ICML 2008]

Joint Feature Map

• Does NOT reward redundancy – Benefit of each word only counted once

• Greedy inference has (1-1/e)-approximation bound– Due to h(x) being monotone submodular

• Linear (joint feature space) – Allows for SVM optimization

• (Used more sophisticated discriminant in experiments.)

Weighted Subtopic Loss• Example:

– x1 covers t1

– x2 covers t1,t2,t3

– x3 covers t1,t3

• Motivation– Higher penalty for not covering popular subtopics

# Docs Loss

t1 3 1/2

t2 1 1/6

t3 2 1/3

• Encode each subtopic as an additional “word” to be covered.

• Use greedy algorithm:

• TREC 6-8 Interactive Track• Retrieving 5 documents

0.469 0.472 0.471

0.30.320.340.360.38

0.40.420.440.460.48

Random Okapi Unweighted Essential Pages SVM-div

Sentiment Classification

• Product reviews• Movie reviews• Political speeches• Discussion forums

• What is the sentiment, and why?– What are the supporting sentences?

Identifying the Supporting Sentences

• Suppose we could extract the supporting sentences

• How can we do this automatically?

87% accuracy 98% accuracy

Joint Feature Map• x = (x1,x2,…,xn) – sentences of an article• s = subset of supporting sentences• y = sentiment label {+1,-1}

[Yessenalina, Yue & Cardie, EMNLP 2010]

Inference

h(x|w) =

[Yessenalina, Yue & Cardie, EMNLP 2010]

Latent Variable SVMs• Input: x = (x1,x2,…,xn) – sentences of an article• Output: y (either +1 or -1), s (supporting sentences)

• Training:

subject to:

, 21minarg

Not convex!

[Yu & Joachims, 2009] [Yessenalina, Yue & Cardie, 2010]

Training Using CCCP• First initialize s(i) for each training instance

• (1) train:

• (2) infer s(i) from w:

• Repeat (1) and (2) …

, 21minarg

[Yu & Joachims, 2009] [Yessenalina, Yue & Cardie, 2010]

Requires finding most violated constraint

Initialization Method AccuracyBaseline Standard SVM 88.6%

Annotated Rationales Zaidan et al., 2007 92.2%

SVM-sle 93.2%Opinion Finder Yessenalina et al., 2009 91.8%

SVM-sle 92.5%

Initialization Method AccuracyBaseline Standard SVM 70.0%

Opinion Finder Thomas et al., 2006 71.3%

Bansal et al., 2008 75.0%

SVM-sle 77.7%

Movie Reviews Corpus

Congressional Debates Corpus

Green sentences denote most positive supporting sentencesUnderlined sentences denote least subjective sentences

Structural SVMs• Train parameterized structured prediction models.• Requires 4 ingredients

– Joint feature map (linear)– Inference– Loss function– Finding most violated constraint

• Applications in information retrieval– Optimizing Mean Average Precision in rankings– Optimizing diversity– Predicting sentiment with latent explanations

Work supported by NSF IIS-0713483, Microsoft Fellowship, and Yahoo! KSC Award.

An Introduction to Structural SVMs and its Application to Information Retrieval

Documents

Learning Ranking Functions with SVMs

ICML2012: Exemplar-SVMs for Visual Object Detection , Label Transfer and Image Retrieval

Group Norm for Learning Latent Structural SVMs

Asset 9110 Horse Svms 2057

Linear Classifiers/SVMs

Introduction to SVMs. SVMs Geometric –Maximizing Margin Kernel Methods –Making nonlinear decision boundaries linear –Efficiently! Capacity –Structural

Structural SVMs 및 Pegasos 알고리즘을 이용한 한국어 개체명 인식leeck/NLP2/01_intro_NLP.pdf · 2017-09-11 · • NLP datasets are high dimensional – One-hot representation

Large Scale Transductive SVMs - bottou.org

9.520 Class 06, 24 February 2003 Ryan Rifkin...Ryan Rifkin Plan Regularization derivation of SVMs Geometric derivation of SVMs Optimality, Duality and Large Scale SVMs SVMs and RLSC:

SVMs for (x) Recognition (From Moghaddam / Yang’s “Gender Classification with SVMs”) Brian Whitman

An Idiot’s guide to Support vector machines (SVMs) · machines (SVMs) R. Berwick, Village Idiot SVMs: A New Generation of Learning Algorithms •Pre 1980: –Almost all learning

SVMs in R tutorial

Learning Structural SVMs with Latent Variablesdanroth/Teaching/CIS-700-006/slides/lat… · Learning Structural SVMs with Latent Variables ICML, 2009 Tsochantaridis et al. Support

Welcome to SVMS Welcome to SVMS Open House 2014 Cougar Team

Cat SVMS 2068

An Introduction to Structural SVMs and its Application to Information Retrieval Yisong Yue Carnegie Mellon University

Learning Structural SVMs with Latent VariablesLearning Structural SVMs with Latent Variables Chun-Nam Yu Dept. of Computer Science, Cornell University October 8-9, IBM SMiLe Workshop

Introduction to SVMs

Con - SVMs

Structured Perceptrons& Structural SVMs · Structured Perceptrons& Structural SVMs CS 159: Advanced Topics in Machine Learning 1 4/6/2017