View
73
Download
0
Category
Preview:
DESCRIPTION
Probabilistic Lexical Models for Textual Inference Eyal Shnarch , Ido Dagan, Jacob Goldberger. The entire talk in a single sentence. lexical textual inference. principled probabilistic model . improves state-of-the art. Outline. lexical textual inference. - PowerPoint PPT Presentation
Citation preview
Bar Ilan University @ IBM July 2012
Probabilistic Lexical Models for Textual Inference
Eyal Shnarch, Ido Dagan, Jacob Goldberger
1/34
Bar Ilan University @ IBM July 2012
The entire talk in a single sentencewe address
with a
which
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
2/34
Bar Ilan University @ IBM July 2012
Outlinewe address
with a
which
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3
3/34
Bar Ilan University @ IBM July 2012
we address
with a
which
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3
4/34
Bar Ilan University @ IBM July 2012
Textual inference – useful in many NLP apps
improves state-of-the-artprincipled probabilistic model
in Belgium Napoleon was defeated
In the Battle of Waterloo, 18 Jun 1815,
the French army, led by
Napoleon, was crushed.
Napoleon was
Emperor of the
French from 1804 to
1815.
lexical textual inference
Napoleon was not tall enough to win
the Battle of Waterloo
At Waterloo Napoleon did surrender...Waterloo - finally facing my Waterloo
Napoleon engaged in a series of wars, and won many
5/34
Bar Ilan University @ IBM July 2012
BIU NLP lab
Chaya Liebeskind
6/34
Bar Ilan University @ IBM July 2012
Lexical textual inference
• Complex systems use parser
• Lexical inference rules link terms from T to H• Lexical rules come from lexical resources• H is inferred from T iff all its terms are inferred
Improves state-of-the-artprincipled probabilistic model lexical textual inference
In the Battle of Waterloo, 18 Jun 1815, the French army, led by Napoleon, was crushed.
in Belgium Napoleon was defeatedText Hypothesis
7/34
1st or 2nd order co-occurrence
Bar Ilan University @ IBM July 2012
Textual inference for ranking
Improves state-of-the-artprincipled probabilistic model lexical textual inference
In which battle was Napoleon defeated?
In the Battle of Waterloo, 18 Jun 1815,
the French army, led by
Napoleon, was crushed.
Napoleon was
Emperor of the
French from 1804 to
1815.
Napoleon was not tall enough to win
the Battle of Waterloo
At Waterloo Napoleon did surrender...Waterloo - finally facing my Waterloo
Napoleon engaged in a series of wars, and won many
12
3
4
5
a
bc
d
e
8/34
Bar Ilan University @ IBM July 2012
Ranking textual inference – prior work
Improves state-of-the-artprincipled probabilistic model lexical textual inference
• Transform T’s parsed tree into H’s parsed tree• Based on principled ML model(Wang et al. 07, Heilman and Smith 10, Wang and Manning 10)
Syntactic-based
methods
• Fast, easy to implement, highly competitive• Practical across genres and languages(MacKinlay and Baldwin 09, Clark and Harrison 10,
Majumdar and Bhattacharyya 10)
Heuristic lexical
methods
9/34
Bar Ilan University @ IBM July 2012
Lexical entailment scores – current practice
• Count covered/uncovered• (Majumdar and Bhattacharyya, 2010; Clark and Harrison, 2010)
• Similarity estimation• (Corley and Mihalcea, 2005; Zanzotto and Moschitti, 2006)
• Vector space• (MacKinlay and Baldwin, 2009)
Mostly heuristic
10/34
principled probabilistic
model
Bar Ilan University @ IBM July 2012
we address
with a
which
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3
11/34
Bar Ilan University @ IBM July 2012
Probabilistic model – overview
Improves state-of-the-artprincipled probabilistic model lexical textual inference
T
H which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushed
)( HTP
knowledge integration
term-level
sentence-level
)( 3hTP )( 1hTP
t1 t2 t3 t4 t5 t6
h1 h2 h3
)( 2hTP
annotations are available at
sentence-level only
x1 x2 x3
12/34
Bar Ilan University @ IBM July 2012
Knowledge integration
• Distinguish resources reliability levels• WordNet >> similarity-based thesauri (Lin, 1998; Pantel and Lin, 2002)
• Consider transitive chains length• The longer a chain is the lower its probability
• Consider multiple pieces of evidence• More evidence means higher probability
which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushed
t
rule1
rule2
transitive chainr
is a rulemultiple evidence
13/34
Bar Ilan University @ IBM July 2012
Probabilistic model – term level
Improves state-of-the-artprincipled probabilistic model lexical textual inference
T
H which battle was Napoleon defeated
)(rR
chainr
rR
chainhtP )()(
Battle of Waterloo French army led by Napoleon was crushed
ORt'
ris a rule
is the reliability level of the
resource which suggested r
1)( hTP
ACL 11 short paper this level parameters: one per input lexical resource
t1 t2 t3 t4 t5 t6
h1 h2 h3
multiple evidence
)(
)](1[hchainsc
chtP
14/34
𝜃𝑊𝑁
𝜃𝑊𝑖𝑘𝑖
Bar Ilan University @ IBM July 2012
Probabilistic model – overview
Improves state-of-the-artprincipled probabilistic model lexical textual inference
T
H which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushed
)( HTP
knowledge integration
term-level
sentence-level
)( 3hTP )( 2hTP )( 1hTP
15/34
Bar Ilan University @ IBM July 2012
Probabilistic model – sentence level
Improves state-of-the-artprincipled probabilistic model lexical textual inference
we define hidden binary random variables:
xt = 1 iff ht is inferred from T (zero otherwise)
H which battle was Napoleon defeated
h1 h2 h3
x1 x2 x3
)( 3hTP )( 2hTP )( 1hTP
final sentence-level decision
AND
y
Modeling with AND gate:• Most intuitively• However
• Too strict• Does not model terms dependency
16/34
Bar Ilan University @ IBM July 2012
Probabilistic model – sentence level
Improves state-of-the-artprincipled probabilistic model lexical textual inference
),|()( 1 jxiykyPkq tttij
this level parameters
}1,0{,, kji
H which battle was Napoleon defeated
h1 h2 h3
x1 x2 x3
y1 y2 y3
)( 3hTP )( 2hTP )( 1hTP
final sentence-level decision
we define another binary random variable:
yt – inference decision for the prefix h1… ht
P(yt = 1) is dependent on yt-1 and xt
M-PLM
xt = 1 iff ht is inferred by T (zero otherwise)
17/34
Bar Ilan University @ IBM July 2012
M-PLM – inference
Improves state-of-the-artprincipled probabilistic model lexical textual inference
)1(),|()()()1(2
11
n
tttttn xyyPxPxPyP
1,,,,,
12
1
nn
n
yyyxx
H which battle was Napoleon defeated
h1 h2 h3
x1 x2 x3
y1 y2 y3
)( 3hTP )( 2hTP )( 1hTP
final sentence-level decision
qij(k)
)2()()()()()(}1,0{,
1
ji
ijtttt kqjxPikyPk
)3()()( 11 kxPk
can be computed efficiently with a forward algorithm
)4()1()1( nnyP
18/34
Bar Ilan University @ IBM July 2012
M-PLM – summary
Annotation final sentence-level decision
Improves state-of-the-artprincipled probabilistic model lexical textual inference
Parameters
resource Observed
Lexical rules which link terms
Learning we developed EM
scheme to jointly learn all parameters
Hidden
19/34
Bar Ilan University @ IBM July 2012
so how our model does?
Improves state-of-the-artprincipled probabilistic model lexical textual inference
In which battle was Napoleon defeated?
In the Battle of Waterloo, 18 Jun 1815,
the French army, led by
Napoleon, was crushed.
Napoleon was
Emperor of the
French from 1804 to
1815.
Napoleon was not tall enough to win
the Battle of Waterloo
At Waterloo Napoleon did surrender...Waterloo - finally facing my Waterloo
Napoleon engaged in a series of wars, and won many
12
3
4
5
20/34
Bar Ilan University @ IBM July 2012
we address
with a
which
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3
21/34
Bar Ilan University @ IBM July 2012
Evaluations – data sets
improves sate-of-the-artprincipled probabilistic model lexical textual inference
Ranking in passage retrieval for QA
(Wang et al. 07)
5700/1500 question-candidate answer pairs from TREC 8-13
Manually annotated
Notable line of work from recent years: Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilman and Smith 10, Wang and Manning 10
Recognizing textual entailment
within a corpus
20,000 text-hypothesis pairs in each RTE-5, RTE-6
Originally constructed for classification
22/34
Bar Ilan University @ IBM July 2012
Evaluations – baselines
Syntactic generative models• Require parsing• Apply sophisticated machine learning methods(Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilman and Smith 10, Wang and Manning 10)
Lexical model – Heuristically Normalized-PLM• AND-gate for the sentence-level• Add heuristic normalizations to addresses its disadvantages (TextInfer
workshop 11)
• Performance in line with best RTE systems
improves sate-of-the-artprincipled probabilistic model lexical textual inference
HN-PLM
23/34
Bar Ilan University @ IBM July 2012
QA results – syntactic baselines
improves sate-of-the-artprincipled probabilistic model lexical textual inference
MAP MRR40
45
50
55
60
65
70
60.91
69.51
Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and Smith
24/34
Bar Ilan University @ IBM July 2012
QA results – syntactic baselines + HN-PLM
improves sate-of-the-artprincipled probabilistic model lexical textual inference
MAP MRR40
45
50
55
60
65
70
60.91
69.51
Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and SmithHN-PLM
+0.7%
+1%
25/34
Bar Ilan University @ IBM July 2012
QA results – baselines + M-PLM
improves sate-of-the-artprincipled probabilistic model lexical textual inference
MAP MRR40
45
50
55
60
65
70
60.91
64.38
72.69
69.51
Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and SmithHN-PLMM-PLM
+3.2%
+3.5%
M-PLM
26/34
Bar Ilan University @ IBM July 2012
RTE results – M-PLM vs. HN-PLM
improves sate-of-the-artprincipled probabilistic model lexical textual inference
RTE-5 MAP RTE-5 MRR RTE-6 MAP RTE-6 MRR40
45
50
55
60
65
70
75
80
85
HN-PLMM-PLM
+7.3%
+1.9%
+6.0%+3.6%
27/34
Bar Ilan University @ IBM July 2012
First approach - summary
Clean probabilistic lexical model• As a lexical component or as a stand alone inference system• Superiority of principled methods over heuristic ones • Attractive passage retrieval ranking method• Code available - BIU NLP downloads
M-PLM limits• Processing is term order dependent• Lower performance on classification vs. HN-PLM
does not normalize well across hypotheses length
improves state-of-the-artprincipled probabilistic model lexical textual inference
28/34
Bar Ilan University @ IBM July 2012
we
addr
ess
with
a
whi
ch
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3A
(ver
y) n
ew
4
second approach:
resources as observers
29/34
Bar Ilan University @ IBM July 2012
each resource is a witness
which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6
h1 h2 h3
t'
30/34
Bar Ilan University @ IBM July 2012
Bottom-up witnesses model
which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6
h1 h2 h3
x1 x2 x3
AND
y
𝑃 (𝑊 (h𝑖 )|𝑥 𝑖=1 )= ∏𝑤∈𝑊 (h𝑖)
𝜃𝑤 ⋅ ∏𝑤∉𝑊 (h𝑖)
(1−𝜃𝑤)
𝜂0≝𝑃 (𝑥 𝑖=1∨𝑦=0)𝜂1≝𝑃 (𝑥𝑖=1∨𝑦=1)
𝜃𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥 𝑖=1)𝜏𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥𝑖=0) 𝑃 (𝑊 (h𝑖 )|𝑥 𝑖=0 )= ∏
𝑤∈𝑊 (h𝑖)𝜏𝑤 ⋅ ∏
𝑤∉𝑊 (h𝑖)(1−𝜏𝑤)
Likelihood
31/34
Bar Ilan University @ IBM July 2012
Advantages of the second approach
Inference:
• Hypothesis length is not an issue• Learn from non-entailing resources• and provide a recall and precision estimation for a resource
¿𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)
𝑃 (𝑊 (𝐻 ))
¿𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)
𝑃 (𝑊 (𝐻 )|𝑦=0 ) ⋅ 𝑃 ( 𝑦=0 )+𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)
32/34
Bar Ilan University @ IBM July 2012
(near) future plans
• Context model• There are other languages than English
• Deploy the new version of a Wikipedia-base lexical resource with the Italian dump
• Test the probabilistic lexical models for other languages• Cross language textual entailment
33/34
Bar Ilan University @ IBM July 2012
Cross Language Textual Entailment
quale battaglia fu sconfitto Napoleone
Battle of Waterloo French army led by Napoleon was crushed
Italian monolingual
English-Italian phrase table
English monolingual
Thank
You
34/34
Bar Ilan University @ IBM July 2012 35/34
Bar Ilan University @ IBM July 2012
Demo examples:
[Bap,WN] no transitivityJack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid
[WN,Wiki] <show graph>Barak Obama's Buick got stuck in Dublin in a large Irish crowdUnited_States_President's car got stuck in Ireland, surrounded by many people
Barak Obama - WN is out of date, need a new version of Wikipedia
Bill_Clinton's Buick got stuck in Dublin in a large Irish crowdUnited_States_President's car got stuck in Ireland, surrounded by many people
------------------------------------------------------------------------------[Bap,WN] this time with <transitivity & multiple evidence> Jack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid
[VO,WN,Wiki]in the Battle_of_Waterloo the French army led by Napoleon was crushedin which battle Napoleon was defeated?
------------------------------------------------------------------------------[all]1. in the Battle_of_Waterloo the French army led by Napoleon was crushed 72%
2. Napoleon was not tall enough to win the Battle_of_Waterloo 47%
3. at Waterloo Napoleon did surrender... Waterloo - finally facing my Waterloo 34%
4. Napoleon engaged in a series of wars, and won many 47%
5. Napoleon was Emperor of the French from 1804 to 18159% [a bit long run]
36/34
Recommended