Upload
basil-hall
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
Statistical Translation and Web Search Ranking
Jianfeng GaoNatural language processing, MSR
July 22, 2011
Who should be here?
• Interested in statistical machine translation and Web search ranking
• Interested in modeling technologies• Look for topics for your master/PhD thesis– A difficult topic: very hard to beat a simple
baseline– An easy topic: others cannot beat it either
3
Outline
• Probability• Statistical Machine Translation (SMT)• SMT for Web search ranking
Probability (1/2)
• Probability space:
– Cannot say • Joint probability:
– Probability that x and y are both true• Conditional probability:
– Probability that y is true when we already know x is true• Independence:
– x and y are independent
Probability (2/2)
• : assumptions on which the probabilities are based
• Product rule –from the def of conditional probability
• Sum rule – a rewrite of the marginal probability def
• Bayes rule – from the product rule
An example: Statistical Language Modeling
Statistical Language Modeling (SLM)
• Model form– capture language structure via a probabilistic
model
• Model parameters– estimation of free parameters using training data
8
Model Form
• How to incorporate language structure into a probabilistic model
• Task: next word prediction– Fill in the blank: “The dog of our neighbor ___”
• Starting point: word n-gram model– Very simple, yet surprisingly effective– Words are generated from left-to-right– Assumes no other structure than words
themselves
9
Word N-gram Model
• Word based model– Using chain rule on its history (=preceding words)
Word N-gram Model• How do we get probability estimates?
– Get text and count!
• Problem of using the whole history– Rare events: unreliable probability estimates– Assuming a vocabulary of 20,000 words, model # parametersunigram P(w1) 20,000bigram P(w2|w1) 400Mtrigram P(w3|w1w2) 8 x 1012
fourgram P(w4|w1w2w3) 1.6 x 1017
From Manning and Schütze 1999: 194
11
Word N-gram Model • Markov independence assumption
– A word depends only on N-1 preceding words– N=3 → word trigram model
• Reduce the number of parameters in the model– By forming equivalence classes
• Word trigram model
𝑃 (𝑤𝑖|¿ 𝑠>𝑤1𝑤2…𝑤𝑖− 2𝑤𝑖 −1 )=𝑃 (𝑤𝑖|𝑤𝑖 −2𝑤 𝑖−1 ¿
...
12
Model Parameters
• Bayesian estimation paradigm• Maximum likelihood estimation (MLE)• Smoothing in N-gram language models
13
Bayesian Paradigm
– – Posterior probability– – Likelihood– – Prior probability– – Marginal probability
• Likelihood versus probability– for fixed , defines a probability over ; – for fixed , defines the likelihood of .
• Never say “the likelihood of the data”• Always say “the likelihood of the parameters given the
data”
14
Maximum Likelihood Estimation (MLE)
• : model; : data
– Assume a uniform prior – is independent of , and is dropped
– where is the likelihood of parameter• Key difference between MLE and Bayesian Estimation
– MLE assume that is fixed but unknown, – Bayesian estimation assumes that itself is a random
variable with a prior distribution
15
MLE for Trigram LM
• It is easy – let us get some real text and start to count
But, why is this the MLE solution?
16
Derivation of MLE for N-gram
• Homework – an interview question of MSR • Hints– This is a constrained optimization problem– Use log likelihood as objective function– Assume a multinomial distribution of LM– Introduce Lagrange multiplier for the constraints
17
Sparse Data Problem
• Say our vocabulary size is |V|• There are |V|3 parameters in the trigram LM– |V| = 20,000 20,0003 = 8 1012 parameters
• Most trigrams have a zero count even in a large text corpus
– oops…
18
Smoothing: Adding One
• Add one smoothing (from Bayesian paradigm)• But works very badly – do not use this
• Add delta smoothing• Still very bad – do not use this
19
Smoothing: Backoff
• Backoff trigram to bigram, bigram to unigram
D(0,1) is a discount constant – absolute discount α is calculated so probabilities sum to 1 (homework)
• Simple and effective – use this one!
20
Outline
• Probability• SMT and translation models• SMT for web search ranking
SMT
and
C: 救援 人员 在 倒塌的 房屋 里 寻找 生还者E: Rescue workers search for survivors in collapsed houses
𝑃 (𝐸∨𝐶)• Translation process (generative story)– C is broken into translation units– Each unit is translated into English– Glue translated units to form E
• Translation models– Word-based models– Phrase-based models– Syntax-based models
Generative Modeling
Art
Science
Engineering
Story
Math
Code
Generative Modeling for
• Story making– how a target sentence is generated from a source
sentence step by step• Mathematical formulation– modeling each generation steps in the generative
story using a probability distribution• Parameter estimation– implementing an effective way of estimating the
probability distributions from training data
Word-Based Models: IBM Model 1
• We first choose the length for the target sentence , according to the distribution .
• Then, for each position in the target sentence, we choose a position in the source sentence from which to generate the -th target word according to the distribution
• Finally, we generate the target word by translating according to the distribution .
Mathematical Formulation
• Assume that the choice of the length is independent of and
• Assume that all positions in the source sentence are equally likely to be chosen
• Assuming that each target word is generated independently from
Parameter Estimation
• Model Form
• MLE on word-aligned training data
• Don’t forget smoothing
Phrase-Based Models
Mathematical Formulation
• Assume a uniform probability over segmentations
• Use the maximum approximation to the sum
• Assume each phrase being translated independently and use distance-based reordering model
Parameter Estimation
MLE: Don’t forget smoothing
Syntax-Based Models
Story
• Parse an input Chinese sentence into a parse tree
• Translate each Chinese constituent into English– VP (PP 寻找 NP, search for NP PP)
• Glue these English constituents into a well-formed English sentence.
Other Two Tasks?
• Mathematical formation– Based on synchronous context free grammar (SCFG)
• Parameter estimation– Learning SCFG from data
• Homework • Let us go thru an example (thanks to Michel
Galley)– Hierarchical phrase model– Linguistically syntax-based models
rescue
workers
for
survivors
in
collapsed
houses
search
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
倒塌 的 房屋 collapsed houses
rescue
workers
for
survivors
in
collapsed
houses
search
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
search for survivors in collapsed houses在 倒塌 的 房屋 里 寻找 生还者
rescue
workers
for
survivors
in
collapsed
houses
search
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
search for survivors in collapsed houses在 倒塌 的 房屋 里 寻找 生还者
A synchronous rule
• Phrase-based translation unit• Discontinuous translation unit• Control on reordering
里 寻找 在
A synchronous grammar
里 寻找 在
倒塌 的 房屋
生还者
Context-free derivation:
search for survivors in collapsed houses在 倒塌 的 房屋 里 寻找 生还者
search for in collapsed houses在 倒塌 的 房屋 里 寻找
search for in 在 里 寻找
A synchronous grammar
里 寻找 在
倒塌 的 房屋
生还者
search for survivors in collapsed housesRecognizes:
search for collapsed houses in survivors
search for survivors collapsed houses in
NNS
PP
VP
VBP
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
Rescue workers search for survivors in collapsed houses.
rescue staff in collapse of house in search survivors
JJ NNS
NPPP
PP
NNS
NP
PP
VP
VBP
IN
NNS
NP
S
NN
NNS
PP
VP
VBP
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
Rescue workers search for survivors in collapsed houses.
rescue staff in collapse of house in search survivors
JJ NNS
NPPP
PP
NNS
NP
PP
VP
VBP
IN
NNS
NP
S
NN
NNS
PP
VP
VBP
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
Rescue workers search for survivors in collapsed houses.
rescue staff in collapse of house in search survivors
JJ NNS
NPPP
PP
NNS
NP
PP
VP
VBP
IN
NNS
NP
S
NN
PP
VP
VBP
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
Rescue workers search for survivors in collapsed houses.
rescue staff in collapse of house in search survivors
JJ NNS
NPPP
PP
NNS
NP
PP
VP
VBP
IN
VP
VP
NP寻找VBP
PP
PPIN
PP NPsearch for
PP
VP
VBP
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
Rescue workers search for survivors in collapsed houses.
rescue staff in collapse of house in search survivors
JJ NNS
NPPP
PP
NNS
NP
PP
VP
VBP
IN
VP-234 NP-57寻找 PP-32PP-32 NP-57search for
SCFG rule:
NNS
PP
VP
VBP
救援 人员 在 倒塌 的 房屋 里 寻找 生还者
Rescue workers search for survivors in collapsed houses.
rescue staff in collapse of house in search survivors
JJ NNS
NPPP
PP
NNS
NP
PP
VP
VBP
IN
NNS
NP
S
NN
47
Outline
• Probability• SMT and translation models• SMT for web search ranking
Web Documents and Search Queries
• cold home remedy • cold remeedy• flu treatment• how to deal with
stuffy nose?
Map Queries to Documents• Fuzzy keyword matching
– Q: cold home remedy– D: best home remedies for cold and flu
• Spelling correction– Q: cold remeedies– D: best home remedies for cold and flu
• Query alteration– Q: flu treatment– D: best home remedies for cold and flu
• Query/document rewriting– Q: how to deal with stuffy nose– D: best home remedies for cold and flu
• Where are we now?
Research Agenda (Gao et al. 2010, 2011)
• Model documents and queries as different languages (Gao et al., 2010)
• Cast mapping queries to documents as bridging the language gap via translation
• Leverage statistical machine translation (SMT) technologies and infrastructures to improve search relevance
Are Queries and Docs just Different Languages?
• A large scale analysis, extending (Huang et al. 2010)
• Divide web collection into different fields, e.g., queries, anchor text, titles, etc.
• Develop a set of language models, each on one n-gram datasets from a different field
• Measure language difference between different fields (queries/docs) via perplexity
Microsoft Web N-gram Model Collection (cutoff = 0)
• Microsoft web n-gram services. http://research.microsoft.com/web-ngram
Perplexity Results
• Test set– 733,147 queries from the May 2009 query log
• Summary– Query LM is most predictive of test queries– Title is better than Anchor in lower order but is worse in higher
order– Body is in a different league
SMT for Document Ranking• Given a query (q), doc (d) can be ranked by
how likely it is that q is rewritten from d,
• An example: phrasal statistical translation for Web document ranking
how to deal with stuffy nose?
Phrasal Statistical Translation for Rankingd: “cold home remedies” titleS: [“cold”, “home remedies”] segmentationT: [“stuffy nose”, “deal with”] translationM: (1 2, 2 1) permutationq: “deal with stuffy nose” query
• Uniform probability over S: • Maximum approximation:
• Max probability assignment via dynamic programming: and
• Model training on query-doc pairs
Mine Query-Document Pairs from User Logs
http://www.agelessherbs.com/BestHomeRemediesColdFlu.html
NO CLICK
NO CLICK
how to deal with stuffy nose?
stuffy nose treatment
cold home remedies
Mine Query-Document Pairs from User Logs
how to deal with stuffy nose?
stuffy nose treatment
cold home remedies
Mine Query-Document Pairs from User Logs
how to deal with stuffy nose?
stuffy nose treatment
cold home remedies
QUERY (Q) Title (T)how to deal with stuffy nose best home remedies for cold and flustuffy nose treatment best home remedies for cold and flucold home remedies best home remedies for cold and flu… … … …go israel forums goisrael communityskate at wholesale at pr wholesale skates southeastern skate supplybreastfeeding nursing blister baby clogged milk ducts babycenterthank you teacher song lyrics for teaching educational children s musicimmigration canada lacolle cbsa office detailed information
• 178 million pairs from 0.5 year log
Evaluation Methodology
• Measurement: NDCG, t-test• Test set: – 12,071 English queries sampled from 1-y log– 5-level relevance label for each query-doc pair– On a tail document sets (click field is empty)
• Training data for translation models:– 82,834,648 query-title pairs
Baseline: Word-Based Models (Berger&Lafferty, 99)
• Basic model:• Mixture model:
• Learning translation probabilities from clickthrough data– IBM Model 1 with EM
ResultsSample IBM-1 word translation probability after EM training on the Query-title pairs
Bilingual Phrases
• Notice that with context information, we have less ambiguous translations
Results
• Ranking results– All features
– Only phrase translation features
Why Do Bi-Phrases Help?
• Length distribution
• Good/bad examples
Generative Topic Models
• Probabilistic latent Semantic Analysis (PLSA)
– d is assigned a single most likely topic vector– q is generated from the topic vectors
• Latent Dirichlet Allocation (LDA) generalizes PLSA– a posterior distribution over topic vectors is used– PLSA = LDA with MAP inference
Q: stuffy nose treatment D: cold home remediesTopic
Q: stuffy nose treatment D: cold home remedies
Bilingual Topic Model
• For each topic z: • For each q-d pair: • Each q is generated by and • Each w is generated by and
Log-likelihood of LDA Given Data
• : distribution of distribution• LDA requires integral over • This is the MAP approximation to LDA
MAP Estimation via EM• Estimate by maximizing joint log likelihood of
q-d pairs and the parameters• E-Step: compute posterior probabilities– ,
• M-Step: update parameters using the posterior probabilities– ,,
Posterior Regularization (PR)
• q and its clicked d are relevant, thus they– Share same prior distribution over topics (MAP)– Weight each topic similarly (PR)
• Model training via modified EM– E-step: for each q-d pair, project the posterior topic
distributions onto a constrained set, where the expected fraction of each topic is equal in q and d
– M-step: update parameters using the projected posterior probabilities
Topic Models for Doc Ranking
Evaluation Methodology
• Measurement: NDCG, t-test• Test set: – 16,510 English queries sampled from 1-y log– Each query is associated with 15 docs– 5-level relevance label for each query-doc pair
• Training data for translation models:– 82,834,648 query-title pairs
Topic Model Results
Summary
• Probability– Basics– A case study of a probabilistic model: N-gram language model
• Statistical Machine Translation (SMT)– Generative modeling (story math code)– Word/phrase/syntax based models
• SMT for web search ranking– View query and doc as different language– Doc ranking via – Word/phrase/topic based models
• Slides/doc will be available at http://research.microsoft.com/~jfgao/
Main Reference• Berger, A., and Lafferty, J. 1999. Information retrieval as statistical translation.
In SIGIR, pp. 222-229.• Gao, J., He, X., and Nie, J-Y. 2010. Clickthrough-based translation models for
web search: from word models to phrase models. In CIKM, pp. 1139-1148.• Gao, J., Toutanova, K., and Yih, W-T. 2011. Clickthrough-based latent semantic
models for web search. In SIGIR.• Huang, J., Gao, J., Miao, J., Li, X., Wang, K., and Behr, F. 2010. Exploring web
scale language models for search query processing. In Proc. WWW 2010, pp. 451-460.
• MacKay, David J. C. 2003. Information Theory, Inference and Learning Algorithms. Cambridge: Cambridge University Press.
• Manning, C., and H. Chutze. 1999. Foundations of statistical natural language processing. MIT Press. Cambridge.
• Philipp Koehn. Statistical Machine Translation. Cambridge University Press. 2009.