Axiomatic Analysis of Smoothing Methods in Language Models for Pseudo-Relevance Feedback

10/1/2015

Axiomatic Analysis of Smoothing Methods in Language Models for Pseudo-Relevance Feedback

HUSSEIN HAZIMEH AND CHENGXIANG ZHAI

UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN

1

Pseudo Relevance Feedback

Judgments:

d1 +

d2 -

d3 +

…

dk -

...

Query Retrieval

Engine

Results:

d1 3.5

d2 2.4

…

dk 0.5

...

User

Document

collection

Judgments:

d1 +

d2 +

d3 +

…

dk -

...

top 10

Pseudo feedback

Assume top 10 docsare relevant

Relevance feedback User judges documents

New

Query

FeedbackLearn from

Examples

2

Pseudo-Relevance Feedback

It’s blind!

Good for high recall information needs

A Blind Superhero. Courtesy of iStock

3

Collection-based Smoothing

Collection-based smoothing is generally used for LM-based retrieval functions and for PRF models

A commonly used collection-based smoothing scheme is Dirichletprior smoothing:

Dirichlet Prior (Smoothing Parameter)

Document Length

Count of Word in Document

4

Study of Smoothing Methods in PRF

We will establish both analytically and empirically that collection-based smoothing is not a good choice for PRF:◦ It forces PRF models to select very common words

Additive smoothing will be shown to outperform the collection-based counterpart

5

How Do LM PRF Models Work?

D1

Dn

… Averaging

Function: 𝑨

Scoring

Function: 𝒇

𝑃(𝑤|𝜃1)

𝑃(𝑤|𝜃𝑛)

𝑃(𝑤|𝜃𝐶)

𝑃(𝑤|𝜃𝐹)

6

How Do LM PRF Models Work?

The feedback LM, 𝜃𝐹, would generally have the following form:

𝐴:ℝ𝑛 → ℝ is an averaging function, e.g. geometric mean

𝑓:ℝ2 → ℝ is a function increasing in the first argument and decreasing in the second

Rewards common words in feedback set

Penalizes common words in collection

7

Problem!

The first argument rewards common words in the collection while the second penalizes them. The analysis shows that the first argument usually “wins”!

Rewards common words in feedback set

and collection

Penalizes common words in collection

Proportional to 𝑃(𝑤|Θ𝐶)

8

Overview of the Analysis

We considered three PRF models in the study:◦ Divergence Minimization Model

◦ Relevance Model

◦ Geometric Relevance Model

Next, we will briefly discuss how the DMM and GRM work and then give an overview of the axiomatic analysis.

The analysis of the RM is very similar to the GRM and the same results apply

9

Divergence Minimization Model (Zhai and Lafferty, 2001)

The DMM solves the following optimization problem:

The solution has a closed form and is given by:

10

Geometric Relevance Model (Seo and Croft, 2010)

An enhanced form of the Relevance Model (RM) that replaces the arithmetic mean used in RM by the geometric mean:

Note that the function above is not is not affected by 𝑃(𝑤|𝜃𝑐), i.e., the model is not designed to penalize common words.

11

Main Axiom: IDF Effect (Clinchant and Gaussier, 2013)

Rationale: A PRF model is expected to penalize common words in the collection in order to select high quality discriminative terms.

Given any two words 𝑤1and 𝑤2 from the feedback set 𝐷1, 𝐷2,

12

DMM with Collection-based smoothing: IDF Effect

Study the sign of:

Not straightforward. Strategy:◦ Find an attainable lower bound on the expression above

◦ Study the sign of the lower bound

◦ If the lower bound is strictly positive, then DMM supports the IDF effect

13

DMM: Results of Analysis

Conclusion: Using collection-based smoothing the DMM will be either consistently reward common terms or will select only one feedback term

14

GRM with Collection-based smoothing: IDF Effect

The GRM cannot support the IDF effect:

It consistently rewards favors common words in the collection

15

Proposed Solution: Additive Smoothing

Words get additional pseudo-counts:

Next, we show how additive smoothing prevents the models from rewarding common terms

16

DMM with Additive Smoothing: IDF Effect

The DMM unconditionally supports the IDF Effect:

Now it is performing the intended objective!

17

DMM: Empirical Validation

Query: “Computer”

18

GRM with Additive Smoothing: IDF Effect

Although the IDF effect is still not supported:

However, common terms are no longer being rewarded!

19

GRM: Empirical Validation

Query: “Computer”

20

Empirical Evaluation: Retrieval Measures

21

Empirical Evaluation: Robustness of Additive Smoothing

22

Measuring the Discrimination of PRF Models

In previous studies, the average of the IDF of the top terms was used as an indicator of how discriminative the terms selected by a PRF method are

Such a measure might not work well in some cases

We propose the Discrimination Measure (DM):

≈ Expected Document Frequency

Constant

23

Empirical Evaluation: Discrimination Measure

A several-fold decrease in the expected document frequency

24

Conclusion

Collection-based smoothing forces PRF models to select very common terms◦ The same problem might exist in other applications where LMs are aggregated

Additive smoothing prevents PRF models from rewarding common terms and increases the retrieval performance significantly

A new measure for quantifying PRF Discrimination

25

Future Work

Should PRF models penalize common words?

Analysis of other smoothing methods such as topic-based smoothing

Inspect areas, other than PRF, where collection-based smoothing is used in aggregating language models

26

Thanks to SIGIR for the Student Travel Grant!

Thank you for Listening!

27