Trained Trigger Language Model for Sentence …ufal.mff.cuni.cz/pire10/Saeedeh-PIRE-Talk.pdfLanguage Models for Information Retrieval Trained Triggering Model Results Summary 16 16

1

PIRE

Trained Trigger Language Model Trained Trigger Language Model

for Sentence Retrieval for Sentence Retrieval

in Question Answering Systems in Question Answering Systems

Saeedeh Saeedeh MomtaziMomtazi

Spoken Language SystemsSpoken Language Systems

Saarland UniversitySaarland University

[email protected]@lsv.uni--saarland.desaarland.de

Research Advisor: Dietrich Research Advisor: Dietrich KlakowKlakow

2

2

OutlineOutline

��Question Answering SystemQuestion Answering System

��Language Models for Information RetrievalLanguage Models for Information Retrieval

��Trained Triggering ModelTrained Triggering Model

��ResultsResults

��SummarySummary

3

3

4

4

5

5

6

6

1 Answer

Internet

pages web1010

Document Retrieval

Sentence Retrieval

Information

Extraction

Answer

Selection

Goal:answer questions like “Who is Warren Moon’s Agent?”

e.g. “Leigh Steinberg”

What is Question Answering?What is Question Answering?

7

7

Sentence RetrievalSentence Retrieval

��Task:Task:

��Finding a small segment of text that contains the answer Finding a small segment of text that contains the answer

[[CorradaCorrada--Emmanuel, Croft, & Murdock, 2003Emmanuel, Croft, & Murdock, 2003]]

��Benefits beyond document retrieval: Benefits beyond document retrieval:

��Documents are very largeDocuments are very large

��Documents span different subject areasDocuments span different subject areas

��The relevant information is expressed much more locallyThe relevant information is expressed much more locally

��Retrieving the sentences Retrieving the sentences simplifies the next step: simplifies the next step:

information extractioninformation extraction

8

8

OutlineOutline

�� Question Answering SystemQuestion Answering System

��Language Models for Information RetrievalLanguage Models for Information Retrieval




9

9

Language Models for IRLanguage Models for IR

�� P(Q|SP(Q|Sii)): language model trained on : language model trained on SSii

�� Ranking sentences in descending order of this probabilityRanking sentences in descending order of this probability

S6

S1

S2

S3

S5S4

S7Query

Q

P(Q|S2)

10

10

Query Likelihood ModelQuery Likelihood Model

�� Unigram language model of sentences and queriesUnigram language model of sentences and queries

[[Song & Croft, 1999Song & Croft, 1999]]

∏=

=Mi

i SqPSQP...1

)|()|(

Query

ModelSentence

ModelQuery

Terms

11

11

� A separate language model is trained for each sentence

in the search space

� The probability is calculated based on the frequency of

query term in the sentence

Maximum Likelihood EstimationMaximum Likelihood Estimation

∏=

=Mi

i SqPSQP...1

)|()|(

∑=

w

S

iSi

wf

qfSqP

)(

)()|(

12

12

ExampleExample

��Question:Question:

��Answers:Answers:

Who invented the automobile?Who invented the automobile?

An automobile powered by An automobile powered by his own engine was his own engine was invented by Karl Benz in invented by Karl Benz in 1885 and granted a patent.1885 and granted a patent.

13

13

ExampleExample



Who Who inventedinventedinventedinventedinventedinventedinventedinvented the the automobileautomobileautomobileautomobileautomobileautomobileautomobileautomobile??

An An automobileautomobileautomobileautomobileautomobileautomobileautomobileautomobile powered by powered by his own engine was his own engine was inventedinventedinventedinventedinventedinventedinventedinvented by Karl Benz in by Karl Benz in 1885 and granted a patent.1885 and granted a patent.

Nicolas Joseph Nicolas Joseph CugnotCugnotinvented the first self invented the first self propelled mechanical propelled mechanical vehicle.vehicle.

14

14

Nicolas Joseph Nicolas Joseph CugnotCugnotinventedinventedinventedinventedinventedinventedinventedinvented the first self the first self propelled mechanical propelled mechanical vehicle.vehicle.

ExampleExample




An An automobileautomobileautomobileautomobileautomobileautomobileautomobileautomobile powered by powered by his own engine was his own engine was inventedinventedinventedinventedinventedinventedinventedinvented by Karl Benz in by Karl Benz in 1885 and granted a patent.1885 and granted a patent.

Thomas Edison Thomas Edison inventedinventedinventedinventedinventedinventedinventedinventedthe first commercially the first commercially practical light.practical light.

Alexander Graham Bell of Alexander Graham Bell of Scotland is the Scotland is the inventorinventorinventorinventorinventorinventorinventorinventorof the first practical of the first practical telephone.telephone.

15

15

OutlineOutline


�� Language Models for Information RetrievalLanguage Models for Information Retrieval




16

16

� A unique model is trained on a large corpus firstly, then

it is being used for all of the sentences to be retrieved

� The trained model is represented by a set of triples:

� is the number of times the word w triggers

the target word w’.

Trained Trigger ModelTrained Trigger Model

∏=

=Mi

i SqPSQP...1

)|()|(

>< )',(,', wwfww C

)',( wwf C

17

17

� Triggering Model:

Trained Trigger ModelTrained Trigger Model

∏=

=Mi

i SqPSQP...1

)|()|(

∑=

=Nj

jii sqPN

SqP..1

)|(1

)|(

∑=

iq

jiC

jiC

jisqf

sqfsqP

),(

),()|(

18

18

Word Unigram ModelWord Unigram Model

S6

S1

S2

S3

S5S4

S7Query

Q

P(Q|S2)

2Sθ

19

19


S6

S1

S2

S3

S5S4

S7Query

Q

P(Q|S5)

5Sθ

20

20

Triggering ModelTriggering Model

S6

S1

S2

S3

S5S4

S7Query

Q

Corpus CCθ

P(Q|S2)

21

21


S6

S1

S2

S3

S5S4

S7Query

Q

Corpus CCθ

P(Q|S5)

22

22


S6

S1

S2

S3

S5S4

S7Query

Q

P(Q|S2)

2Sθn-gram

23

23


S6

S1

S2

S3

S5S4

S7Query

Q

Corpus CCθ

P(Q|S2)

word

triggering

Self Triggering

Inside Sentence

Across Sentence

Question and Answer Pair

24

24

Self TriggeringSelf Triggering

� Each word can only trigger itself

� Works similar to the basic bag-of-words model

� The words that appeared in both the query and the sentence have the higher priority

� It is an essential part of a retrieval engine and have to be used beside any other triggering models

25

25

Inside Sentence TriggeringInside Sentence Triggering

� Idea: there is a relation between words appear in the same sentences

� Considers that each word in a sentence triggers all other words in the same sentence

� Uses a large unannotated corpus for training

� The sentence retrieval can retrieve sentences which do not share many terms with the query, but their terms frequently co-occur with query terms in the same sentences of the training corpus.

26

26

Nicolas Joseph Nicolas Joseph CugnotCugnotinventedinventedinventedinventedinventedinventedinventedinvented the first self the first self propelled mechanical propelled mechanical vehicle.vehicle.

ExampleExample




Thomas Edison Thomas Edison inventedinventedinventedinventedinventedinventedinventedinventedthe first commercially the first commercially practical light.practical light.

Alexander Graham Bell of Alexander Graham Bell of Scotland is the Scotland is the inventorinventorinventorinventorinventorinventorinventorinventorof the first practical of the first practical telephone.telephone.

27

27

Inside Sentence TriggeringInside Sentence Triggering

“The word automobileautomobileautomobileautomobilemeaning a vehiclevehiclevehiclevehicle that moves itself.”

“An automobileautomobileautomobileautomobileincludes at least two seats located one behind the other and attachable to a vehiclevehiclevehiclevehicle floor.”

28

28

Across Sentence TriggeringAcross Sentence Triggering

� Idea: two adjacent sentences mostly talk about the same topic by using different words with the same concept and meaning

� Considers that each word of a sentence triggers all of the words of the next sentence in the training corpus

� Uses a large unannotated corpus for trainings

� Uses a wider context than inside sentence triggering

29

29

ExampleExample



Where was the first McDonaldWhere was the first McDonald’’s built?s built?

Two brothers from Two brothers from Manchester, Dick and Mac Manchester, Dick and Mac McDonald, open the first McDonald, open the first McDonaldMcDonald’’s in California.s in California.

30

30

ExampleExample



Where was the Where was the first McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonald’’’’’’’’ssssssss built?built?

Two brothers from Two brothers from Manchester, Dick and Mac Manchester, Dick and Mac McDonald, open the McDonald, open the first first first first first first first first McDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonald’’’’’’’’ssssssss in California.in California.

The site of the The site of the first first first first first first first first McDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonald’’’’’’’’ssssssss to be to be franchised by Ray Kroc is franchised by Ray Kroc is now a museum in Des now a museum in Des Plaines, Illinois.Plaines, Illinois.

The The first McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonald’’’’’’’’ssssssss TV TV commercial was a pretty commercial was a pretty lowlow--budget affair.budget affair.

31

31

Two brothers from Two brothers from Manchester, Dick and Mac Manchester, Dick and Mac McDonald, open the McDonald, open the first first first first first first first first McDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonald’’’’’’’’ssssssss in California.in California.

ExampleExample



Where was the Where was the first McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonald’’’’’’’’ssssssss builtbuiltbuiltbuiltbuiltbuiltbuiltbuilt??

The site of the The site of the first first first first first first first first McDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonaldMcDonald’’’’’’’’ssssssss to be to be franchised by Ray Kroc is franchised by Ray Kroc is now a museum in Des now a museum in Des Plaines, Illinois.Plaines, Illinois.

The The first McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonaldfirst McDonald’’’’’’’’ssssssss TV TV commercial was a pretty commercial was a pretty lowlow--budget affair.budget affair.

32

32

Across Sentence TriggeringAcross Sentence Triggering

“The structure of Eiffel Tower was builtbuiltbuiltbuilt between 1887 and 1889 as the entrance arch for the Exposition Universelle, a World’s Fair marking the centennial celebration of the French Revolution.”“The tower was inaugurated on 31 March 1889, and openedopenedopenedopened on 6 May.”

“Wembley Stadium was builtbuiltbuiltbuiltby Australian company Brookfield Multiplex.”“The stadium was scheduled to openopenopenopen on 13 May 2006.”

33

33

Question and Answer Pair TriggeringQuestion and Answer Pair Triggering

� Idea: there is a relation between words appear in a pair of question and answer sentence

� Considers that each word in the question triggers all words in its answer sentence

� Requires a supervised training

� Uses a question-answer sentence corpus for training

34

34

ExampleExample



How high is Everest?How high is Everest?

Everest is Everest is 29,029 feet.29,029 feet.

35

35

ExampleExample



How high is How high is EverestEverestEverestEverestEverestEverestEverestEverest??

EverestEverestEverestEverestEverestEverestEverestEverest is is 29,029 feet.29,029 feet.

EverestEverestEverestEverestEverestEverestEverestEverest is is located in Nepal.located in Nepal.

EverestEverestEverestEverestEverestEverestEverestEverest has two main has two main climbing routes.climbing routes.

36

36

EverestEverestEverestEverestEverestEverestEverestEverest is is 29,029 feet.29,029 feet.

ExampleExample



How How highhighhighhighhighhighhighhigh is is EverestEverestEverestEverestEverestEverestEverestEverest??

EverestEverestEverestEverestEverestEverestEverestEverest is is located in Nepal.located in Nepal.

EverestEverestEverestEverestEverestEverestEverestEverest has two main has two main climbing routes.climbing routes.

37

37

Question and Answer Pair TriggeringQuestion and Answer Pair Triggering

Q: “How highhighhighhigh is Pikes peak?”A: “Pikes peak, Colorado At 14,110 feetfeetfeetfeet, altitude sickness is a consideration when driving up this mountain.”

Q: “How highhighhighhigh is Mount Hood?”A: “Mount Hood is in the Cascade Mountain range and is 11,245 feetfeetfeetfeet.”

38

38

OutlineOutline



�� Trained Triggering ModelTrained Triggering Model



39

39

CorporaCorpora

� Inside Sentence

� Across Sentence

� Question and Answer Pair

AQUAINT1 Corpus• Consists of Engish newswire text, extracted from:

• the Xinhua (XIN)• the New York Times (NYT)• the Associated Press Worldstream (APW)

• Contains • almost 450 million word tokens • as 23 million sentences• as 1.5 million documents.

QASP corpus

• Consists of TREC QA track• from 2002 to 2004• 985 questions

•Prepared via Amazon MTurk

Yahoo! Answers Corpus

• derived fromhttp://answers.yahoo.com/

• collected in 10/25/2007• containing 4,483,032 questions

40

40

ExperimentsExperiments

�� TREC data setTREC data set

�� Development set: TREC05 Development set: TREC05 (316 questions)(316 questions)

�� Test set: TREC06 Test set: TREC06 (365 questions)(365 questions)

�� JudgmentJudgment

�� QASP CorpusQASP Corpus

�� Prepared via Amazon Prepared via Amazon MTurkMTurk

�� Evaluation MetricsEvaluation Metrics

�� Mean Average PrecisionMean Average Precision

�� Precision@5Precision@5

�� Mean Reciprocal Rank Mean Reciprocal Rank

41

41

Results Results

0.37010.3701

MAPMAP

0.50470.5047

MRRMRR

Maximum Maximum LikelohoodLikelohood

ModelModel

0.22670.2267

P@5P@5

42

42

Results Results

0.23490.23490.52190.52190.38060.3806Self TTLMSelf TTLM

0.37010.3701

MAPMAP

0.50470.5047

MRRMRR


ModelModel

0.22670.2267

P@5P@5

43

43

Results Results

0.13600.13600.32630.32630.22660.2266Question and Answer Pair TTLM (QASP)Question and Answer Pair TTLM (QASP)

0.23490.23490.52190.52190.38060.3806Self TTLMSelf TTLM

0.10520.10520.25850.25850.19110.1911Inside Sentence TTLMInside Sentence TTLM

0.03440.0344

0.23670.2367

0.37010.3701

MAPMAP

0.04150.0415

0.30470.3047

0.50470.5047

MRRMRR

0.00990.0099Question and Answer Pair TTLM (Yahoo)Question and Answer Pair TTLM (Yahoo)

Across Sentence TTLMAcross Sentence TTLM


ModelModel

0.13600.1360

0.22670.2267

P@5P@5

44

44

Results (Linear Interpolation) Results (Linear Interpolation)

+ QA Pair (QASP)+ QA Pair (QASP)

+ Inside Sentence+ Inside Sentence

0.23490.2349

P@5P@5

0.52190.5219

MRRMRR

0.38060.3806

MAPMAP

Self TriggeringSelf Triggering Maximum LikelihoodMaximum Likelihood

0.37010.3701

MAPMAP

0.50470.5047

MRRMRR

+ QA Pair (Yahoo)+ QA Pair (Yahoo)

+ Across Sentence+ Across Sentence

BaselineBaseline

ModelModel

0.22670.2267

P@5P@5

45

45


all the differences are statistically significant

at the level of p-value<0.01 based on t-tests.

0.26220.26220.54920.54920.42080.42080.26630.26630.54080.54080.42040.4204+ QA Pair (QASP)+ QA Pair (QASP)

0.25870.25870.55720.55720.43510.43510.26280.26280.54690.54690.43140.4314+ Inside Sentence+ Inside Sentence

0.26450.2645

0.25930.2593

0.23490.2349

P@5P@5

0.56180.5618

0.54800.5480

0.52190.5219

MRRMRR

0.43580.4358

0.42740.4274

0.38060.3806

MAPMAP


0.43710.4371

0.43810.4381

0.37010.3701

MAPMAP

0.56540.5654

0.56310.5631

0.50470.5047

MRRMRR

0.26280.2628+ QA Pair (Yahoo) + QA Pair (Yahoo)


BaselineBaseline

ModelModel

0.26050.2605

0.22670.2267

P@5P@5

46

46






0.26450.2645

0.25930.2593

0.23490.2349

P@5P@5

0.56180.5618

0.54800.5480

0.52190.5219

MRRMRR

0.43580.4358

0.42740.4274

0.38060.3806

MAPMAP


0.43710.4371

0.43810.4381

0.37010.3701

MAPMAP

0.56540.5654

0.56310.5631

0.50470.5047

MRRMRR



BaselineBaseline

ModelModel

0.26050.2605

0.22670.2267

P@5P@5

47

47






0.26450.2645

0.25930.2593

0.23490.2349

P@5P@5

0.56180.5618

0.54800.5480

0.52190.5219

MRRMRR

0.43580.4358

0.42740.4274

0.38060.3806

MAPMAP


0.43710.4371

0.43810.4381

0.37010.3701

MAPMAP

0.56540.5654

0.56310.5631

0.50470.5047

MRRMRR



BaselineBaseline

ModelModel

0.26050.2605

0.22670.2267

P@5P@5

48

48






0.26450.2645

0.25930.2593

0.23490.2349

P@5P@5

0.56180.5618

0.54800.5480

0.52190.5219

MRRMRR

0.43580.4358

0.42740.4274

0.38060.3806

MAPMAP


0.43710.4371

0.43810.4381

0.37010.3701

MAPMAP

0.56540.5654

0.56310.5631

0.50470.5047

MRRMRR



BaselineBaseline

ModelModel

0.26050.2605

0.22670.2267

P@5P@5

49

49


50

50

OutlineOutline



�� Trained Triggering ModelTrained Triggering Model

�� ResultsResults


51

51

SummarySummary

�� Introducing question answering systemsIntroducing question answering systems

�� The necessity of the sentence retrieval in a QA systemThe necessity of the sentence retrieval in a QA system

�� Using language models for sentence retrievalUsing language models for sentence retrieval

�� Describing the current unigram model and its problemsDescribing the current unigram model and its problems

�� Proposing trained triggering model with different types:Proposing trained triggering model with different types:

�� selfself

�� inside sentenceinside sentence

�� across sentenceacross sentence

�� question and answer pairquestion and answer pair

�� Linear interpolation of different modelsLinear interpolation of different models

52

52

ReferencesReferences

�� BilottiBilotti, M. W., 2004. Query Expansion Techniques for Question Answering, M. W., 2004. Query Expansion Techniques for Question Answering. MS Thesis in Electrical . MS Thesis in Electrical Engineering and Computer Science, Massachusetts Institute of TecEngineering and Computer Science, Massachusetts Institute of Technologyhnology

�� CorradaCorrada--Emmanuel, A., Croft, W. B., and Murdock, V., 2003. Answer PassagEmmanuel, A., Croft, W. B., and Murdock, V., 2003. Answer Passage Retrieval for Question e Retrieval for Question Answering, in CIIR Technical Report, AmherstAnswering, in CIIR Technical Report, Amherst

�� Ponte, J. and Croft, W.B. 1998. A language modeling approach to Ponte, J. and Croft, W.B. 1998. A language modeling approach to information retrieval. In Proceedings of information retrieval. In Proceedings of the 1998 ACM SIGIR Conference on Research and Development in Infthe 1998 ACM SIGIR Conference on Research and Development in Information Retrieval, 275ormation Retrieval, 275--281 281

�� Lafferty, J. and Lafferty, J. and ZhaiZhai, C. 2001. Document language models, query models, and risk mini, C. 2001. Document language models, query models, and risk minimization for mization for information retrieval. In Proceedings of the 2001 ACM SIGIR Confinformation retrieval. In Proceedings of the 2001 ACM SIGIR Conference on Research and Development in erence on Research and Development in Information Retrieval, 111Information Retrieval, 111--119. 119.

�� HiemstraHiemstra, D. A. 1998. Linguistically Motivated Probabilistic Model of In, D. A. 1998. Linguistically Motivated Probabilistic Model of Information Retrieval. Second formation Retrieval. Second European Conference on Digital Libraries, 569European Conference on Digital Libraries, 569--584.584.

�� Song, F., & Croft, W.B. 1999. A general language model for inforSong, F., & Croft, W.B. 1999. A general language model for information retrieval. In Proceedings of the mation retrieval. In Proceedings of the 22nd annual international ACM22nd annual international ACM--SIGIR conference on research and development in information retrSIGIR conference on research and development in information retrieval, ieval, 279279--280, New York.280, New York.

�� ZhaiZhai, C. and Lafferty, J. 2001. A study of smoothing methods for lan, C. and Lafferty, J. 2001. A study of smoothing methods for language models applied to ad hoc guage models applied to ad hoc information retrieval. In W.B. Croft, D.J. Harper, D.H. Kraft, &information retrieval. In W.B. Croft, D.J. Harper, D.H. Kraft, & J. J. ZobelZobel (Eds.), Proceedings of the 24th (Eds.), Proceedings of the 24th annual international ACMSIGIR conference on research and developannual international ACMSIGIR conference on research and development in information. ment in information.

�� GaoGao, J., , J., NieNie, J.Y., Wu, G., , J.Y., Wu, G., CaoCao, G. 2004. Dependence language model for information retrieval. , G. 2004. Dependence language model for information retrieval. In: In: Proceedings of the 27th annual international conference on ReseaProceedings of the 27th annual international conference on Research and development in information rch and development in information retrieval. retrieval.

53

53

Thanks for your attention!Thanks for your attention!