Probabilistic Model for Definitional Question Answering

Kyoung-Soo Han, Young-In Song, and Hae-Chang RimKorea University

SIGIR 2006

INTRODUCTION

Definitional question answering is a task of answering definitional questions used for finding out conceptual facts or essential events about the question target.

EX: “What is X?” or “Who is X?”

INTRODUCTION

A short passage cannot answer the definitional questions because a definition needs several essential information (information nuggets) about the target

Each answer nugget is naturally represented by a short noun phrase or a verb phrase

Good definitional QA systems have to find out more answer nuggets with shorter length.

Topic and Definition

We can suppose that the following sentences are answer candidates for a question “What is NASA?”.

S1: NASA is the agency responsible for the public space program of the USA.

S2: NASA was established in 1958. S3: The headquarters of NASA is located in Washington, D.C. S4: NASA announced the new annual budget. S5: John who works for NASA gave a housewarming party

yesterday. S6: Ji-Sung Park is a famous football player from South Korea.

Topic and Definition

Among the candidates, the sentences representing the topic of “NASA” are {S1, S2, S3, S4}, and those representing the definition are {S1, S2, S3, S6}.

{S1, S2, S3} as the answer to the question.

Definitional Question Answering Model

the answer A to a definitional question about the target is the intersection of T and D

T: The sentence S includes the contents related to the topic of the target X.

D: The sentence S represents the definition. Definitional QA can be considered as a task

of finding the sentences which maximize a joint probability P(T,D|S)

Definitional Question Answering Model

P(T,D|S) = P(T|S)P(D|T,S) assuming T and D are conditionally

independent given S

Definitional Question Answering Model

It is not necessary to calculate the exact probability in order to rank the sentences.

If the sentence S is a sequence of words w1w2 · · ·wn, the function is rewritten as follows:

Definitional Question Answering Model

P(w1,n|T) is the probability that the word sequence w1,n is generated from the target topic, and we call it topic language model.

P(w1,n|D) is the probability that w1,n is generated from the definition representations, and we call it definition language model.

P(w1,n) is the prior probability of w1,n, and we call it general language model.

General Language Model

P(w1,n) = P(w1)P(w2|w1)P(w3|w1,2) · · · P(wn|w1,n−1)

Assuming the word occurrences are independent of one another

Topic Language Model

We model the target topic using the following several evidences: Top ranked documents R retrieved from the

collection by the query X: Definitions E for the target X from external

resources (such as online dictionary) Top ranked web pages W retrieved from the

WWW by the query X:

Topic Language Model

Where α + β + γ = 1

Each probability is estimated by Dirichlet smoothing

Definition Language Model

We constructed the definition corpus by collecting the definitions for arbitrary definition targets from the online resources, and estimated the probability using the definition corpus.

The word probability distribution can be different depending on the domain of the definition target.

For example, “president”, “scientist”, “born”, and “died” will frequently occur in the definition for a person.

On the other hand, “established”, “member”, “headquarters”, and “branch” will do in the definition for an organization.

Definition Language Model

P(wi|DtX ) is the probability that word wi is generated from the definition corpus whose domain for definition target is equal to the domain tX for question target

P(wi|Dall) is the probability that wi is generated from the definition corpus for all domains.

λ is a interpolation parameter

Definition Language Model

We used three domains in this paper: person, organization, and term

DEFINITIONAL QUESTIONANSWERING SYSTEM

Given a definitional question, the question target is extracted and its type is identified.

From a question such as “Who is Andrew Carnegie?”, for example, “Andrew Carnegie” is extracted by using simple rules.

The type of the target is identified by a named entity tagger, BBN Identifinder

We classify the target into three types: person, organization, and term. If a target is not classified into person or organization by the named entity tagger, it is classified into term.

Document Retrieval

Answer candidates are extracted from the retrieved relevant documents.

The document retrieval is carried out by using BM25 scoring function of OKAPI.

The query consists of each word of the question target

Answer Candidate Extraction

We try to extract target-related parts of sentences using syntactic structure of the sentences.

If the parts are extracted, they are used as answer candidates. Otherwise, the sentences are used as the candidates.

We extract noun and verb phrases from the sentences using the syntactic definition patterns

Answer Candidate Selection

A = {aj |LDS(aj ) > δsel }

Experiments Setup

We have experimented with 50 TREC 2003 topics and 64 TREC 2004 topics, and we found the answer from the AQUAINT corpus used for TREC Question Answering track evaluation.

The TREC answer set for the definitional QA task consists of several definition nuggets for each target, and each nugget is a short string.

We evaluated our system using the automatic measure POURPRE.

The POURPRE estimates the TREC metric, nugget recall, precision, and F-measure, using term co-occurrences between answer nugget and system output.

Experiments Setup

For topic modeling, we collected external definitions from : Acronym Finder, Biography.com, Columbia Encyclopedia, Wikipedia, FOLDOC, The American Heritage Dictionary of the English Language, Online Medical Dictionary, and Google Glossary.

We also used ten web pages retrieved by the Google search engine and five local documents retrieved from the AQUAINT corpus for topic modeling.

For definition modeling, we constructed definition corpus from the above sites according to the target type: 14,904 persons, 994 organizations, and 3,639 terms entries. We processed top 200 documents retrieved in all experiments.

Topic Modeling

The α, β, γ are weight of top ranked documents R, external definitions E, and top ranked web pages W, respectively, in equation 7 for the topic modeling. λ and μ are set to 0.6 and 2,000, respectively. δsel is not applied

Topic Modeling

The experimental results show: The target topic is best represented by the

external definitions. As the external definitions provide core information about the question target without noise

The external definitions are the most confident, followed by the top ranked documents and web pages, in order.

Definition Modeling

λ is the degree to which the target type has an effect on the definition modeling.

Comparison with Other Systems

proposed(1000,10) is the system whose target length of the answer and score threshold δsel is set to 1,000 byte and 10, respectively.

CONCLUSIONS

We proposed a probabilistic model for definitional QA, analyzing the problem into two main components, topic and definition.

The experimental results show that the external definition which has almost no noise is the most valuable information for topic modeling