Techniques Used in Modern Techniques Used in Modern

Question-Answering SystemsQuestion-Answering Systems

Candidacy Exam

Elena FilatovaDecember 11, 2002

CommitteeLuis Gravano Columbia UniversityVasileios Hatzivassiloglou Department of Computer ScienceRebecca J. Passonneau

Present Present vsvs Past Research on QAPast Research on QA

Current systems– Mainly systems written for TREC conference

• factoid questions• short answers• huge text collections

Related systems– IR

• queries vs questions• return documents vs short answers

– Systems based on semantic representations (Lehnert): • questions about one text vs text collections • inference from semantic structure of a text vs searching for

an answer in the text– One type of output (NP) from a closed collection (Kupiec)

answer inference vs answer extraction

Lehner’t systemLehner’t system

John loved Mary but she didn’t want to marry him. One day, adragon stole Mary from the castle. John got on top of hishorse and killed the dragon. Mary agreed to marry him. Theylived happily ever after.

Q: Why did Mary agree to marry John? A: Because she was indebted to him

Problems stated:– right classification– dependency of answer inference procedure on the type of the

question

Current QA SystemsCurrent QA Systems

questionanalysis

question queryextracteddocuments

rules foranswer

list of

answers

InformationInformation ExtractionExtraction

•right query

•long text

•domain dependency

•predefined types of answers

Information Information RetrievalRetrieval

PlanPlan

• Classification

• Information (document) retrieval– Query formation

• Information extraction– Passage extraction

– Answer extraction

• Usage of answer redundancy on Web in QA

• QA for restricted domain

• Evaluation procedure for current QA systems and analysis of the performance

Classification and QAClassification and QA

questionanalysis

question queryextracteddocuments

rules foranswer

list of

answers

Theory of ClassificationTheory of Classification

Rosch et al: classification of basic objects

World is structured: real-world attributes do not occur independently of each other:

object_has(wings) => P(object_has(feathers)) > P(object_has(fur))

Each category (class) – set of attributes that are common for all the objects in the category

Types of categories• Superordinate – small amount of common attributes (furniture)• Subordinate – a lot of common attributes (floor lamp, desk lamp)• Basic – optimal amount of common attributes (lamp): basic

objects are the most inclusive categories which delineate the correlation structure of the environment

Though classification is a converging problem for objects, it is not possible to compile a list of all possible basic categories.

QA classification. QA classification. • Hierarchical/nonhierarchical classification

– Even if there exist hierarchy in the classification it can be represented as flat: detailed classes + other class

• Amount of types(MULDER – 3 types vs Webclopedia – over 140 types)

• Trade off between– Detailed classes for better answer extraction and– High precision in defining the classes

• Usage of semantics

• Usage of syntax– Most of syntactic parsers are built on corpora which do no

contain a lot of questions (WSJ) => need of additional corpus

• Attempts to automate this process– Maximum Entropy (Ittycheriah)– Classifiers (Li&Roth)

Why QA classification is important?Why QA classification is important?

Usage of question type for1. query construction

• question keywords + filtering mechanism (Harabagiu)

• synonyms and syn.sets from WordNet (Webclopedia)

in both cases there is no connection with possible answer space

• information retrieval (Agichtein, Berger)there is connection between question and answer spaces but these types do not give the type of the answer

2. searching for a correct answer in the passage extracted from a text

Logical FormsLogical Forms

•Syntactic analysis plus semantic => logical form•Mapping of question and potential answer LFs to find the best match (Harabagiu, Webclopedia)

Query formationQuery formation

• WordNet: synonyms, hyponyms, etc.• Morphology: verbal forms, plural/single nouns,

etc.• Knowledge of the domain (IBM’s system)• Statistical methods for connecting question and

answer spaces:– Agichtein: automatic acquisition of patterns that might be

good candidates for query expansion4 ‘types ‘ of question

– Berger: to facilitate query modification (expansion) each question term gets a set of answer terms

FQA: closed set of question-answer pairs

Information retrievalInformation retrieval

• Classical IR is the first step of QA• Vector-space model (calculation of similarity between terms

in the query and terms in the document)• IR techniques used in current QA systems are usually for

one database (either web or TREC collection)• Is it possible to apply Distributed IR techniques?

– domain restricted QA with extra knowledge about the text collection IBM system

– “splitting” one big collection of documents into smaller collections about specific topics

– it might require change in classification: type of the question might cause the changes in query formulation, document extraction process, answer extraction process

questionanalysis

question queryextracteddocuments

rules foranswer

list of

answers

InformationInformation ExtractionExtraction

Information Information RetrievalRetrieval

Passage extractionPassage extraction

• Passages of particular length (Cardie) + Vector representation for each passage

• Paragraphs or sentences

• Classical text excerpting– Each sentence is assigned a score– Retrieved passages are formed by taking the sentences with

the highest score

• Global-Local Processing (Salton)

• McCallum: passage extraction based not only on words but also on other features (e.g. syntactic constructions)

Information ExtractionInformation Extraction

• Domain dependency (Grishman)predefined set of attributes for the search specific for eachtopic, e.g. terrorism: victims, locations, perpetrators

• usually a lot of manually tagged data for training or

• texts divided into two groups: one topic – all other texts (Riloff)

in both cases division into topics is anecessary step which is not applicable to open domain

QA systems

What information can be extracted (IE)What information can be extracted (IE)

• Named entities (NE-tagging)– Numbers (incl. dates, ZIP codes, etc.)– Proper names (locations, people, etc.)– Other depending on the systemTREC8 – 80% questions asked for NEs

NEs might also support

• Correlated entity: mini-CV (Srihari)Who is Julian Hill?name; age; gender; position; affiliation; education

• General events (Srihari)Who did what to whom when

More complicated IE techniques lead QA back to AI approach

Answer Extraction

Three main techniques for answer extraction are based on:

1. syntactic-semantic tree dependencies: (Harabagiu, Webclopedia) LF of the question is mapped to LF of possible answers

2. surface patterns (Webclopedia) – <Name> (<Answer> -)– <Name> was born on <Answer>Good patterns require detailed classification: NUMBER vs DOB

3. text window – Cardie: query-dependant text summarization of text passages

with/without syntactic and semantic information

LF mapping classical MTsurface patterns example-based MTtext window statistical MT

Usage of WebUsage of Web (Answer redundancy)(Answer redundancy)

Multiple formulation of answer can useful for:1. IR stage: increased chances to find an answer that

matches query (Clarke, Brill)no need in searching for an exact formulation of the answer

2. IE stage: facilitation of answer extraction (Agichtein, Ravichandran, Brill)

create a list of patterns which might contain the answereither completely automatic (Agichtein) or using handwritten

filters based on question types and domain (Brill)

3. Answer validation (Magnini) correct answer redundancy

)(*)(

),(),(

AspPQspP

AspQspPAspQspPMI

Domain restricted applicationsDomain restricted applications

• FAQ (different from IR or QA)– match the input question with a list of already existing

questions– predefined output (according to the above question

matching)

• Rillof– 5 types of questions– answer extraction from a given text => no IR stage– always there is an answer (unique answer)

• IBM system– based on good knowledge of inner structure of IBM web-

site– Use of FAQ techniques

results are better than for open-domain QA systems

restricted-domain MTvs

open-domain MT

EvaluationEvaluationIR and IE have different evaluation measures

– IR: each document is marked either relevant/non-relevant recall + precision

– IE: gold standard answer key enumerates all acceptable responses recall + precision

– QA: mean reciprocal rank (MRR) • For each question:

receive score equal to reciprocal of rank of first correct response, or 0 if no correct response found.

• Overall system score is mean of individual question scores.

N

KiMRR

N

i 1

/1

N – amount of questions asked;Ki = rank of the correct answer or 0; RAR =1/ Ki

Future of QAFuture of QA

FROM TO

Questions: Simple facts

Questions: Complex: Uses Judgments Terms; Knowledge of User Context Needed

Answers:Simple Factoid

Answers found in Single Document

Answers:Search Mult. Sources;

Fusion of Info; Resolution of Conflicting Data;

Interpretations, Conclusions