Is Question Answering

an Acquired Skill?Soumen

ChakrabartiG. Ramakrishnan

D. ParanjpeP. Bhattacharyya

IIT Bombay

QA Chakrabarti 3

Web search and QA Information need – words relating

“things” + “thing” aliases = telegraphic Web queries• Cheapest laptop with wireless

best price laptop 802.11• Why is the sky blue? sky blue reason• When was the Space Needle built?

“Space Needle” history Entity and relation extraction technology

better than ever (SemTag, KnowItAll)• Ontology extension (e.g., is a kind of)• List extraction (e.g., is an instance of)• Slot-filling (author X wrote book Y)

QA Chakrabarti 4

Factoid QA Specialize given domain to a token

related to ground constants in the query• What animal is Winnie the Pooh?

• hyponym(“animal”) NEAR “Winnie the Pooh”

• When was television invented?• instance-of(“time”) NEAR “television” NEAR

synonym(“invented”)

Three kinds of useful question tokens• Appear unchanged in passage (selector)• Specialize to answer tokens (atype)• Improve belief in answer via synonymy etc.

QA Chakrabarti 5

A new relational view of QA

Entity class or atype may be expressed by• A finite IS-A hierarchy (e.g. WordNet, TAP)• A surface pattern matching infinitely many strings

(e.g. “digit+”, “Xx+”, “preceded by a preposition”)

Match selectors, specialize atype to answer tokens

Question Atypeclues Selectors

Answerpassage

Questionwords

“Landingzone”

DirectsyntacticmatchEntity class

IS-ALimit searchto certain rows

Locate whichcolumn to read

“Landing zone”

Attributeor column

name

QA Chakrabarti 6

Benefits of the relational view “Scaling up by dumbing down”

• Next stop after vector-space• Far short of real knowledge representation

and inference• Barely getting practical at (near) Web scale

Can set up as a learning problem: train with questions and answers embedded in passage context

Transparent, self-tuning, easy to deploy• Feature extractors used in entity taggers• Relational/graphical learning on features

QA Chakrabarti 7

Subproblems Identify atype clues

• Easy: who, when, where, how many, how tall…

• Harder: What…, which…, name… Map atype clues to likely entity classes

• Data- and task-driven question classification• Train quickly on new corpus and QA samples

Identify selectors for keyword query• Based on question context and global stats

Get candidate passages from IR system• Re-rank candidate passages

QA Chakrabarti 8

Mapping “self-evident” atypes Whoperson, whentime, whereplace Not always trivial: how_many vs. when Question classification + handcrafted map

• Needs task knowledge and skilled effort• Laborious to move to new corpus, language…

Task-driven information extraction• Enough info in training QA pairs to learn map

Map clue to a generalization of the answer• Surface patterns: hasDigit, [in] DDDD, NNP,

CD• WordNet-based: region#n#3, quantity#n#1

QA Chakrabarti 9

Mapping exampleshow who

fast manyfar rich wrote first

How fast can a cheetah run?

A cheetah can chase its preyat up to 90 km/h

How fast does light travel?

Nothing moves faster than186,000 miles per hour, thespeed of light

rate#n#2

abstraction#n#6NNS

rate

#n#

2m

agnit

ude_r

ela

tion#

n#

1

mile

#n#

3lin

ear_

unit

#n#

1

measu

re#

n#

3definit

e_q

uanti

ty#

n#

1

paper_

money#

n#

1cu

rrency

#n#

1

writer, composer,artist, musician

NNP, person

explorer

WordNet

QA Chakrabarti 10

What…, which…, name… atype clues

Assumption: Question sentence has a wh-word and a main/auxiliary verb

Observation: Atype clues are embedded in a noun phrase (NP) adjoining the main or auxiliary verb

Heuristic: Atype clue = head of this NP• Use a shallow parser and apply rule

Head can have attributes• Which (American (general)) is buried in

Salzburg?• Name (Saturn’s (largest (moon)))

QA Chakrabarti 11

Atype clue extraction stats

Question type

#Questions#Extracted correctly

what 630 612which 29 28name 23 20

Simple heuristic surprisingly effective If successful, extracted atype is mapped to

WordNet synset (mooncelestial body etc.) If no atype of this form available, try the

“self-evident” atypes (who, when, where, how_X etc.)

QA Chakrabarti 12

Learning selectors Which question words are likely to appear

(almost) unchanged in an answer passage?• Constants in select-clauses of SQL queries• Guides backoff policy for keyword query

Local and global features• POS of word, POS of adjacent words, case info,

proximity to wh-word• Suppose word is associated with synset set S

• NumSense: size of S (how polysemous is the word?)

• NumLemma: average #lemmas describing s S

POS@0 POS@1POS@-1

QA Chakrabarti 13

Selector results Decision trees better than logistic regression

• F1=81% as against LR F1=75%• Intuitive decision branches• But logistic regression gives scores for query

backoff

Global features (IDF, NumSense, NumLemma) essential for accuracy• Best F1 accuracy with local features alone: 71—73%• With local and global features: 81%

N um Lem m a@ 0<=2.5 N um Lem m a@ 0>2.5

N um Sense@ 0<=9 N um Sense@ 0>9

PO S@ -1=N oun ...

PO S@ 0=Adj

PO S@ -1=N oun

N um Lem m a@ 0<=1.82 N um Lem m a@ 0>1.82

PO S@ 0=Verb

QA Chakrabarti 14

Putting together a QA system

QASystem

Wordnet

POSTagger

TrainingCorpus

Shallow p

arser

Learn

ing

tools

N-E

Tag

ger

QA Chakrabarti 15

Question

PassageIndex

Corpus

Sentence splitterPassage indexer

Candidatepassage

Keyword query

Keyword querygenerator

ShallowParser

Noun andverb markers

AtypeExtractor

Atype clues

Learning to rerank passagesSample features:•Do selectors match? How many?•Is some non-selector passage token a specialization of the question’s atype clue?•Min, avg linear token distance between candidate token and matched selectors

Learning to rerank passagesSample features:•Do selectors match? How many?•Is some non-selector passage token a specialization of the question’s atype clue?•Min, avg linear token distance between candidate token and matched selectors

LogisticRegression

Rerankedpassages

Putting together a QA systemTokenizer

POS TaggerTaggedquestion

TokenizerPOS Tagger

Entity Extractor

Taggedpassage

SelectorLearner

Is QA pair?

QA Chakrabarti 16

Learning to re-rank passages Remove passage tokens matching

selectors• User already knows these are in passage

Find passage token/s specializing atype

For each candidate token collect• Atype of question, original rank of passage• Min, avg linear distances to matched

selectors• POS and entity tag of token if available

Ushuaia, a port of about 30,000 dwellers set between the Beagle Channel and …

How many inhabitants live in the town of Ushuaia

selector matchSurface pattern hasDigits

WordNet match

5 tokens apart 1

QA Chakrabarti 17

Effect of re-ranking results Categorical and

numeric attributes Logistic regression Good precision,

poor recall Use logit score to

re-rank passages Rank of first correct

passage shifts substantially

194479

1

10

100

1000

1 2 3 4 5 6 7 8 9 10Answer at rank

Fre

quen

cy

BaselineRerank

Log scale

QA Chakrabarti 18

Mean reciprocal rank studies

nq = smallest rank among answer passages

Re-ranking reduces nq drastically

MRR = (1/|Q |) qQ(1/nq) Substantial gain in MRR TREC 2000 top MRRs:

0.76 0.71 0.46 0.46 0.310

100

200

300

0 100 200 300Initial rank

Fin

al r

ank

TREC 2000TREC 2002

QA Chakrabarti 19

Generalization across corpora

Across-year numbers close to train/test split on a single year

Features and model seem to capture corpus-independent linguistic Q+A artifacts

QA Chakrabarti 20

Re-ranking benefits by question type

All question types benefit from re-ranking

Benefits differ by question type

Large benefits for “what” and “which” questions, thanks to WordNet

Without WordNet customization

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

whe

n

wha

t

whe

re

ho

w

whi

ch

how

ma

ny

how

mu

ch

Question type

MR

R

Pre-reranking

Post-reranking

QA Chakrabarti 21

Conclusion A clean-room view of QA as

feature extraction plus learning• Recover structure info from question• Learn correlations between question

structure and passage features Competitive accuracy with negligible

domain expertise or manual intervention Ongoing work

• Use redundancy available from the Web• Model how selector and atype are related• Treat all question types uniformly