Is Question Answering
an Acquired Skill?Soumen
ChakrabartiG. Ramakrishnan
D. ParanjpeP. Bhattacharyya
IIT Bombay
QA Chakrabarti 3
Web search and QA Information need – words relating
“things” + “thing” aliases = telegraphic Web queries• Cheapest laptop with wireless
best price laptop 802.11• Why is the sky blue? sky blue reason• When was the Space Needle built?
“Space Needle” history Entity and relation extraction technology
better than ever (SemTag, KnowItAll)• Ontology extension (e.g., is a kind of)• List extraction (e.g., is an instance of)• Slot-filling (author X wrote book Y)
QA Chakrabarti 4
Factoid QA Specialize given domain to a token
related to ground constants in the query• What animal is Winnie the Pooh?
• hyponym(“animal”) NEAR “Winnie the Pooh”
• When was television invented?• instance-of(“time”) NEAR “television” NEAR
synonym(“invented”)
Three kinds of useful question tokens• Appear unchanged in passage (selector)• Specialize to answer tokens (atype)• Improve belief in answer via synonymy etc.
QA Chakrabarti 5
A new relational view of QA
Entity class or atype may be expressed by• A finite IS-A hierarchy (e.g. WordNet, TAP)• A surface pattern matching infinitely many strings
(e.g. “digit+”, “Xx+”, “preceded by a preposition”)
Match selectors, specialize atype to answer tokens
Question Atypeclues Selectors
Answerpassage
Questionwords
“Landingzone”
DirectsyntacticmatchEntity class
IS-ALimit searchto certain rows
Locate whichcolumn to read
“Landing zone”
Attributeor column
name
QA Chakrabarti 6
Benefits of the relational view “Scaling up by dumbing down”
• Next stop after vector-space• Far short of real knowledge representation
and inference• Barely getting practical at (near) Web scale
Can set up as a learning problem: train with questions and answers embedded in passage context
Transparent, self-tuning, easy to deploy• Feature extractors used in entity taggers• Relational/graphical learning on features
QA Chakrabarti 7
Subproblems Identify atype clues
• Easy: who, when, where, how many, how tall…
• Harder: What…, which…, name… Map atype clues to likely entity classes
• Data- and task-driven question classification• Train quickly on new corpus and QA samples
Identify selectors for keyword query• Based on question context and global stats
Get candidate passages from IR system• Re-rank candidate passages
QA Chakrabarti 8
Mapping “self-evident” atypes Whoperson, whentime, whereplace Not always trivial: how_many vs. when Question classification + handcrafted map
• Needs task knowledge and skilled effort• Laborious to move to new corpus, language…
Task-driven information extraction• Enough info in training QA pairs to learn map
Map clue to a generalization of the answer• Surface patterns: hasDigit, [in] DDDD, NNP,
CD• WordNet-based: region#n#3, quantity#n#1
QA Chakrabarti 9
Mapping exampleshow who
fast manyfar rich wrote first
How fast can a cheetah run?
A cheetah can chase its preyat up to 90 km/h
How fast does light travel?
Nothing moves faster than186,000 miles per hour, thespeed of light
rate#n#2
abstraction#n#6NNS
rate
#n#
2m
agnit
ude_r
ela
tion#
n#
1
mile
#n#
3lin
ear_
unit
#n#
1
measu
re#
n#
3definit
e_q
uanti
ty#
n#
1
paper_
money#
n#
1cu
rrency
#n#
1
writer, composer,artist, musician
NNP, person
explorer
WordNet
QA Chakrabarti 10
What…, which…, name… atype clues
Assumption: Question sentence has a wh-word and a main/auxiliary verb
Observation: Atype clues are embedded in a noun phrase (NP) adjoining the main or auxiliary verb
Heuristic: Atype clue = head of this NP• Use a shallow parser and apply rule
Head can have attributes• Which (American (general)) is buried in
Salzburg?• Name (Saturn’s (largest (moon)))
QA Chakrabarti 11
Atype clue extraction stats
Question type
#Questions#Extracted correctly
what 630 612which 29 28name 23 20
Simple heuristic surprisingly effective If successful, extracted atype is mapped to
WordNet synset (mooncelestial body etc.) If no atype of this form available, try the
“self-evident” atypes (who, when, where, how_X etc.)
QA Chakrabarti 12
Learning selectors Which question words are likely to appear
(almost) unchanged in an answer passage?• Constants in select-clauses of SQL queries• Guides backoff policy for keyword query
Local and global features• POS of word, POS of adjacent words, case info,
proximity to wh-word• Suppose word is associated with synset set S
• NumSense: size of S (how polysemous is the word?)
• NumLemma: average #lemmas describing s S
POS@0 POS@1POS@-1
QA Chakrabarti 13
Selector results Decision trees better than logistic regression
• F1=81% as against LR F1=75%• Intuitive decision branches• But logistic regression gives scores for query
backoff
Global features (IDF, NumSense, NumLemma) essential for accuracy• Best F1 accuracy with local features alone: 71—73%• With local and global features: 81%
N um Lem m a@ 0<=2.5 N um Lem m a@ 0>2.5
N um Sense@ 0<=9 N um Sense@ 0>9
PO S@ -1=N oun ...
PO S@ 0=Adj
PO S@ -1=N oun
N um Lem m a@ 0<=1.82 N um Lem m a@ 0>1.82
PO S@ 0=Verb
QA Chakrabarti 14
Putting together a QA system
QASystem
Wordnet
POSTagger
TrainingCorpus
Shallow p
arser
Learn
ing
tools
N-E
Tag
ger
QA Chakrabarti 15
Question
PassageIndex
Corpus
Sentence splitterPassage indexer
Candidatepassage
Keyword query
Keyword querygenerator
ShallowParser
Noun andverb markers
AtypeExtractor
Atype clues
Learning to rerank passagesSample features:•Do selectors match? How many?•Is some non-selector passage token a specialization of the question’s atype clue?•Min, avg linear token distance between candidate token and matched selectors
Learning to rerank passagesSample features:•Do selectors match? How many?•Is some non-selector passage token a specialization of the question’s atype clue?•Min, avg linear token distance between candidate token and matched selectors
LogisticRegression
Rerankedpassages
Putting together a QA systemTokenizer
POS TaggerTaggedquestion
TokenizerPOS Tagger
Entity Extractor
Taggedpassage
SelectorLearner
Is QA pair?
QA Chakrabarti 16
Learning to re-rank passages Remove passage tokens matching
selectors• User already knows these are in passage
Find passage token/s specializing atype
For each candidate token collect• Atype of question, original rank of passage• Min, avg linear distances to matched
selectors• POS and entity tag of token if available
Ushuaia, a port of about 30,000 dwellers set between the Beagle Channel and …
How many inhabitants live in the town of Ushuaia
selector matchSurface pattern hasDigits
WordNet match
5 tokens apart 1
QA Chakrabarti 17
Effect of re-ranking results Categorical and
numeric attributes Logistic regression Good precision,
poor recall Use logit score to
re-rank passages Rank of first correct
passage shifts substantially
194479
1
10
100
1000
1 2 3 4 5 6 7 8 9 10Answer at rank
Fre
quen
cy
BaselineRerank
Log scale
QA Chakrabarti 18
Mean reciprocal rank studies
nq = smallest rank among answer passages
Re-ranking reduces nq drastically
MRR = (1/|Q |) qQ(1/nq) Substantial gain in MRR TREC 2000 top MRRs:
0.76 0.71 0.46 0.46 0.310
100
200
300
0 100 200 300Initial rank
Fin
al r
ank
TREC 2000TREC 2002
QA Chakrabarti 19
Generalization across corpora
Across-year numbers close to train/test split on a single year
Features and model seem to capture corpus-independent linguistic Q+A artifacts
QA Chakrabarti 20
Re-ranking benefits by question type
All question types benefit from re-ranking
Benefits differ by question type
Large benefits for “what” and “which” questions, thanks to WordNet
Without WordNet customization
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
whe
n
wha
t
whe
re
ho
w
whi
ch
how
ma
ny
how
mu
ch
Question type
MR
R
Pre-reranking
Post-reranking
QA Chakrabarti 21
Conclusion A clean-room view of QA as
feature extraction plus learning• Recover structure info from question• Learn correlations between question
structure and passage features Competitive accuracy with negligible
domain expertise or manual intervention Ongoing work
• Use redundancy available from the Web• Model how selector and atype are related• Treat all question types uniformly