AQUAINT Program: Overview Dr. John Prange, Info-X R&D Thrust Director Dr. Lynn Franklin, Dep Info-X R&D Thrust Director [email protected]; [email protected]

AQUAINT Program: Overview

Dr. John Prange, Info-X R&D Thrust DirectorDr. Lynn Franklin, Dep Info-X R&D Thrust Director

[email protected]; [email protected] (Prange) / 443-479-6604 (Franklin)

301-688-7092 (ARDA Office)http://www.ic-arda.org

October 2004

2

Where is the Taj Mahal?

Let’s Start with a Simple, Factual, Question ---

Question ????

???

How Do We Find Information Today?

3

Traditional Information Retrieval (IR) Approach

Question ?

System SpecificQuery

e.g. Boolean Key WordEquation

DataSource

e.g Large Text

Archive

Traditional Information Retrieval

Ranked List of Hopefully “Relevant”

Documents. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

4

Where is the Taj Mahal?

Or Is It ???

Use Your Favorite Search Engine

It Depends !!!

Answer: Agra, India

5

Where is the Taj Mahal (“Hotel”)?

Alternative Answer #1

Answer: Bombay (Mumbai), India

6

Where is the (“Trump”) Taj Mahal?

Answer: Atlantic City, NJ


7


Where is the Taj Mahal (“Restaurant”)?

Answer: Utrecht, Netherlands

8

Single, Factoid Question ?

Ranked List of Hopefully “Relevant”

Documents. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

System SpecificQuery; often Tailored

to Question TypeTraditional Information Retrieval

SingleData

Source

Move Closerto the Questione.g. QuestionClassification

QA

Next Generation Approaches:Question Answering (QA) Systems

“Answer”

Move Closerto the Answere.g. Passage

Retrieval

ShallowAnalysis

9

“Ask Jeeves” Approach

•Start with Your Question

• Identify Key Words & Classifies the Type of Question

• Respond with rephrased “Questions” for which “Ask Jeeves” knows the Answer

• Provide Additional Web Sites as a fall back position (a la --- a more traditional web search engine)

10

Direct KnowledgeEntry by Domain

Experts

ParallelDevelopment by

Distributed Teams

Rapid KnowledgeFormation

Comprehensive(Million-Axiom)

Knowledge BasesGene rate plau sib le

crisis scenarios

Uncover co nnectedactivities, thre ats

Reason a bout no velcrisis situations

Mon ito r and in terpre tmassive da ta steams

Gene rate po ssiblecour se s of actions

Perfor m vulnerab ilityana lyses

Reason a bout no velbatt le fie ld sit uations

Mon ito r and in terpre tchang in g battlefield

event s

Answe r cause & effe ctque st io ns about events

Answe r question s aboutfor ce capabilities

Retrieve f acts relevant toa crisis

CrisisUnderstanding

Answe r question s aboutter rain

Commander’sAssociate

10 K

100 K

1,000 K

Need to create newknowledge at a rate of 400

axioms per hour

(With HPKB technology, a 5-personteam can create knowledge at a

rate o f 40 axioms per hour)

Biological Weapons (BW)Knowledge

• Basic knowledge of space, time,causality, general physics

• Biology, & biologica l threats• BW R&D, produce, weaponize• Geo-po litical behavior & terrorism

Required

6 Months 12 Months

10 K

100 K

1,000 K

HPKB

Development Time

UpperOntology

Mid-LevelTheories

Domain-S pecificTheories

Rapid Knowledge Formation (RKF)

Structured Knowledge-Base Approach

Deepest QA but Limited to Given Subject Domain

•Create comprehensive Knowledge Base(s) or other Structured Data Base(s)

• At the 10K Axiom Level -- Capable of Answering factual questions within domain

• At the 100K Axiom Level -- Answer cause & effect/capability Questions

• At the 1000K Axiom Level -- Answer Novel Questions; ID alternatives

11

Overarching Context / Operational Requirement

Who is thisadvisor?

What do weknow about

him/her?

What are his/her views?

What influence does he/she have on FM?

And still more questions ???

In a foreign news broadcast a team of analysts observe a previously unknown individual conferring with the Foreign Minister. They suspect

that he/she is really a new senior advisor.

Does this signal that other

policy changes are coming?

Information Analysts

Advanced Question Answering

12

Overarching Context /Operational Requirement

AdvancedQA

Extract & AnalyzeResults

DeeperAutomated

Understanding

Answers

Interpret Results& Formulate the Answers

Provide Answers in a Form

Analysts Want

Ranked Lists of

“Relevant” Data Objects

System SpecificQueries; Fully Tailoredto Series of Questions

ExtendTraditional Information Retrieval

MultipleHeterogeneous

DataSources

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

Multi-Media Multi-Media Structured Structured

Other Other

Text Text Voice Voice

InterpretingComplex

QA Scenario within a

Larger ContextFactoid

Questions?

WhyQuestions

?

InterpretiveQuestions?

Judgement Questions?

OtherQuestions?


Predictive Questions

?


13

1616QA Workshop - ACL 2001QA Workshop - ACL 2001





Ranked List of Hopeful ly “Relevant”

Documents


Documents



DataSource

e.g Large Text

Archive

DataSource

e.g Large Text

Archive

Traditional InformationRetrieval (IR) Approach

Question ?Question ?

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

2121QA Workshop - ACL 2001QA Workshop - ACL 2001

System SpecificQuery; often Ta ilored

to Question Type

System SpecificQuery; often Ta ilored

to Question Type


Documents


Documents. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .



SingleData

Source

SingleData

Source

Single, FactoidQuestion ?

Single, FactoidQuestion ?Move Closer

to the Questione.g. QuestionClassi fication

Move Closerto the Questione.g. QuestionClassi fication

Q&AQ&A

Next Generation Approaches:Question & Answering (Q&A) Systems

“Answer” “Answer”

Move Closerto the Answere.g. P assage

Retrieval

Move Closerto the Answere.g. P assage

Retrieval

ShallowAnalysisShallowAnalysis

Commercial World & Current R&D EffortsAre Addressing the Next GenerationBut Only Selected Content Understanding Barriers Are Being Aggressively Attacked

26QA Workshop - ACL 2001

Overarching Context /Operational Requirement

AdvancedQA

Extract & AnalyzeResults

DeeperAutomated

Understanding

Answers

Interpret Results& Formulate the Answers

Provide Answers in a Form

Analysts Want

Ranked Lists of

“Re levant” Data Objects

System SpecificQueries; Fully Tailoredto S eries of Questions

ExtendTraditionalInformationRetrieval

MultipleHeterogeneous

DataSources

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .. . . . . . . . . .

Mult i-Media Mult i-Media St ructured St ructured

Other Other

Text Text Voice Voice

Inte rpretingComplex

QA Scenario within a

Larger Context FactoidQuest ions?

WhyQuest ions?

InterpretiveQuest ions?

JudgementQuest ions?

OtherQuest ions?

InformationAnalysts

PredictiveQuest ions?


Multiple KeyBarriers toContentUnderstandingWill Be AggressivelyAttacked

Advanced Question Answering Is Skipping Ahead Two Generations

14

AQUAINTAdvanced QUestion & Answering for INTelligence

• What it is and What it is not . . .

– Question & Answering Aimed at the “Information Professional” --- Not just the Casual User

– Rich, Contextually-based Question Scenarios --- Full Range of Questions --- Not just Isolated, Factoid Questions

– Places much higher premium on knowledge and reasoning across very broad domains

– Open Domain, Multiple Media, Multiple Languages, Multiple Genre, Structured and Unstructured Data --- Not just a Focused Data Environment

15

Increasing Complexity Levels of Questions & Answers

Level 1”Simple

Factual QA’s"

Level 2"Template &

Multi-valued QA’s”

Level 3“Cross Media &

Cross Document QA’s"

Level 4”Context-Based

QA Scenarios”

FULL COMPLEXITY OF QUESTIONS & ANSWERS RANGES:

FROM: TO:

Questions: Simple Facts Questions: Complex; Uses Judgement TermsKnowledge of User Context Needed;Broad Scope

Answers: Simple Answers found in Answers: Search Multiple Sources (in multipleSingle Document Media/languages); Fusion of

information; Resolution of conflictingdata; Multiple Alternatives; AddingInterpretation; Drawing Conclusions

Current Near Term Mid Term Long Term

Advanced QA:Ramping up to the Full Complexity of Questions & Answers

16

Future

Fully Intersected;Automatically

Generated;Variable Structure/Format;

Full Context Responses

Full Context-Based

QuestionScenario

Level III

Full Context-Based

QuestionScenario

Fully Intersected;Automatically

Generated;Variable Structure/Format;

Full Context Responses

Level II

Variable NarrativeSummary;

Multi-Media Presentations;

Simple InterpretedResults

Cross MediaCross Document

Simple Judgement

Level I

Fixed Templatesor

Tabular Lists

Mulit-ValuedFactual QuestionsQuestions

Answers

Today

50/250 BytePassage from

Single TextDocument

SingleFactualIsolated

Questions

Data Chasm

Missing Data

MANY Heterogeneous Data Sources;

All Types, Sizes, Locations

IncreasingVolumes

(Petabyte & up)

Synthesis Across“Documents”/Media

ContradictoryDataReliability

of Data & SourceMultiple

Perspectives

Advanced QA:Attacking the Data Chasm

17

Structured / Semi-Structured

KB’s DB’s“Tagged Data”(e.g. Web Data)

Unstructured

HumanLanguage

Media

Sensor

Economic

Geospatial

VisualData

Video Still Images

Text

Documents

Speech

Multi-Media

Technical /Abstract

Other

Newswire / News Broadcast

Technical

Formal / Informal Communication

Other

Language

English

ForeignLanguage 1

ForeignLanguage 2

ForeignLanguage N

Genre

Advanced QA:Complex QA Across Data Types

18

Advanced QA:Much Deeper Understanding of Human Language is Required

• Some times SMALL differences can produce significantly different results/interpretations:

– Stop Words

• “Books {by; for; about} kids”

– Attachments

• “The man saw the woman in the park with the telescope.”

– Co-reference

• “John {persuaded; promised} Bill to go. He just left.”

• “Mary took the pill from the bottle. She swallowed it.”

• Other times BIG differences can produce the same/similar results:

– “Name the films in which Jude Law starred.”

– “Jude Law played a leading role in which movies?”

– “In what Hollywood productions did Jude Law receive top billing?”

19

Advanced QA:Is Time Our Achilles Heel?

• Real Difficulties Exist in:

– Extracting, correctly interpreting time references & then creating manageable timelines

– Estimating & updating changing reliability of information over time

– Processing information in time sequence e.g. Tracking the details of an evolving event over time -- A whole different set of problems

• And of course:

– We can’t forget all of the issues related to the timeliness of the system’s response to our question(s) -- we’ll need at least “near real time responses”

March April May June July August

20

• Different sources do not report simultaneously on an event.

• Data from different sources may be near real-time or take years to arrive.

• The hypothesis of today may be thrown out by new data arriving next week.

Event

H Hour

Event Planning Aftermath of the Event

H-n H-nH+1H-1

Collectors 1,2,3 observe event

Collector 1 reports

Collector 2 reports Collector

3 reports

Collector 4 observe event-related info &

reports

Collector 5 picks up historical

event planning material in a

raid

Collector 1 observes

event planning &

reports

• Analysis is dependent on a time continuum where data on a future event is found in the historical patterns established in event planning stages. As incoming data is evaluated against historical data, outcomes may change.

Advanced QA:The Challenge of Time in Analysis

21

Advanced QA:The Need for Ever Increasing Knowledge -- Of All Types

** Knowledge Requirement would be better represented with a whole “quiver of arrows” of different sizes, lengths and types

DIMENSIONS OF THE QUESTIONPART OF THE QA PROBLEM

DIMENSIONS OF THE ANSWERPART OF THE QA PROBLEM

Context

Judgement

Scope

Fusion

Interpretation

MultipleSources

QA R&D Program

QA R&D Program

Advanced AdvancedSimpleFactual

Question

SimpleAnswer,SingleSource

Increasing

Knowledge Requirements **

IncreasingKnowledgeRequirements **

22

• A Different Paradigm may be useful when handling QA Scenarios:

• Current Analytic Paradigm:

– Sequentially “Filter Down” to the

final result

Processing & Analysis

Data

Results

– Works when QA’s are independent, isolated activities

– Cast a “wider net” while searching

for “golden nuggets” (Answers)

AnswersSpace of Data Objects and Sources

How Wide to Cast the “Net”?

What Info to Retain? In what form?

For how long?

– Automatically Extract, Represent,

and Preserve “closely related”

background information within

context of the QA Scenario

Background

Discarded

Advanced QA:The Need for a Different Paradigm

23

Overarching Context / Operational Requirement

Who is thisadvisor?

What do weknow about

him/her?

What are his/her views?

What influence does he/she have on FM?

And still more questions ???

In a foreign news broadcast a team of analysts observe a previously unknown individual conferring with the Foreign Minister. They suspect

that he/she is really a new senior advisor.

Does this signal that other

policy changes are coming?


Advanced QA:Need for Improved Reasoning & Learning

FOCUS

24

Advanced Reasoning:• Use Multi-level Plans• Create and evaluate chains of reasoning• Reason across heterogeneous data sources• Infer answers from data extracted from multiple sources when the answer is not explicitly stated • Utilize Link Analysis & Evidence Discovery• Plus other strategies

New SeniorAdvisor

Associates Associates Follow-upLeads

Follow-upLeads

“Bio”………..….……..…….………..….……..…….………..….……..…….…………...

“Views: Past & Present” .….… ….…...……. ….…...……. ….…...……. ….…...……. ….…..

Summarized Results

Collected Views

TV & RadioBroadcasts,Newspapers

& OtherArchives

Raw “Bio”Information

Education

Past Positions

Family

Travels

Other Activities

Summarized Results

Cross Fertilization

Advanced Learning:• Automatically learn new or modify existing reasoning strategies

Advanced QA:Need for Improved Reasoning & Learning

25

Interested External

Stakeholders

ARDA’s Info-X Program Partners

Active IC /Government

Partners

RecentAdditions

• NGIC• DHS

26

QUESTION????

Clarification

Other Analysts

Question & RequirementContext; Analyst Background

Knowledge

Multimedia Examples

Natural Statement ofQuestion;

Use of

QueryAssessment,

Advisor,Collaboration

Question Under- standing andInterpretation

Knowledge Bases;Technical Databases

AQUAINT:R&D Focused on Three Functional Components

Question & Answer Context

•Relevant information extracted and combined where possible;•Accumulation of Knowledge across “Documents”•Cross “Document” Summaries created;•Language/Media Independent Concept Representation•Inconsistencies noted;•Proposed Conclusions and Inferences Generated

Determinethe

Answer

Relevant “Documents”

MultipleRanked

Lists

Single, Merged

Ranked List ofRelevant “Documents”

Queries

Relevant“Knowledge”

KBQueries

Multiple Sources;Multiple Media;Multi-Lingual;Multiple Agencies

MultipleSource

SpecificQueries

Translate Queriesinto Source Specific Retrieval Languages

Partially Annotated & Structured Data

Automatic Metadata Creation

SupplementalUse

Supple- mentalUse

Query Refinement based on Analyst

Feedback

Iterative Refinementof Results based

on Analyst Feedback

AnalystFeed-back

FINAL ANSWER

Results of Analysis• Formulate Answer for Analyst in form they want

• Multimedia Navigation Tools for Analyst Review

AnswerFormulation

ProposedAnswer

AnswerContext

Operational Requirement / Cognitive Environment

27

Cross Cutting/Enabling Technologies Research Issues

QUESTION????

FINAL ANSWER

AnswerFormulation

Question Under-

standing and Inter-pretation

InformationRetrievalProcess

Analysis &SynthesisProcess

Determinethe Answer

AQUAINTPhase I

Solicitation

Annotated and ‘Ground Truthed’ Data

Component Level / End-to-End Testing & Evaluation

Component Integration and System Architecture Issues

SeparateCoordinated

Activities

AQUAINT:Separate, Coordinated Activities

28

AQUAINT Program Contractors

CarnegieMellonUniv. Univ. of

Albany

Univ. ofMassachusetts

BBN (2)

IBM

Columbia Univ.

Rutgers Univ.

Princeton Univ.

Univ. of Texas-Dallas

Language Computer Corp. (2)

CycorpSAIC

Univ. of SouthernCalifornia

/ Info ScienceInstitute

SRI

Stanford Univ.

Univ. of California-Berkeley

Univ. of Colorado-Boulder

OriginalHNC Software New Mexico

State University (2)

Univ. of Maryland –Baltimore County (UMBC)

CoGen Tex

Language Computer Corp.

Univ. of SouthernCalifornia

/ Info ScienceInstitute

CarnegieMellon

Univ. (2)

+ New

33

AQUAINT Program Phase 2 Contractors

Prime Contractors (18)

ColumbiaUniv.

Palo AltoResearch

Center

Arizona State Univ.

IBM T. J.Watson Center

Univ. of Illinois-Urbana-Champaign

Language ComputerCorporation (2)

Univ. of TexasAt Dallas

Cycorp

CarnegieMellon

Univ. (2)

Univ. ofPittsburgh

Sub Contractors (16)

Rutgers Univ.

USC

USC / ISI (2)

TexasTech

MonmouthUniv.

UC-Berkeley(ICSI)

StanfordUniv. (2)

Univ. of Colorado

Univ. ofPennsylvania

Lehman College

Univ. of Utah

Cornell Univ.

GeorgetownUniv.

MITRE

BBN

BrandeisUniv.

Princeton Univ.

Univ. ofAlbany

SPAWAR

MIT

34

AQUAINT Phase 2 Projects (Spring 04 – Spring 06)

Total End-to-End Systems (10) (Systems 1-5)

Organization Title Investigator Topical Focus

Data Dimension ARDA Agent

University of Texas at Dallas // University of California-

Berkeley (International Computer Science Institute)

// Stanford University

AQUINAS: Answering QUestions using INference and Advanced Semantics

Sanda Harabagiu // Srini Narayanan // Chris

Manning

End-to-End

System

Focused Data Strategy

CIA

Cycorp GINKO, an End-to-end Intelligence QA System

Based on Contextualized Knowledge Dossiers

David Schneider &

Michael Whitbrock

End-to-End

System

Diverse Data Strategy

CIA

Language Computer Corporation

Advanced Techniques for Multimodal Question

Answering

Dan Moldovan End-to-End

System


CIA

Language Computer Corporation

HIREQA-ML: High Precision Interactive Question

Answering Using Multiple Languages

Sanda Harabagiu End-to-End

System


CIA

Massachusetts Institute of Technology

A Tripartite Question Answering Architecture for

Integrating Diverse Knowledge Resources

Boris Katz End-to-End

System


CIA

35

Total End-to-End Systems (10) (Systems 6-10)

Organization Title Investigator Topical Focus

Data Dimension ARDA Agent

Columbia University // University of Colorado //

Stanford Univ. // Univ. of Texas-

Dallas

Fusing Rich Information Extracted from Multiple Media and Languages to

Generate Contextualized Complex Answers

Vasileios Hatzivassiloglou & Kathleen McKeown // Dan Jurafsky & James Martin & Wayne Ward

End-to-End

System


DIA

University at Albany,(SUNY) /

Rutgers University / Lehman College

(CUNY)

HITIQA-2: the Intelligence Analyst’s Assistant in High Quality Interactive

Question Answering

Tomek Strzalkowski // Paul Kantor // Boris

Yamrom

End-to-End

System


DIA

BBN Technologies Breaking the Cross-lingual Barrier to Question Answering

Ralph Weischedel End-to-End

System


NSA

IBM T. J. Watson Research Center

A Question-Answering (QA)-Based Information Gathering Environment

David Ferrucci & John Prager

End-to-End

System


NSA

Carnegie Mellon University

JAVELIN II: Scenarios and Variable-Precision Reasoning for Advanced

Question Answering from Multilingual, Distributed Sources

Eric Nyberg & Teruko Mitamura

End-to-End

System


NSA


36

Organization Title Investigator Topical Focus Data Dimension ARDA Agent

Arizona State University // Texas Tech University //

Monmouth University

Answering complex questions and performing deep reasoning in advanced question answering

systems

Chitta Baral / Richard Scherl // Michael Gelfond

Component Elements


CIA

Carnegie Mellon University // University of

Southern California

Informedia Contexture: Analyzing and Synthesizing Video and Verbal

Context for Intelligence Analysis Dialogues

Howard D. Wactlar // Ram Nevatia

Component Elements


NSA

Emphasis on One or more Advanced QA System Components (2)


37

Organization Title Investigator Topical Focus Data Dimension ARDA Agent

Palo Alto Research Center, Inc.

Two-way Bridge between Language and Logic

Daniel G. Bobrow & Ron Kaplan

Cross Cutting / Enabling

Technologies


CIA

Princeton University // University of Southern California (Information

Sciences Institute)

WordNet for Question Answering

Christiane Fellbaum / George Miller // Jerry

Hobbs


Technologies

Focus Data Strategy

NSA

Brandeis University // Georgetown University // University of Southern California (Information

Sciences Institute)

Temporal Awareness Algorithms for Natural

Language Texts

James Pustejovsky // Inderjeet Mani // Jerry

Hobbs


Technologies

Focus Data Strategy

NSA

University of Pittsburgh // Cornell University //

University of Utah

Opinions in Question Answering

Janyce Wiebe // Claire Cardie // Ellen Riloff


Technologies


DIA

University of Illinois at Urbana-Champaign //

University of Pennsylvania

Kindle: Knowledge and Inference via Description

Logics for Natural Language

Dan Roth // Martha Palmer


Technologies


NSA

SPAWAR // MITRE Gazatteer Exploitation for Question Answering

Beth Sundheim // Scott Mardis


Technologies


ARDA

Focused Effort -- Cross Cutting / Enabling Technologies (6)


38

HIGHLIGHTS• Dramatic progress on linguistic approach that converts question and

relevant passages into logical forms and then arrives at answer through a powerful combination of an extended “WordNet” and a logic prover


39



• Made significant strides in extending QA from isolated, factoid questions to far more complex “Who is / What is” questions that require combining information from multiple, potentially duplicative or contradictory document sources


40

More Complex Question Types

• Definitions– What is Tikrit?

• Biographies – Who is Mahmoud Abbas?

• Events– What happened in Baghdad on Thanksgiving?

• Different Perspectives / Opinions– What people think of Mahmoud Abbas’ resignation?

• Lists– What names of chewing gums are found in the AQUAINT corpus?

• Relationships– The analyst is interested in the line of succession of the Saudi

government, and the relationship between the individuals in their royal family. King Fahd is the current ruler, but is in poor health. Who is next in line, and what is his relationship to King Fahd? Who, if anyone, has been designated as second in line?

41

Example Definition *

What is Tikrit?

Tikrit is a power center for Sunni Arab tribes that may hold out for as long as possible out of fear of losing power to the nation’s Shiite majority (12). Baghdad may be the capital of Iraq, but Tikrit is Saddam country (15). Other experts caution that the years of preferential treatment towards the residents of Tikrit may cause them to stand by Saddam Hussein to the end (4). …

* Reference: Columbia Univ. / Univ. of Colorado AQUAINT Briefing

42




• Progress made on developing multi-engine QA system that combines linguistic, statistical & KB approaches


43

Available Answering Agents

• Predictive Annotation Agent– General-purpose agent, used in almost all cases.

• Statistical Query Agent– Also general-purpose. Courtesy Roukos/Ittycheriah

• Description Agent– Generic descriptions (appositions, parentheticals etc.)

• Structured Knowledge Agent– Answers from WordNet/KSP/Cyc

• Pattern-Based Agent– Looks for specific syntactic patterns based on semantic form

• Dossier Agent– Calls PIQUANT recursively with multiple factoid questions

• Profile Agent– Currently standalone – used for Relationship Pilot

* Reference: IBM AQUAINT Briefing

44

PIQUANT Architecture *

KSP-BasedAnswering Agent

KSP-BasedAnswering Agent

Predictive Annot.Answering AgentPredictive Annot.Answering Agent

Answering Agents

StatisticalAnswering Agent

StatisticalAnswering Agent

Definitional QAnswering Agent

Definitional QAnswering Agent

Question

AnswerResolution

Answer

QGoals

Answers

QFrameAnswerClassification

QuestionAnalysis

Knowledge Source Portal

Semantic Search

WordNet

Cyc

KeywordSearch

Pattern-BasedAnswering Agent

Pattern-BasedAnswering Agent

QPlanGenerator

QPlanExecutor

AQUAINT

TREC

EB

Web

CNS

* Reference: IBM AQUAINT Briefing

45* Reference: IBM AQUAINT Briefing

Multiple QA Agents Approach *What is the largest city in England?

• Text Match– Find text that says “London is the largest city in England” (or

paraphrase). Confidence is confidence of NL parser * confidence of source.

• “Superlative” Search– Find a table of English cities and their populations, and sort.– Find a list of the 10 largest cities in the world, and see which are in

England. • Uses logic: if L > all objects in set R then L > all objects in set E R.

– Find the population of as many individual English cities as possible, and choose the largest.

• Heuristics– London is the capital of England. (Not guaranteed to imply it is the

largest city, but this is very frequently the case.)

• Complex Inference – E.g. “Birmingham is England’s second-largest city”; “Paris is larger

than Birmingham”; “London is larger than Paris”; “London is in England”.

46




• Progress made on developing multi-engine QA system that combines linguistic, statistical & KB approaches

• Executed Pilot Evaluations for multiple complex QA Types; Developed Metrics for evaluating QA Systems at the Scenario Task Level; Full Evaluation of all End-to-End QA Systems late in Phase 2


47

June Sunrise over Kirkwall Bay in the Orkney Islands of Scotland

Your QuestionsYour Questions& Comments& Comments

48

Contact Information

Dr. John Prange, Info-X R&D Thrust Program DirectorDr. Lynn Franklin, Info-X R&D Thrust Program Dep Dir

• Web Pages: http://www.ic-arda.org (Internet)

• Phones: 443-479-8006 (Prange) 443-479-6604

(Franklin) 301-688-7092 (ARDA Office)

800-276-3747 (ARDA Office)

• FAX: 301-688-7401 (ARDA Office)

• E-Mail: [email protected] (Internet E-Mail)

[email protected] (Internet E-Mail) [email protected](Internet E-Mail)

• Location: Room 12A69 NBP #1Suite 6644

9800 Savage RoadFort Meade, MD 20755-6644