Question Answering (and Textual...

QuestionAnswering(andTextualEntailment)

Prof.SameerSinghCS295:STATISTICALNLP

WINTER2017

March14,2017

BasedonslidesfromDanJurafsky,YejinChoi,StephenClark,DanKlein,Niranjan Balasubramanian,andeveryoneelsetheycopiedfrom.

Upcoming…

• Homework4wasdueonlastnight• Lowestgradeofthehomeworks willbedroppedHomework

• Finalreportdueinaweek:March20,2017• Instructionscomingsoon:ACLstyle,5pages(+references)Project

• Papersummariesduetonight• Summary2gradedSummaries

CS295:STATISTICALNLP(WINTER2017) 2

TA/InstructorEvaluationsareavailable!

Outline

QuestionAnswering

TextualEntailment

IR-BasedQASystem

OtherExtensions

Outline

QuestionAnswering

TextualEntailment

IR-BasedQASystem

OtherExtensions

QuestionsinModernSystemsFactoidquestions◦ Whowrote“TheUniversalDeclarationofHumanRights”?◦ Howmanycaloriesarethereintwoslicesofapplepie?◦ Whatistheaverageageoftheonsetofautism?◦ WhereisAppleComputerbased?

Complex(narrative)questions:◦ Inchildrenwithanacutefebrileillness,whatistheefficacyofacetaminopheninreducingfever?

◦ WhatdoscholarsthinkaboutJefferson’spositionondealingwithpirates?

Commercialsystems:mainlyfactoidquestions

WhereistheLouvreMuseumlocated? InParis,France

What’stheabbreviation forlimitedpartnership? L.P.

What arethenamesofOdin’sravens? Huginn andMuninn

What currencyisusedinChina? Theyuan

Whatkindofnutsareusedinmarzipan? almonds

WhatinstrumentdoesMaxRoachplay? drums

WhatisthetelephonenumberforStanfordUniversity? 650-723-2300

ParadigmsforQA

IR-basedapproaches◦ TREC;IBMWatson;Google

Knowledge-basedandHybridapproaches◦ IBMWatson;AppleSiri;WolframAlpha;TrueKnowledgeEvi

Manyquestionscanalreadybeansweredbywebsearch

IR-basedQuestionAnswering

IR-basedFactoidQA

DocumentDocumentDocument

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Answer Type Detection

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

Knowledge-basedQA(Siri)Buildasemanticrepresentationofthequery◦ Times,dates,locations,entities,numericquantities

Mapfromthissemanticstoquerystructureddataorresources◦ Geospatialdatabases◦ Ontologies(Wikipediainfoboxes,dbPedia,WordNet,Yago)◦ Restaurantreviewsourcesandreservationservices◦ Scientificdatabases

Hybridapproaches(Watson)Buildashallowsemanticrepresentationofthequery

GenerateanswercandidatesusingIRmethods◦ Augmentedwithontologiesandsemi-structureddata

Scoreeachcandidateusingricherknowledgesources◦ Geospatialdatabases◦ Temporalreasoning◦ Taxonomicalclassification

IBM’sWatson

WonJeopardy onFebruary16,2011!

WILLIAMWILKINSON’S“ANACCOUNTOFTHEPRINCIPALITIESOF

WALLACHIAANDMOLDOVIA”INSPIREDTHISAUTHOR’SMOSTFAMOUSNOVEL

BramStoker

CS295:STATISTICALNLP(WINTER2017)

MotivationforWatson

SingleSourceisnotSufficient

WatsonArchitecture

Models

Answer & Confidence

Question

Evidence

Sources

Models

ModelsPrimarySearch

CandidateAnswer

Generation

Answer Sources Evidence

RetrievalEvidence Scoring

Learned Modelshelp combine and

weigh the Evidence

Hypothesis and Evidence Scoring

Merging &

Ranking

Synthesis

WatsonPerformance

Outline

QuestionAnswering

TextualEntailment

IR-BasedQASystem

OtherExtensions

IR-basedFactoidQA

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

IR-basedFactoidQAQUESTIONPROCESSING◦ Detectquestiontype,answertype,focus,relations◦ Formulatequeriestosendtoasearchengine

PASSAGERETRIEVAL◦ Retrieverankeddocuments◦ Breakintosuitablepassagesandrerank

ANSWERPROCESSING◦ Extractcandidateanswers◦ Rankcandidates

◦ usingevidencefromthetextandexternalsources

FactoidQ/A

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

QuestionProcessingAnswerTypeDetection◦ Decidethenamedentitytype(person,place)oftheanswer

QueryFormulation◦ ChoosequerykeywordsfortheIRsystem

QuestionTypeclassification◦ Isthisadefinitionquestion,amathquestion,alistquestion?

FocusDetection◦ Findthequestionwordsthatarereplacedbytheanswer

RelationExtraction◦ Findrelationsbetweenentitiesinthequestion

QuestionProcessing

AnswerType:USstate

Query:twostates,border,Florida,north

Focus:thetwostates

Relations:borders(Florida,?x,north)

They’rethetwostatesyoucouldbereenteringifyou’recrossingFlorida’snorthernborder

AnswerTypes:NamedEntities

WhofoundedVirginAirlines?PERSON

WhatCanadiancityhasthelargestpopulation?CITY

PartofAnswerTypeTaxonomy

LOCATION

NUMERIC

ENTITY HUMAN

ABBREVIATIONDESCRIPTION

country city state

datepercent

sizedistance

individual

currency

animal

definition

reason expression

abbreviation

Li,Roth.LearningQuestionClassifiers.COLING(2002)

AnswerTypes

MoreAnswerTypes

AnswertypesinWatson

2500answertypesin20,000Jeopardyquestionsample◦ Themostfrequent200answertypescover<50%ofdata

The40mostfrequentJeopardyanswertypes

he,country,city,man,film,state,she,author,group,here,company,president,capital,star,novel,character,woman,river,island,king,song,part,series,sport,singer,actor,play,team,show,actress,animal,presidential,composer,musical,nation,book,title,leader,game

CS295:STATISTICALNLP(WINTER2017) 28Ferrucci etal.BuildingWatson:AnOverviewoftheDeepQA Project.AIMagazine.2010

AnswerTypeDetection

Hand-writtenRules

Regularexpression-basedrulescangetsomecases:◦ Who{is|was|are|were}PERSON

Otherrulesusethequestionheadword:◦ (theheadwordofthefirstnounphraseafterthewh-word)◦ WhichcityinChinahasmostforeignfinancialcompanies?◦ WhatisthestateflowerofCalifornia?

MachineLearning

QuestionwordsandphrasesPart-of-speechtagsParsefeatures(headwords)NamedEntitiesSemanticallyrelatedwords

FactoidQ/A

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

KeywordSelectionAlgorithm1.Selectallnon-stopwordsinquotations2.SelectallNNPwordsinrecognizednamedentities3.Selectallcomplexnominals withtheiradjectivalmodifiers4.Selectallothercomplexnominals5.Selectallnounswiththeiradjectivalmodifiers6.Selectallothernouns7.Selectallverbs8.Selectalladverbs9.SelecttheQFWword(skippedinallprevioussteps)10.Selectallotherwords

CS295:STATISTICALNLP(WINTER2017) 31Moldovan,Harabagiu,Pasca,Mihalcea,Goodrum,GirjuandRus.TREC(1999)

Choosingkeywords

Whocoinedtheterm“cyberspace”inhisnovel“Neuromancer”?

cyberspace/1Neuromancer/1term/4novel/4coined/7

FactoidQ/A

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

PassageRetrieval

RetrievedocumentsusingIR◦ querytermsaskeywords

Step2Segmentthedocumentsintoshorterunits◦ somethinglikeparagraphs

Step3Passageranking◦ Useanswertypetohelprerank passages

FeaturesforPassageRanking

NumberofNamedEntitiesoftherighttypeinpassage

Numberofquerywordsinpassage

NumberofquestionN-gramsalsoinpassage

Proximityofquerykeywordstoeachotherinpassage

Longestsequenceofquestionwords

Rankofthedocumentcontainingpassage

Eitherinrule-basedclassifiersorwithsupervisedmachinelearning

FactoidQ/A

DocumentDocume

ntDocumentDocume

ntDocument

Question Processing

PassageRetrieval

Query Formulation

Question

Passage Retrieval

Document Retrieval

Answer Processing

Answer

passages

Indexing

RelevantDocs

AnswerExtractionRunananswer-typenamed-entitytaggeronthepassages◦ Eachanswertyperequiresanamed-entitytaggerthatdetectsit◦ IfanswertypeisCITY,taggerhastotagCITY◦ CanbefullNER,simpleregularexpressions,orhybrid

Returnthestringwiththerighttype:◦ WhoistheprimeministerofIndia(PERSON)◦ ManmohanSingh,PrimeMinisterofIndia,hadtoldleftleadersthatthedealwouldnotberenegotiated.

◦ HowtallisMt.Everest?(LENGTH)◦ TheofficialheightofMountEverestis29035feet

RankingCandidateAnswersButwhatiftherearemultiplecandidateanswers!

Q:WhowasQueenVictoria’ssecondson?

AnswerType:Person

• Passage:TheMariebiscuitisnamedafterMarieAlexandrovna,thedaughterofCzarAlexanderIIofRussiaandwifeofAlfred,thesecondsonofQueenVictoriaandPrinceAlbert

RankingCandidateAnswersButwhatiftherearemultiplecandidateanswers!

Q:WhowasQueenVictoria’ssecondson?

AnswerType:Person

• Passage:TheMariebiscuitisnamedafterMarieAlexandrovna,thedaughterofCzarAlexanderIIofRussiaandwifeofAlfred,thesecondsonofQueenVictoriaandPrinceAlbert

FeaturesforMLAnswertypematch:Candidatecontainsaphrasewiththecorrectanswertype.

Patternmatch:Regularexpressionpatternmatchesthecandidate.

Questionkeywords:#ofquestionkeywordsinthecandidate.

Keyworddistance:Distanceinwordsbetweenthecandidateandquerykeywords

Noveltyfactor:Awordinthecandidateisnotinthequery.

Appositionfeatures:Thecandidateisanappositivetoquestionterms

Punctuationlocation:Thecandidateisimmediatelyfollowedbyacomma,period,quotationmarks,semicolon,orexclamationmark.

Sequencesofquestionterms:Thelengthofthelongestsequenceofquestiontermsthatoccursinthecandidateanswer.

ScoringCandidatesinWatsonEachcandidateanswergetsscoresfrom>50components◦ (fromunstructuredtext,semi-structuredtext,triplestores)◦ logicalform(parse)matchbetweenquestionandcandidate◦ passagesourcereliability◦ geospatiallocation◦ Californiais”southwestofMontana”

◦ temporalrelationships◦ taxonomicclassification

Outline

QuestionAnswering

TextualEntailment

IR-BasedQASystem

OtherExtensions

AskMSR

QuestionProcessing Search

AnswerExtractionAnswerScoring

Dumais, Banko, Brill, Lin, Ng,SIGIR(2002)

Step1:RewriteQueries

Intuition: Questions are often syntactically quite closeto sentences with the answer• Where istheLouvreMuseumlocated?

• TheLouvreMuseumislocated in Paris• Who createdthecharacterofScrooge?

• Charles DickenscreatedthecharacterofScrooge

FeedbackLoops:FALCON

AllenAIScienceChallenge

Whichobjectisthebestconductorofelectricity?(A) awaxcrayon(B)aplasticspoon(C)arubbereraser(D)anironnail

Fourthgradersareplanningaroller-skaterace.Whichsurfacewouldbethebestforthisrace?(A)gravel(B)sand(C)blacktop(D)grass

Fourthgradersareplanningaroller-skaterace.Whichsurfacewouldbethebestforthisrace?(A)gravel(B)sand(C)blacktop (D)grass

§ Informationretrievalmethodsfail§ Wordco-occurrencemethodsstruggle

Graders are commonly used in the construction and maintenance of dirt roads and gravel roads

Also strong correlations between:grass « racegravel « surface

Astudentputstwoidenticalplantsinthesametypeandamountofsoil.Shegivesthemthesameamountofwater.Sheputsoneoftheseplantsnearasunnywindowandtheotherinadarkroom.Thisexperimenttestshowtheplantsrespondto

(A)light(B)air(C)water(D)soil

Astudentputstwoidenticalplantsinthesametypeandamountofsoil.Shegivesthemthesameamountofwater.Sheputsoneoftheseplantsnearasunnywindowandtheotherinadarkroom.Thisexperimenttestshowtheplantsrespondto

(A)light (B)air(C)water(D)soil

Knowledge needed(forexample):§ nearsunnywindow® receivelight§ inadarkroom® nolight§ testX’sresponsetoY

® compareX+YwithX+not(Y)

Outline

QuestionAnswering

TextualEntailment

IR-BasedQASystem

OtherExtensions

NaturalLanguage&Meaning

Meaning

Language

AmbiguityVariability

interpretatio

n expression

InferencevsEntailment

MeaningRepresentation

Natural Language

Inference

TextualEntailment

• Adirectionalrelationbetweentwotextfragments:Text(t) andHypothesis(h):

t entails h (t Þ h) if humans reading t will infer that h is most likely true

• Assuming“commonbackgroundknowledge”–whichisindeedexpectedfromapplications

Example

CS295:STATISTICALNLP(WINTER2017) 56https://cogcomp.cs.illinois.edu/page/resource_view/9

MoreSentencePairs

1. Some students came to school by car.Some students came to school.

2. No students came to school by car. Some students came to school.

3. John drove legally. John drove.

4. John drove predictably.John drove.

5. Legally, John could drive.John drove.

EntailmentwithKnowledge

Fortextualentailmenttoholdwerequire:◦ text ANDknowledgeÞ h,but◦ knowledgeshouldnotentailh alone

Systemsare notsupposedtovalidateh’struthregardlessoft(e.g.bysearchinghontheweb)

t entails h (t Þ h) if humans reading t will infer that h is most likely true

Example

TEXT: …WhilenooneaccusesMadonnaofdoinganythingillegalinadoptingthe4-year-oldgirl,reportedlynamedMercy,therearequestionsnonethelessabouthowMadonnaisabletonavigateMalawi's18-to-24monthvettingperiodinjustamatterofdaysorweeks…

HYPOTHESIS:

Madonnais50yearsold.

ThirdLabel:Contradictions

t contradicts h (t ^ h) if humans reading t will find the relations/events described by h to be highly unlikely given t.

Example

MoreExamples

Applications

QuestionAnswering

InformationExtraction

MachineTranslation

InformationRetrieval

Question Expected answer formWho bought Overture? >> X bought Overture

Overture’s acquisitionby Yahoo

Yahoo bought Overture

text hypothesized answer

entails

Heabhorredthemen’sunctuousways.

Hedislikedthemen’sflatteringways.

Similaritybasedonwhetherdocumententailsthequery.

Question Answering (and Textual...

Documents

Relation Alignment for Textual Entailment Recognition

Global Learning of Textual Entailment Graphs

Overview of the Fourth Recognising Textual Entailment Challenge

The Sixth PASCAL Recognizing Textual Entailment Challenge

Textual Entailment, QA4MRE, and Machine Reading Peter Clark Vulcan Inc

Global Learning of Textual Entailment Graphs...Global Learning of Textual Entailment Graphs Jonathan Berant The Blavatnik School of Computer Science, Raymond and Beverly Sackler Faculty

Assessing the Impact of Frame Semantics on Textual Entailment

Qall-Me – Qall-Me Framework and Textual …qallme.fbk.eu/Qallme-Edits.pdfQall-Me – Qall-Me Framework and Textual Entailment Based Question Answering Author: Qall-Me Consortium

NATURAL LANGUAGE INFERENCE: FROM TEXTUAL ENTAILMENT …jchai/Thesis/chen-zhang-dissertation-2010.pdf · NATURAL LANGUAGE INFERENCE: FROM TEXTUAL ENTAILMENT TO CONVERSATION ENTAILMENT

From Textual Entailment to Knowledgeable Machines

A Textual Entailment System using Web based Machine

Textual Entailment - UPC Universitat Politècnica de Catalunyaageno/anlp/textualEntailment.pdf · Textual Entailment 8 Textual Entailment • Textual entailment recognition is the

A Machine Learning based Textual Entailment Recognition System of

Textual Entailment 1 Textual Entailment Al Akhwayn Summer School 2015 Horacio Rodríguez TALP Research Center Dept. Computer Science Universitat Politècnica

Recognizing Partial Textual Entailment

IMTKU Textual Entailment System Demoresearch.nii.ac.jp/ntcir/workshop/OnlineProceedings9/NTCIR/08... · IMTKU Textual Entailment System for Recognizing Inference in Text at NTCIR-9

TEXTUAL ENTAILMENT - profs.info.uaic.roadiftene/rezumatAI.pdf2 1. Introducere Recunoa şterea inferen ţelor textuale (textual entailment recognition) (Dagan et al., 2005) este una

IMTKU Textual Entailment System for Recognizing …research.nii.ac.jp/.../RITE/12-NTCIR10-RITE2-TuC_slides.pdfIMTKU Textual Entailment System for Recognizing Inference in Text at NTCIR-10

Building Evaluation Sets for Textual Entailment Recognition

The Fifth PASCAL Recognizing Textual Entailment Challenge...1 Introduction Textual Entailment Recognition is a generic task that captures major semantic inference needs across many