Informa(on Extrac(on · My mug broke into pieces immediately. The robot broke my favorite mug with a wrench. Semantic Role Labeling (SRL) 3 subj v obj prep subj v prep adv thing broken

Informa(onExtrac(on

WeiXu(many slides from Greg Durrett, Luheng He, Emma Strubell)

ThisLecture

‣ Howdowerepresentinforma(onforinforma(onextrac(on?

‣ OpenInforma(onExtrac(on

‣ Rela(onextrac(on

‣ Slotfilling

‣ Seman(crolelabeling/abstractmeaningrepresenta(on

IE:TheBigPicture

‣ Howdowerepresentinforma(on?Whatdoweextract?

‣ Slotfillers‣ En(ty-rela(on-en(tytriples(fixedontologyoropen)

‣ Seman(croles

‣ Abstractmeaningrepresenta(on

Seman(cRoleLabeling

Seman(cRoleLabeling

‣ Findout5Wintext—“whodidwhattowhom,whenandwhere”

‣ Iden(fypredicate,disambiguateit,iden(fythatpredicate’sarguments

FigurefromHeetal.(2017)

Ques(on-AnswerDrivenSRL

FigurefromFitzGeraldetal.(2018)

Seman(cRoleLabeling

My mug broke into pieces immediately.

The robot broke my favorite mug with a wrench.

Semantic Role Labeling (SRL)

3 FigurefromHeetal.(2017)

Seman(cRoleLabeling





3

vsubj obj prep

subj v prep adv

Seman(cRoleLabeling




3

vsubj obj prep

subj v prep adv

thing broken

thing broken

breaker instrument

pieces (final state) temporal


Seman(cRoleLabeling




3

vsubj obj prep

Frame: break.01role description

ARG0 breakerARG1 thing brokenARG2 instrumentARG3 pieces

ARG4 broken away from what?

subj v prep adv

thing broken

thing broken

breaker instrument

pieces (final state) temporal

ARG0 ARG1 ARG2

ARG3ARG1 ARGM-TMP

�11 4

The Proposition Bank (PropBank)

Core roles: Verb-specific roles (ARG0-

ARG5) defined in frame files

Frame: break.01

role descriptionARG0 breakerARG1 thing broken

ARG2 instrument

ARG3 pieces

ARG4 broken away from what?

Frame: buy.01

role descriptionARG0 buyerARG1 thing bough

ARG2 seller

ARG3 price paid

ARG4 benefactive

role descriptionTMP temporalLOC locationMNR mannerDIR directionCAU causePRP purpose…

Adjunct roles: (ARGM-) shared

across verbs

Annotated on top of the Penn Treebank Syntax

PropBank Annotation Guidelines, Bonial et al., 2010

4

Paul Kingsbury and Martha Palmer.From Treebank to PropBank. 2002TheProposi(onBank(PropBank)


Seman(cRoleLabeling


‣ Iden(fypredicates(love)usingaclassifier(notshown)

‣ Iden(fyARG0,ARG1,etc.asataggingtaskwithaBiLSTMcondi(onedonlove

‣ Othersystemsincorporatesyntax,jointpredicate-argumentfinding

Residual/HighwayConnec(ons

�1323

input from the previous layer

recurrent input

from the prev. timestep

output to the next layer

References: Deep Residual Networks, Kaiming He, ICML 2016 Tutorial

Training Very Deep Networks, Srivastava et al., 2015

Non-linearity

ht shortcut

ht�1 F(ct�1,ht�1,xt)

residual netht + xt

gated highway network:

rt � ht + (1� rt) � xt

rt = �(f(ht�1,xt))

xt

xt

new output:

Model - (2) Highway ConnectionsDeep BiLSTM Tagger Highway Connections Variational

DropoutViterbi Decoding w\

Hard Constraints

23

input from the previous layer

recurrent input

from the prev. timestep

output to the next layer

References: Deep Residual Networks, Kaiming He, ICML 2016 Tutorial

Training Very Deep Networks, Srivastava et al., 2015

Non-linearity

ht shortcut

ht�1 F(ct�1,ht�1,xt)

residual netht + xt

gated highway network:

rt � ht + (1� rt) � xt

rt = �(f(ht�1,xt))

xt

xt

new output:

Model - (2) Highway ConnectionsDeep BiLSTM Tagger Highway Connections Variational

DropoutViterbi Decoding w\

Hard Constraints


SRLSystems

�1416

SRL Systems

syntactic features

candidate argument spans

labeled arguments

prediction

labeling

ILP/DP

sentence, predicate

argument id.

Pipeline Systems

Deep BiLSTM

Hard constraints

BIO sequence

prediction

sentence, predicate

*This work

Punyakanok et al., 2008 Täckström et al., 2015 FitzGerald et al., 2015

sentence, predicate

BIO sequence

prediction

Deep BiLSTM + CRF layer

Viterbi

context window features

End-to-end Systems

Collobert et al., 2011 Zhou and Xu, 2015 Wang et. al, 2015

He et al., 2017

The System on previous slide


(syntax-based)

[LISA]

10YearsofPropBankSRL

F1 Syntax-based

End-to-enddeep NN

[FitzGerald, Täckström, Ganchev, Das]

[Täckström, Ganchev, Das]

[Zhou & Xu]

[He, Lee, Lewis, Zettlemoyer] [He, Lee, Levy,

Zettlemoyer]

[Tan, Wang, Xie, Chen, Shi]

[Toutanova, Haghigi, Manning][Punyakanok, Roth, Yih]

FigurefromStrubelletal.(2018)

Seman(cRoleLabeling

‣ Canwecombinethetwoapproaches—incorporatesyntac(cinforma(onintoneuralnetworks?

‣Mul(-tasklearningwithrelatedtasks,e.g.,part-of-speechtagging,dependencyparsing…

‣ Syntac(cally-informedself-afen(on:usetheTransformertoencorethesentence;inonehead,tokenafendstoitslikelysyntac(cparents;innextlayer,tokensobserveallotherparents.

committee awards Strickland advanced opticswhoNobelp+1

Recall:Transformer(mul(-headself-afen(on)

Multi-head self-attention + feed forward


Layer 1

Layer p

Multi-head self-attention + feed forwardLayer J

committee awards Strickland advanced opticswhoNobel

Recall:Transformer(mul(-headself-afen(on)FigurefromStrubelletal.(2018)

Layer p

QKV

MH

M1

Feed Forward

Feed Forward

Feed Forward

Feed Forward

Feed Forward

Feed Forward

Feed Forward

biaffine parser

biaffine parser

biaffine parser

biaffine parser

biaffine parser

biaffine parser

biaffine parser

[Dozat and Manning 2017]committee awards Strickland advanced opticswhoNobel

optics advanced

who Strickland

awards committee

NobelA

Syntac(cally-InformedSelf-Afen(on


Syntac(cally-InformedSelf-Afen(on

Syntactically-informed self-attention

Multi-head self-attention + feed forwardLayer 1

Layer p



Linguis(cally-InformedSelf-Afen(onFigurefromStrubelletal.(2018)

Layer p Syntactically-informed self-attention


Linguis(cally-InformedSelf-Afen(on

Layer 1 Multi-head self-attention + feed forward

Multi-head self-attention + feed forwardLayer r

NNP NN VBZ/PRED NNP WP VBN/PRED NN


FigurefromStrubelletal.(2018)

Layer 1

Syntactically-informed self-attentionLayer p



Layer r Multi-head self-attention + feed forward

nn nsubj root dobj nsubj rcmod dobj


NNP NN VBZ/PRED NNP WP NNVBN/PRED


Layer 1 Multi-head self-attention + feed forward




Layer p






argspredicates

Bilinear



Layer p






argspredicates

Bilinear



Layer p




class labels



B-ARG0

argspredicates

Bilinear



Layer p






B-ARG0

argspredicates

Bilinear



Layer p






B-ARG0

argspredicates

Bilinear



Layer p




I-ARG0 B-V B-ARG1 O O O



B-ARG0

argspredicates

Bilinear



Layer p




I-ARG0 B-V B-ARG1 O O OLinguis(cally-InformedSelf-Afen(on


B-ARG0

argspredicates

Bilinear



Layer p






B-ARG0

argspredicates

Bilinear



Layer p






Layer p






B-ARG0

argspredicates

Bilinear

I-ARG0 B-V B-ARG1 O O OO O O B-ARG0 B-R-ARG0 B-V B-ARG1

pospreds

labeledparse

SRL


GloVe ELMo

in-domain (dev) in-domain (dev)

He et al. 2017 81.5 ---

He et al. 2018 81.6 85.3

SA 82.39 85.26

LISA 82.24 85.35

+D&M 83.58 85.17

+Gold 86.81 87.63

Strubelletal.(2018)


WhySRLisdifficult?orNLPingeneral

‣ Syntac(cAlterna(onThe robot plays piano.

The music plays softly.

ARG0player

ARG1thing performed

ARG2instrument

ARGM-MNR

The cafe is playing my favorite song.

ARG1thing performed

ARG0player

subjects

objects

6

Syntactic Alternation PP Attachment Long-range Dependencies

‣ Preposi(onalPhrase(PP)Afachment

I eat [pasta] [with delight].ARG0eater

ARG1meal

ARGM-MNRmanner

I eat [pasta with broccoli].ARG0eater

ARG1meal

7

Syntactic Alternation Prepositional Phrase (PP) Attachment Long-range

Dependencies


8

We flew to Chicago.ARG1

passengerARGM-GOL

goal

We remember the nice view flying to Chicago.ARG1

passengerARGM-GOL

goal

We remember John and Mary flying to Chicago.ARG1

passengerARGM-GOL

goal

Syntactic Alternation PP Attachment Long-range Dependencies‣ LongDependencies


‣ Evenharderforout-of-domaindataSRL is even harder for out-domain data …

“Dip chicken breasts into eggs to coat”

Active, Ser133-phosphorylated CREB effects transcription of CRE-dependent genes via interaction with the 265-kDa …

9


AbstractMeaningRepresenta(on


‣ Graph-structuredannota(on

‣ Nodesarevariableslabeledbyconcepts;edgesareseman(crela(onsAMR – Abstract Meaning Representation

“Bob ate four cakes that he bought.”(x2 / eat-01

:ARG0 (x1 / person

:name (n / name

:op1 "Bob")

:wiki "Bob_X")

:ARG1 (x4 / cake

:quant 4

:ARG1-of (x7 / buy-01

:ARG0 x1)))

e / eat-01

x4 / cakex1 / person

x7 / buy-01

"Bob_X"

name

4

FigurefromGruzi(setal.(2014)


Banarescuetal.(2014)

‣ SupersetofSRL:fullsentenceanalyses,containscoreferenceandmul(-wordexpressionsaswell

‣ F1scoresinthe60s:hard!

‣ Socomprehensivethatit’shardtopredict,buts(lldoesn’thandletenseorsomeotherthings…

Summariza(onwithAMR

Liuetal.(2015)

‣MergeAMRsacrossmul(plesentences

‣ Summariza(on=subgraphextrac(on

SlotFilling

SlotFilling‣Mostconserva(ve,narrowformofIE

IndianExpress—Amassiveearthquakeofmagnitude7.3struckIraqonSunday,103kms(64miles)southeastofthecityofAs-Sulaymaniyah,theUSGeologicalSurveysaid,reportsReuters.USGeologicalSurveyiniKallysaidthequakewasofamagnitude7.2,beforerevisingitto7.3.

magnitude time

epicenter

Speaker:AlanClark

“GenderRolesintheHolyRomanEmpire”

AllagherCenterMainAuditorium

Thistalkwilldiscuss…

speakertitle

location

FreitagandMcCallum(2000)

‣ Oldwork:HMMs,laterCRFstrainedperrole

SlotFilling:MUC

HaghighiandKlein(2010)

‣ Keyaspect:needtocombineinforma(onacrossmul(plemen(onsofanen(tyusingcoreference

SlotFilling:Forums

‣ Extractcode-relatedconceptsinsoowareengineeringforums,butnoteverythingthatlookslikeaconceptisaconcept(e.g.,windows)

avariablenameinthiscontext

JeniyaTabassum,MounicaMaddela,WeiXu,AlanRi8er.“CodeandNamedEn?tyRecogni?oninStackOverflow”inACL(2020)

Rela(onExtrac(on

Rela(onExtrac(on‣ Extracten(ty-rela(on-en(tytriplesfromafixedinventory

ACE(2003-2005)

DuringthewarinIraq,AmericanjournalistsweresomeKmescaughtinthelineoffire

Located_In

‣ Systemscanbefeature-basedorneural,lookatsurfacewords,syntac(cfeatures(dependencypaths),seman(croles

Na(onality

‣ Problem:limiteddataforscalingtobigontologies

‣ UseNER-likesystemtoiden(fyen(tyspans,classifyrela(onsbetweenen(typairswithaclassifier

HearstPaferns

Hearst(1992)

Xsuchas[list] ciKessuchasBerlin,Paris,andLondon.

‣ Syntac(cpafernsespeciallyforfindinghypernym-hyponympairs(“isa”rela(ons)

Berlinisacity

otherciKesincludingBerlin

YisaX

otherXincludingY

‣ Totallyunsupervisedwayofharves(ngworldknowledgefortaskslikeparsingandcoreference(BansalandKlein,2011-2012)

DistantSupervision

Mintzetal.(2009)

[StevenSpielberg]’sfilm[SavingPrivateRyan]islooselybasedonthebrothers’story

‣ Iftwoen((esinarela(onappearinthesamesentence,assumethesentenceexpressestherela(on

‣ Lotsofrela(onsinourknowledgebasealready(e.g.,23,000film-directorrela(ons);usethesetobootstrapmoretrainingdata

Allisonco-producedtheAcademyAward-winning[SavingPrivateRyan],directedby[StevenSpielberg]

Director

Director

DistantSupervision

Mintzetal.(2009)

‣ Learndecentlyaccurateclassifiersfor~100Freebaserela(ons‣ Couldbeusedtocrawlthewebandexpandourknowledgebase

DistantSupervision

FigurefromJiangetal.(2016)

‣ Inherentlyhavenoiseintrainingdata,needspecialmethods(e.g.,mul(-instancelearning)tohandlefalseposi(vesANDfalsenega(ves.

falseposi(ve

???

Mul(-instanceLearning

Xuetal.(2013),Pershinaetal.(2014),Tabassum(2016)

Nega%ve'Bags' Posi%ve'Bags''

A'bag'is'labeled'posi%ve,'if''there'is'at#least#one'posi%ve'example'

A'bag'is'labeled'nega%ve,'if''all'the'examples'in'it'are'nega%ve'

‣ Insteadoflabelsoneachindividualinstance,thelearneronlyobserveslabelsonbagsofinstances.

Mul(-instanceLearning

Xuetal.(2013),Pershinaetal.(2014),Tabassum(2016)

OpenIE

OpenInforma(onExtrac(on

‣ Typicallynofixedrela(oninventory

‣ “Open”ness—wanttobeabletoextractallkindsofinforma(onfromopen-domaintext

‣ Acquirecommonsenseknowledgejustfrom“reading”aboutit,butneedtoprocesslotsoftext(“machinereading”)

TextRunner‣ Extractposi(veexamplesof(e,r,e)triplesviaparsingandheuris(cs

BarackObama,44thpresidentoftheUnitedStates,wasbornonAugust4,1961inHonolulu

=>Barack_Obama,wasbornin,Honolulu

‣ TrainaNaiveBayesclassifiertofiltertriplesfromrawtext:usesfeaturesonPOStags,lexicalfeatures,stopwords,etc.

‣ 80xfasterthanrunningaparser(whichwasslowin2007…)

Bankoetal.(2007)

‣ Usemul(pleinstancesofextrac(onstoassignprobabilitytoarela(on

Exploi(ngRedundancy

Bankoetal.(2007)

‣ Concrete:definitelytrueAbstract:possiblytruebutunderspecified

‣ Hardtoevaluate:canassessprecision ofextractedfacts,buthowdoweknowrecall?

‣ 9Mwebpages/133Msentences

‣ 2.2tuplesextractedpersentence,filterbasedonprobabili(es

ReVerb

Faderetal.(2011)

‣Moreconstraints:openrela(onshavetobeginwithverb,endwithpreposi(on,becon(guous(e.g.,wasbornon)

‣ Extractmoremeaningfulrela(ons,par(cularlywithlightverbs

ReVerb

Faderetal.(2011)

‣ Foreachverb,iden(fythelongestsequenceofwordsfollowingtheverbthatsa(sfyaPOSregex(V.*P)andwhichsa(sfyheuris(clexicalconstraintsonspecificity

‣ Findthenearestargumentsoneithersideoftherela(on

‣ Annotatorslabeledrela(onsin500documentstoassessrecall

Takeaways

‣ Rela(onextrac(on:cancollectdatawithdistantsupervision,usethistoexpandknowledgebases

‣ Slotfilling:(edtoaspecificontology,butgivesfine-grainedinforma(on

‣ OpenIE:extractslotsofthings,buthardtoknowhowgoodorusefultheyare

‣ Cancombinewithstandardques(onanswering

‣ Addnewfactstoknowledgebases

‣ SRL/AMR:handleabunchofphenomena,butmoreorlesslikesyntax++intermsofwhattheyrepresent

‣Many,manyapplica(onsandtechniques

Documents

Informa(on Extrac(on · My mug broke into pieces immediately. The robot broke my favorite mug with a wrench. Semantic Role Labeling (SRL) 3 subj v obj prep subj v prep adv thing broken