Online Videos - cs224d.stanford.educs224d.stanford.edu/lectures/CS224d-Lecture 4.pdf• “TV” and “telly” ... • The gradient that arrives at and updates the word vectors can

OnlineVideos

• FERPA

• Signwaiverorsitonthesidesorintheback

• Offcameraquestiontimebeforeandafterlecture

• Questions?

4/5/16RichardSocherLecture1,Slide 1

CS224dDeepNLP

Lecture4:WordWindowClassification

andNeuralNetworks

RichardSocher

Feedbacksofar

• ~70%goodwithcurrentspeed,• 15%toofastà Pleasevisitofficehours• 15%tooslow

• Math,whenglossedoverà notrequired,foodforthoughtforadvancedstudents

• Lecturesdryà understandingbasicsimportant,• Startingnextweekwewillbecomemoreconceptual,introduce

complexmodelsandgainpracticalintuitions


Feedbacksofar

• Giventhefeedback:clearlydefinewordvectorupdatestoday,movedeadlineofPSet1by2days

• Projectideas:• 2types• moreinfonextweek• myofficehour

• Detail:Intuitionforwordvectorcontextwindow• Relativecontextdifferencesmalleràmoresimilarvectors


OverviewToday:

• Generalclassificationbackground

• Updatingwordvectorsforclassification

• Windowclassification&crossentropyerrorderivationtips

• Asinglelayerneuralnetwork!

• (Max-Marginlossandbackprop)


Refresher:Classificationsetupandnotation

• Generallywehaveatrainingdatasetconsistingofsamples

{xi,yi}Ni=1

• xi - inputs,e.g.words(indicesorvectors!),contextwindows,sentences,documents,etc.

• yi - labelswetrytopredict,• e.g.otherwords• class:sentiment,namedentities,buy/selldecision,• later:multi-wordsequences


Classificationintuition

• Trainingdata:{xi,yi}Ni=1

• Simpleillustrationcase:• Fixed2dwordvectorstoclassify• Usinglogisticregression• à lineardecisionboundaryà

• GeneralML:assumexisfixedandonlytrainlogisticregressionweightsWandonlymodifythedecisionboundary


VisualizationswithConvNetJS byKarpathy!http://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html

Classificationnotation

• Crossentropylossfunctionoverdataset{xi,yi}Ni=1

• Whereforeachdatapair(xi,yi):

• Wecanwritef inmatrixnotation andindexelementsofitbasedonclass:


Classification:Regularization!

• Reallyfulllossfunctionoveranydatasetincludesregularizationoverallparametersµ:

• Regularizationwillpreventoverfittingwhenwehavealotoffeatures(orlateraverypowerful/deepmodel)• x-axis:morepowerfulmodelormoretrainingiterations

• Blue:trainingerror,red:testerror


Details:GeneralMLoptimization

• Forgeneralmachinelearningµ usuallyonlyconsistsofcolumnsofW:

• Soweonlyupdatethedecisionboundary


VisualizationswithConvNetJS byKarpathy

Classificationdifferencewithwordvectors

• Commonindeeplearning:• LearnbothWandwordvectorsx


Verylarge!

OverfittingDanger!

Losinggeneralizationbyre-trainingwordvectors

• Setting:Traininglogisticregressionformoviereviewsentimentandinthetrainingdatawehavethewords• “TV”and“telly”

• Inthetestingdatawehave• “television”

• Originallytheywereallsimilar(frompre-trainingwordvectors)

• Whathappenswhenwetrainthewordvectors?


TVtelly

television


• Whathappenswhenwetrainthewordvectors?• Thosethatareinthetrainingdatamovearound• Wordsfrompre-trainingthatdoNOTappearintrainingstay

• Example:• Intrainingdata:“TV”and“telly”• Intestingdataonly:“television”


TVtelly

television:(


• Takehomemessage:

Ifyouonlyhaveasmalltrainingdataset,don’ttrainthewordvectors.

Ifyouhavehaveaverylargedataset,itmayworkbettertotrainwordvectorstothetask.


TVtelly

television

Sidenoteonwordvectorsnotation

• ThewordvectormatrixLisalsocalledlookuptable• Wordvectors=wordembeddings =wordrepresentations(mostly)• Mostlyfrommethodslikeword2vecorGlove

|V|

L =d ……

aardvarka…meta…zebra• Thesearethewordfeaturesxword fromnowon

• Conceptuallyyougetaword’svectorbyleftmultiplyingaone-hotvectore byL:x =Le2 d£ V¢ V£ 1

[]

15

Windowclassification

• Classifyingsinglewordsisrarelydone.

• Interestingproblemslikeambiguityariseincontext!

• Example:auto-antonyms:• "Tosanction"canmean"topermit"or"topunish.”• "Toseed"canmean"toplaceseeds"or"toremoveseeds."

• Example:ambiguousnamedentities:• Parisà Paris,Francevs ParisHilton• Hathawayà BerkshireHathawayvs AnneHathaway



• Idea:classifyawordinitscontextwindowofneighboringwords.

• Forexamplenamedentityrecognitioninto4classes:• Person,location,organization,none

• Manypossibilitiesexistforclassifyingonewordincontext,e.g.averagingallthewordsinawindowbutthatloosespositioninformation



• Trainsoftmax classifierbyassigningalabeltoacenterwordandconcatenatingallwordvectorssurroundingit

• Example:ClassifyParisinthecontextofthissentencewithwindowlength2:

…museumsinParisareamazing….

Xwindow =[xmuseums xin xParis xare xamazing ]T

• Resultingvectorxwindow =x2 R5d,acolumnvector!


Simplestwindowclassifier:Softmax

• Withx=xwindow wecanusethesamesoftmax classifierasbefore

• Withcrossentropyerrorasbefore:

• Buthowdoyouupdatethewordvectors?


same

predictedmodeloutputprobability

Updatingconcatenatedwordvectors

• Shortanswer:Justtakederivativesasbefore

• Longanswer:Let’sgooverthestepstogether(you’llhavetofillinthedetailsinPSet 1!)

• Define:• :softmax probabilityoutputvector(seepreviousslide)• :targetprobabilitydistribution(all0’sexceptatgroundtruthindexofclassy,whereit’s1)

• andfc =c’th elementofthefvector

• Hard,thefirsttime,hencesometipsnow:)


• Tip1:Carefullydefineyourvariablesandkeeptrackoftheirdimensionality!

• Tip2:Knowthychainruleanddon’tforgetwhichvariablesdependonwhat:

• Tip3:Forthesoftmax partofthederivative:Firsttakethederivativewrt fc whenc=y(thecorrectclass),thentakederivativewrt fc whenc≠ y(alltheincorrectclasses)



• Tip4:Whenyoutakederivativewrtoneelementoff,trytoseeifyoucancreateagradientintheendthatincludesallpartialderivatives:

• Tip5:Tolaternotgoinsane&implementation!à resultsintermsofvectoroperationsanddefinesingleindex-ablevectors:



• Tip6:Whenyoustartwiththechainrule,firstuseexplicitsumsandlookatpartialderivativesofe.g.xi orWij

• Tip7:Tocleanitupforevenmorecomplexfunctionslater:Knowdimensionalityofvariables&simplifyintomatrixnotation

• Tip8:Writethisoutinfullsumsifit’snotclear!



• Whatisthedimensionalityofthewindowvectorgradient?

• x istheentirewindow,5d-dimensionalwordvectors,sothederivativewrt toxhastohavethesamedimensionality:



• Thegradientthatarrivesatandupdatesthewordvectorscansimplybesplitupforeachwordvector:

• Let• Withxwindow =[xmuseums xin xParis xare xamazing ]

• Wehave



• Thiswillpushwordvectorsintoareassuchtheywillbehelpfulindeterminingnamedentities.

• Forexample,themodelcanlearnthatseeingxin asthewordjustbeforethecenterwordisindicativeforthecenterwordtobealocation



• ThegradientofJwrt thesoftmax weightsW!

• Similarsteps,writedownpartialwrt Wij first!• Thenwehavefull

What’smissingfortrainingthewindowmodel?


Anoteonmatriximplementations

4/5/16RichardSocher28

• Therearetwoexpensiveoperationsinthesoftmax:

• Thematrixmultiplication andtheexp

• Aforloopisneverasefficientwhenyouimplementitcomparedvs whenyouusealargermatrixmultiplication!

• Examplecodeà



• Loopingoverwordvectorsinsteadofconcatenatingthemallintoonelargematrixandthenmultiplyingthesoftmax weightswiththatmatrix

• 1000loops,bestof3:639µsperloop10000loops,bestof3:53.8µsperloop



• ResultoffastermethodisaCxNmatrix:

• Eachcolumnisanf(x)inournotation(unnormalized classscores)

• Matricesareawesome!

• Youshouldspeedtestyourcodealottoo

Softmax (=logisticregression)isnotverypowerful


• Softmax onlygiveslineardecisionboundariesintheoriginalspace.

• Withlittledatathatcanbeagoodregularizer

• Withmoredataitisverylimiting!

Softmax (=logisticregression)isnotverypowerful


• Softmax onlylineardecisionboundaries

• à Lamewhenproblemiscomplex

• Wouldn’titbecooltogetthesecorrect?

NeuralNetsfortheWin!


• Neuralnetworkscanlearnmuchmorecomplexfunctionsandnonlineardecisionboundaries!

Fromlogisticregressiontoneuralnets

34

Demystifyingneuralnetworks

Neuralnetworkscomewiththeirownterminologicalbaggage

…justlikeSVMs

Butifyouunderstandhowsoftmax modelswork

Thenyoualreadyunderstand theoperationofabasicneuralnetworkneuron!

AsingleneuronAcomputationalunitwithn(3) inputs

and1outputandparametersW,b

Activationfunction

Inputs

Biasunitcorresponds tointerceptterm

Output

35

Aneuronisessentiallyabinarylogisticregressionunit

hw,b(x) = f (wTx + b)

f (z) = 11+ e−z

w,b aretheparametersofthisneuroni.e.,thislogisticregressionmodel

36

b:Wecanhavean“alwayson”feature,whichgivesaclassprior,orseparateitout,asabiasterm

Aneuralnetwork=runningseverallogisticregressionsatthesametimeIfwefeedavectorofinputsthroughabunchoflogisticregressionfunctions,thenwegetavectorofoutputs…

Butwedon’thavetodecideaheadoftimewhatvariablestheselogisticregressionsaretryingtopredict!

37

Aneuralnetwork=runningseverallogisticregressionsatthesametime…whichwecanfeedintoanotherlogisticregressionfunction

Itisthelossfunctionthatwilldirectwhattheintermediatehiddenvariablesshouldbe,soastodoagoodjobatpredictingthetargetsforthenextlayer,etc.

38

Aneuralnetwork=runningseverallogisticregressionsatthesametime

Beforeweknowit,wehaveamultilayerneuralnetwork….

39

Matrixnotationforalayer

Wehave

Inmatrixnotation

wheref isappliedelement-wise:

a1

a2

a3

a1 = f (W11x1 +W12x2 +W13x3 + b1)a2 = f (W21x1 +W22x2 +W23x3 + b2 )etc.

z =Wx + ba = f (z)

f ([z1, z2, z3]) = [ f (z1), f (z2 ), f (z3)]40

W12

b3

Non-linearities (f):Whythey’reneeded

• Example:functionapproximation,e.g.,regressionorclassification• Withoutnon-linearities,deepneuralnetworkscan’tdoanythingmorethanalineartransform

• Extralayerscouldjustbecompileddownintoasinglelineartransform:W1W2x =Wx

• Withmorelayers,theycanapproximatemorecomplexfunctions!

41

Amorepowerfulwindowclassifier

• Revisiting

• Xwindow =[xmuseums xin xParis xare xamazing ]


ASingleLayerNeuralNetwork

• Asinglelayerisacombinationofalinearlayerandanonlinearity:

• Theneuralactivationsacanthenbeusedtocomputesomefunction

• Forinstance,asoftmax probabilityoranunnormalized score:

43

Summary:Feed-forwardComputation

44

Computingawindow’sscorewitha3-layerneuralnet:s=score(museumsinParisareamazing)

Xwindow =[xmuseums xin xParis xare xamazing ]

Nextlecture:


Trainingawindow-basedneuralnetwork.

Takingmoredeeperderivativesà Backprop

Thenwehaveallthebasictoolsinplacetolearnaboutmorecomplexmodels:)

Probablyfornextlecture…


Anotheroutputlayerandlossfunctioncombo!

47

• Sofar:softmax andcross-entropyerror(exp slow)

• Wedon’talwaysneedprobabilities,oftenunnormalized scoresareenoughtoclassifycorrectly.

• Also:Max-margin!

• Moreonthatinfuturelectures!

NeuralNetmodeltoclassifygrammaticalphrases


• Idea:Trainaneuralnetworktoproducehighscoresforgrammatical phrasesofspecificlengthandlowscoresforungrammaticalphrases

• s =score(catchillsonamat)

• sc =score(catchillsMenloamat)

Anotheroutputlayerandlossfunctioncombo!

• Ideafortrainingobjective• Makescoreoftruewindowlargerandcorruptwindow’sscorelower(untilthey’regoodenough):minimize

• Thisiscontinuous,canperformSGD49

TrainingwithBackpropagation

AssumingcostJis>0,itissimpletoseethatwecancomputethederivativesofs andsc wrt alltheinvolvedvariables:U,W,b,x

50


• Let’sconsiderthederivativeofasingleweightWij

• Thisonlyappearsinsideai

• Forexample:W23 isonlyusedtocomputea2

x1 x2x3 +1

a1 a2

s U2

W23

51


DerivativeofweightWij:

52

x1 x2x3 +1

a1 a2

s U2

W23

whereforlogisticf


DerivativeofsingleweightWij :

Localerrorsignal

Localinputsignal

53

x1 x2x3 +1

a1 a2

s U2

W23

• Wewantallcombinationsofi =1,2 and j=1,2,3

• Solution:Outerproduct:whereisthe“responsibility”comingfromeachactivationa


• FromsingleweightWij tofullW:

54

x1 x2x3 +1

a1 a2

s U2

W23


• Forbiasesb,weget:

55

x1 x2x3 +1

a1 a2

s U2

W23


56

That’salmostbackpropagationIt’ssimplytakingderivativesandusingthechainrule!

Remainingtrick:wecanre-usederivativescomputedforhigherlayersincomputingderivativesforlowerlayers

Example:lastderivativesofmodel,thewordvectorsinx


• Takederivativeofscorewithrespecttosinglewordvector(forsimplicitya1dvector,butsameifitwaslonger)

• Now,wecannotjusttakeintoconsiderationoneaibecauseeachxj isconnectedtoalltheneuronsaboveandhencexj influencestheoverallscorethroughallofthese,hence:

Re-usedpartofpreviousderivative57

Summary


Documents

Online Videos - cs224d.stanford.educs224d.stanford.edu/lectures/CS224d-Lecture 4.pdf• “TV” and “telly” ... • The gradient that arrives at and updates the word vectors can