Informa(on Retrieval - CLSP · quickly ﬁnd documents that contain both terms. – (This operaon is someSmes referred to as merging posSngs lists: this slightly counterintuiSve (contrary

Informa(onRetrieval

Dr.QaiserAbbasDepartmentofComputerScience&IT,

UniversityofSargodha,Sargodha,40100,[email protected]

Saturday,27February16 1

ProcessingBooleanqueries•  HowdoweprocessaqueryusinganinvertedindexandthebasicBoolean

retrievalmodel?Considerprocessingthesimpleconjunc-vequery:BrutusANDCalpurniaovertheinvertedindexparSallyshowninFigure1.3.We:–  1.LocateBrutusintheDicSonary–  2.RetrieveitsposSngs–  3.LocateCalpurniaintheDicSonary–  4.RetrieveitsposSngs–  5.IntersectthetwoposSngslists,asshowninFigure1.5.


ProcessingBooleanqueries

•  Theintersec-onoperaSonisthecrucialone:weneedtoefficientlyintersectposSngslistssoastobeabletoquicklyfinddocumentsthatcontainbothterms.–  (ThisoperaSonissomeSmesreferredtoasmergingposSngslists:thisslightlycounterintuiSve(contrarytointuiSonortocommon-senseexpectaSon.)namereflectsusingthetermmergealgorithmforageneralfamilyofalgorithm;herewearemergingthelistswithalogicalANDoperaSon.)

•  ThereisasimpleandeffecSvemethodofintersecSngposSngslistsusingthemergealgorithm(seeFigure1.6).






•  wemaintainpointersintobothlistsandwalkthroughthetwoposSngslistssimultaneously,inSmelinearinthetotalnumberofposSngsentries.

•  Ateachstep,wecomparethedocIDpointedtobybothpointers.Iftheyarethesame,weputthatdocIDintheresultslist,andadvancebothpointers.OtherwiseweadvancethepointerpoinSngtothesmallerdocID.

•  IfthelengthsoftheposSngslistsarexandy,theintersecSontakesO(x+y)operaSons.

•  Tousethisalgorithm,itiscrucialthatposSngsbesortedbyasingleglobalordering.UsinganumericsortbydocIDisonesimplewaytoachievethis.



•  Exercise1.4(page12)Forthequeriesbelow,canwesSllrunthroughtheintersecSoninSmeO(x+y),wherexandyarethelengthsoftheposSngslistsforBrutusandCaeser?Ifnot,whatcanweachieve?a.BrutusANDNOTCaeserb.BrutusORNOTCaeserSolu(ona.Page10ofthebookdefinesthecomplexityofqueryingO(N)asO(x+y)wherexandyarelengthsoftheposSngsliststobeintersected.ForthegivencondiSonBrutusANDNOTCaeser,considerthefollowingposSngslistCase1-whentheposSngslistforBrutushaslessernumberofposSngsthanthatforCaeser:–  Brutus1 3 10 21–  Caesar1 6 9 23 45 57WehavetofindthesetofdocumentsthathaveBrutusanddonothaveCaeser.Weusethefollowinglogic



PosiSonpointerp1tothefirstposSngintheposSngslistforBrutusandpointerp2tothefirstposSnginposSngslistfortermCaesar.ComparetheDocIDspointedbyeachpointer(CompareDocID(p1)andDocID(p2)

1.  IfDocID(p1)=DocID(p2),thenitmeansthatthedocIDinthatposSngcontainsboththetermsBrutusandCaeser.Wedonotwantthis.SomovetothenextposSnginboththelists.Gotopoint2(compareoperaSon).

2.  IfDocID(p1)<DocID(p2),thenitmeansthatDocID(p1)hasthetermBrutusANDNOTCaeser.Thisiswhatwewant,sostoretheDocID(p1)inananswerarrayMovethepointerfortermBrutustothenextposSnginthelist.Gotopoint2(CompareoperaSon)

3.  IfDocID(p1)>DocID(p2),WemovethepointerforCaesertothenextposSng.Gotopoint2(compareoperaSon).

WerunthecompareandincrementloopSllthep1pointstoNULLandthenwestoptheoperaSon.

WedonotneedtoruntheoperaSonSllp2pointstoNULLaswerequiretheDocIDsthathaveBrutusinit.e.g.wedonothavetoconsiderposSngs45and57inthelistforCaeser.

Answer=3,10,21



Thus,theComplexityofQueryingisO(x+y1),wherexisthelengthoftheposSngslistforthetermthathastobeintheexpressionandy1isthelengthoftheposSngslisttraversed,forthetermtobeexcluded,whenxreachesnull.Inthiscase,O(x+y1)<=O(x+y),wherey1<=yCase2-WhenposSngslistforBrutusisgreaterthanthatforCaeser.Brutus1 5 11 21 45 55Caeser1 11 170Eveninthiscase,theenSrelengthofposSngslistforBrutushastobetraversed(x),onlythatlengthofposSngslistforCaeserhastobetraversed(y1),Sllp1reachesnull,ThustheComplexityofQueryingisO(x+y1),wherey1<=y



Solu(onb.ForBrutusORNOTCaeser,weneedtofinddocumentshavingthetermBrutus,cannothaveCaeserOrnothavingthetermCaeser,canhaveBrutusorcannothaveBrutus.Brutus1 5 11 21 45 55Caeser1 11 170Other2 10 11 33 34HeretheenSrelength(x)ofposSngslistforBrutushastobetraversedtofindDocIDsthatcontainBrutus,TheenSrelength(z)ofposSngslistforOtherhastobetraversed.Similarly,Thelength(y1)ofposSngslistforCaeserhastobetraversedSllxandzbothreachNULL.ThustheComplexityofQueryingisO(x+y1+z),wherey1<=y



•  WecanextendtheintersecSonoperaSontoprocessmorecomplicatedquerieslike:–  (BrutusORCaesar)ANDNOTCalpurnia

•  Queryop)miza)onistheprocessofselecSnghowtoorganizetheworkofansweringaquerysothattheleasttotalamountofworkneedstobedonebythesystem.– AmajorelementofthisforBooleanqueriesistheorderinwhichposSngslistsareaccessed.Whatisthebestorderforqueryprocessing?



•  ConsideraquerythatisanANDoftterms,forinstance:–  BrutusANDCaesarANDCalpurnia

•  Foreachofthetterms,weneedtogetitsposSngs,thenANDthemtogether.ThestandardheurisScistoprocesstermsinorderofincreasingdocumentfrequency:–  ifwestartbyintersecSngthetwosmallestposSngslists,

thenallintermediateresultsmustbenobiggerthanthesmallestposSngslist,andwearethereforelikelytodotheleastamountoftotalwork.So,fortheposSngslistsinFigure1.3,weexecutetheabovequeryas:

–  (CalpurniaANDBrutus)ANDCaesar•  ThisisafirstjusSficaSonforkeepingthefrequencyof

termsinthedicSonary:itallowsustomakethisordering-decisionbasedonin-memorydatabeforeaccessinganyposSngslist.



•  ConsidernowtheopSmizaSonofmoregeneralqueries,suchas:–  (maddingORcrowd)AND(ignobleORstrife)AND(killedORslain)

•  Asbefore,wewillgetthefrequenciesforallterms,andwecanthen(conservaSvely)esSmatethesizeofeachORbythesumofthefrequenciesofitsdisjuncts.WecanthenprocessthequeryinincreasingorderofthesizeofeachdisjuncSveterm.



•  Exercise1.7[⋆]– Recommendaqueryprocessingorderfor(tangerineORtrees)AND(marmaladeORskies)AND(kaleidoscopeOReyes)

– giventhefollowingposSngslistsizes:



•  Exercise1.7[⋆]–  (kaleidoscopeOReyes)(300,321)AND(tangerineORtrees)(363,465)AND(marmaladeORskies)(379,571)

– However,dependingontheactualdistribuSonofposSngs,(tangerineORtrees)maywellbelongerthan(marmaladeORskies),becausethetwocomponentsoftheformeraremoreasymmetric.

AssignmentNo.2


AssignmentNo.2


TheextendedBooleanmodelversusrankedretrieval

•  TheBooleanretrievalmodelcontrastswithrankedretrievalmodelssuchasthevectorspacemodel(SecSon6.3),inwhichuserslargelyusefreetextqueries,thatis,justtypingoneormorewordsratherthanusingapreciselanguagewithoperatorsforbuildingupqueryexpressions,andthesystemdecideswhichdocumentsbestsaSsfythequery.

•  AstrictBooleanexpressionovertermswithanunorderedresultssetistoolimitedformanyoftheinformaSonneedsthatpeoplehave,andthesesystemsimplementedextendedBooleanretrievalmodelsbyincorporaSngaddiSonaloperatorssuchastermproximityoperators.

•  Aproximityoperatorisawayofspecifyingthattwotermsinaquerymustoccurclosetoeachotherinadocument,whereclosenessmaybemeasuredbylimiSngtheallowednumberofinterveningwordsorbyreferencetoastructuralunitsuchasasentenceorparagraph.



•  Example1.1:CommercialBooleansearching:Westlaw.Westlaw(hop://www.westlaw.com/)isthelargestcommerciallegalsearchservice(intermsofthenumberofpayingsub-scribers),withoverhalfamillionsubscribersperformingmillionsofsearchesadayovertensofterabytesoftextdata.Theservicewasstartedin1975.In2005,Booleansearch(called“TermsandConnectors”byWestlaw)wassSllthedefault,andusedbyalargepercentageofusers,althoughrankedfreetextquerying(called“NaturalLanguage”byWestlaw)wasaddedin1992.HerearesomeexampleBooleanqueriesonWestlaw:–  Informa-onneed:InformaSononthelegaltheoriesinvolvedinprevenSngthe

disclosureoftradesecretsbyemployeesformerlyemployedbyacompeSngcompany.Query:"tradesecret"/sdisclos!/sprevent/semploye!

–  Informa-onneed:Requirementsfordisabledpeopletobeabletoaccessawork-place.Query:disab!/paccess!/swork-sitework-place(employment/3place)

–  Informa-onneed:Casesaboutahost’sresponsibilityfordrunkguests.Query:host!/p(responsib!liab!)/p(intoxicat!drunk!)/pguest



•  Notethelong,precisequeriesandtheuseofproximityoperators,bothuncommoninwebsearch.Submioedqueriesaverageabouttenwordsinlength.UnlikewebsearchconvenSons,aspacebetweenwordsrepresentsdisjuncSon(theSghtestbindingoperator),&isANDand/s,/p,and/kaskformatchesinthesamesentence,sameparagraphorwithinkwordsrespecSvely.Doublequotesgiveaphrasesearch(consecuSvewords);seeSecSon2.4(page39).TheexclamaSonmark(!)givesatrailingwildcardquery(seeSecSon3.2,page51);thusliab!matchesallwordsstarSngwithliab.AddiSonallywork-sitematchesanyofworksite,work-siteorworksite;seeSecSon2.2.1(page22).TypicalexpertqueriesareusuallycarefullydefinedandincrementallydevelopedunSltheyobtainwhatlooktobegoodresultstotheuser.



•  HerewejustmenSonafewofthemainaddiSonalthingswewouldliketobeabletodo:–  WewouldliketobeoerdeterminethesetoftermsinthedicSonaryandto

provideretrievalthatistoleranttospellingmistakesandinconsistentchoiceofwords.

–  Itisovenusefultosearchforcompoundsorphrasesthatdenoteaconceptsuchas“operaSngsystem”.AstheWestlawexamplesshow,wemightalsowishtodoproximityqueriessuchasGatesNEARMicrosoL.Toanswersuchqueries,theindexhastobeaugmentedtocapturetheproximiSesoftermsindocuments.

–  ABooleanmodelonlyrecordstermpresenceorabsence,butovenwewouldliketoaccumulatemorefrequentevidence.TobeabletodothisweneedtermfrequencyinformaSoninposSngslists.

–  Booleanqueriesjustretrieveasetofmatchingdocuments,butcommonlywewishtohaveaneffecSvemethodtoorder(or“rank”)thereturnedresults.Thisrequireshavingamechanismfordeterminingadocumentscorewhichencapsulateshowgoodamatchadocumentisforaquery.



•  Exercise1.12[⋆]WriteaqueryusingWestlawsyntaxwhichwouldfindanyofthewordsprofessor,teacher,orlecturerinthesamesentenceasaformoftheverbexplain.


Documents

Informa(on Retrieval - CLSP · quickly ﬁnd documents that contain both terms. – (This operaon is someSmes referred to as merging posSngs lists: this slightly counterintuiSve (contrary