5. Randomized Algorithms for Optimizing Large Join Queries - Ioannidis

7/30/2019 5. Randomized Algorithms for Optimizing Large Join Queries - Ioannidis

1/10

RANDOMIZED ALGORITHMS FOR OPTIMIZING LARGE JOIN QUERIES +Yannis E. IoannidisYounkyung Cha KangComputer ScrencesDepartmentUnrvemty of WtsconsuaMadtson, WI 53706

AbstractQuery optmuzatton for relatmnal database ystems S a combma-tonal optumzahon problem, whtch makesexhaustrvesearchunac-ceptableas the query size grows Randomrzedalgortthms. such asSimulated Annealmg (SA) and Iteratrve Improvement (II), areviable a ltemattves to exhaushve search We have adapted hesealgonthms to the optrmizatton of proJect-se lect-Jam ueries Wehave tested them on large quertes of various types wrth drfferentdatabases, oncludmg that m most casesSA tdentrfies a lower cos taccessplan than II To explain thts result, we have studied theshapeof the cost funchon over the solutmn spaceassociatedwithsuch queues and we have conJectured hat tt resembles a cupwith relatively small vsrtatrons at the bottom Thus has msptred anew Two PhaseOptrmtzatton algonthm, which 1s a combmattonof Sunulated Annealing and Iteratrve Improvement Expenmentalresults show that Two PhaseGpttmtzatron outperforms the ongt-nal algonthms m te rms of both output quality and runnmg tune

1. INTRODUCTIONQuery opttmrzatron 1s an expensive process, pnmartlybecause he number of alternative accessplans for a query growsat least exponentrally with the number of relations partrcipatmg nthe query The apphcatton of several useful heunsttcs ehmmatessome altematrves hat are hkely to be suboptlmal [Sell791 but rtdoes not change the combmatorralnature of the problem Futuredatabasesystems will need to optunrze queries of much lughercomplextty than current ones Thts mcreasen complexity may becaused by an increase m the number of relattons m a query

[Kns86], by an mcreasem the number of queries that are opttm-rzed collecttvely (global optrmrzatron) GranSl. Se1188], r by theemergence f recursive queues The heunstrcally pnmmg, almostexhaustive search algonthms used by current ophmtzers aremadquate for queries of the expectedcomplexny Thus, the needto develop new query opbmtzatton algonthms becomesapparentRandomized algonthms have been successfully applied tovarious combmatortal optmuzatron problems Two such algo-rnhms, Smulated Annealrng [Ktrk83] and Iterative Improvement[Naha86]. are the best known such algornhms and have been pro-posed for query optlmtzahon of large quenes as well Ioanm&sand Wong applied Stmulated Annealmg to the optmnzattonof

Partlally supported by NSF under Grant IRI-8703592

Pemws~on to copy wthout fee all or part of t hts matenal IS granted prowdedthat the copa are not made or dntnbuted for dwect wmmercnal advantage, theACM copy& nota and the utle of the pubbcatnonand ,ts date appear, andnouce 1sglen that cqymg IS by pcn~ssmn of the Asswat!on for ComputmgMachmay To cow othenvw, or to repubb sh, requres a fee and/or spenticpWllllSSlO0 19w ACM 089791 365 S/!WC035/0312 $150

some ecurstve quenes [Ioan87] Swami and Gupta apphed bothSimulated Annealmg and Iterattve Improvement on opttmtzauonof select-project-Jam uenes [Swam881In thrs paper, we address he problem of usmg randomtzedoptmuzatton algorithms for select-project-jamqueries The majorcontnbutrons of the paper are the followmg Fast, we comparethe performance of Srmulated Armeahng and Iterattve Improve-ment by conductmg several expenments wtth a variety of quenesand databases We show that the former almost always producesa better output, whrch 1sdfferent from the results of the study ofSwam1and Gupta [Swam881 Second. we map the shapeof thecost functron over the spaceof altemattve access lans by a senesof expenments on that space We conclude that the shapeof thecost functton resemblesa cup, which leads to an explanatton of

the behavior of Sunulated Annealmg and Iterattve Improvementand the dtfference with the results of Swamt and Gupta Thud,msptredby the above analysrs,we propose a new query opumtza-tton a lgorithm that exhtbtts supertor performancem terms of bothoutput quality and nmnmg umeThs paper 1s orgamzed as follows In Sectton 2, wedescribe the specifics of our adaptatton of Stmulated Annealmgand Iteratrve Improvement to query optrmrzatron and introducethe Two PhaseGpomtzatton algorithm In sectton 3, we presentthe results of an extensive experrmentalperformanceevaluatton ofthe three algonthms In sectron4, we descnbe an analysis of theshape of the cost functton over the space of altemattve accessplans for a query. basedon whrch we explam the behavior of thealgonthms Sectron 5 describes the results of a hnnted set ofexpenmentswrth the three algonthms on an enhancedset of query

processmgaltemattves Finally, Sectton 6 contams some relatedwork and Sectron7 concludes and discusses uture work2. RANDOMIZED ALGORITHMS FOR QUERYOPTIMIZATION

Each solution to a combmatonal opttmtzatron problem canbe thought of as a state m a space,1e , a node m the graph, thatincludes all such sohmons Each state has a cost assoctatedwrthrt, which 1s given by some problem-specrfic cost functton Thegoal of an optrmtzauon algorithm 1s o find a state wtth the glo-bally mmmum cost. Randomtzed algontbms usually performrandom walks in the state spacevia a series of moves The statesthat can be reached m one move from a state S are called then&hors of S A move 1scalled uphJl (downlull) rf the cost ofthe source state IS ower (higher) than the cost of the destmauonstate A state 1s a local muumum d m all paths startmg at thatstate any downhrll move comes after at least one uplull move Astate 1sa global mtmtnumd rt has the lowest cost among all s tatesA state 1son a plateau d rt has no lower cost neighbor and yet rtcan reach ower cos t stateswithout uphtll moves Usmg the abovetermmology we descnbe three randomtzed opttmtzatron algo-nthms We also discusshow we adapted hese algonthms to queryoptimizatton

312


2/10

2 1. Generic AlgorithmsIn the descnpnons below, we make use of a fictmous stateS, whose cost 1s 00 Also, cost(S) 1s the cost of state S, andneighbors(S) 1s he set of neighbors of state S

2.1.1. Iterative Improvement (II)The genenc Iterative Improvement (II) algorithm 1sshownm Figure 2 1 The mner loop of II 1scalled a locaf optunczatronA local ophmlzahon starts at a random state and unproves thesolution by repeatedly acceptmg andom downhill moves until itreaches a local mmlmum II repeats these local opurmzauonsuntil a stoppmng-con&ton 1s met, at which pomt It returns thelocal nummum with the lowest cost found As tie approachesm, the probability that II will vlsd the global mmunum approaches1 [Naha86] However, given a fimte amount of tune, thealgonthms performancedependson the charactenstlcsof the costfuncnon over the state space and the connect~vlty of the latter asdeterminedby the neighbors of each state

procedure I() (mmS = S,,while not (stoppmng~condt~on)o (S = random state,while not (foc~~mmunum(S)) do (S = random state m neighbors(S),if cost(S) c cost(S) hen S = S,1if cost(S)c cost(mmS) hen mmS = S,1retum(mmS),Figure 2.1. Iterabve Improvement

2.1.2. Simulated Annealing (SA)A local ophmlzatlon m II perfo rms only downhdl movesIn conbask Srmulated Annealmg (SA) does accept uphlll moveswith someprobablhty, trymg to avold bemg caught m a high costlocal mlmmum The genenc algonthm+ 1s shown m Figure 2 2

SA was ongmally derived by analogy to the processof annealmgof crystals We use the same termmology for the algorithmparameters as in the ongmal proposal (The termmology wasadopted rom the analogousphysical process The mner loop ofSA 1scalled a stage Each stage1sperformed under a fixed valueof a parameterT. called temperature,wluch controls the probabll-lty of acceptmguphill moves This probabihty 1sequal to e-AcT,where AC 1s he difference between the cost of the new state andthat of the ongmal one Thus, the probablhty of acceptmg anuphill move 1s a monotomcally mcreasmg unction of the tem-perature and a monotomcally decreasmg function of the costdifference Each stage ends when the algonthm 1sconsldered ohave reached an equdtbrrum Then, the temperature 1s reducedaccordmg o some unction and another stagebegms,1e , the tem-perature 1s owered as time passes The algorithm stops when It 1sconsidered to be frozen, 1e , when the temperature 1s equal tozero It has been shown theoretlcally that, under certam In Ftgure. 2 2. we keep track of the mt nmm m cost state found(mmS) In the end, tt ts mmS that ts reported as the answer, whereas apure verston of SA would report the state to whtch the algortthm has con-verged The version tn Figure 2 2 can only tmprove an the results of thepure verston and IS the one that we use tn thts study

con&hons satisfied by someparametersof the algonthm, as tem-perature approacheszero, the algorithm converges to the globalmmunum [Rome851 Agam, given a fimte amount of tune toreduce the temperature, he algonthms performance dependsonthe charactensticsof the cost function over the state spaceand thecomechvUy of the latterprocedure SA() (s = so,T=To,mmS=S,while not wozen) do (while not (equhbruun) do (S = random statem neighbors(S).AC = cost@) - cost(S),if (AC < 0) then S = S,if (AC > 0) then S = S with probability e-AcT,if cost(S) c cost(mmS) hen mmS = S,1T = reduce(T),1retum(mmS).1 Figure 2.2: Sunulated Annealmg

2.13. Two Phase Optimization (2PO)In this subsechon,we mtroduce the Two Phase Optrmrza-tron (2pO) algorithm, which 1s a combmauon of II and SA Asthe name suggests,2FOcan be divided mto two phases In phase1. II 1s un for a small period of hme. I e , a few local ophmlza-tlons are performed The output of that phase, wluch 1s he bestlocal mmunum found, 1s the lmfial state of the next phase Inphase2, SA IS run with a low lmt.ml temperature Intmtlvely, thealgorithm chooses a local mmunum and then searches he areaaround it, still bemg able to move m and out of local mmuna, butpract&ly unable to chmb up very high hills Thus, 2P0 1sappropnate when such an ab&ty 1s not necessary for proper

OptitIIlZah~, wluch is the case for select-pro ject-Jam queryoptmuzauon as we demonstratem the followmg sechons2.2. Problem Specific Parameters

When genenc randomized optmuzatlon algorithms areapplied to a partu~lar problem, there are several parameters hatneed to be specified based on the specific charactensbcsof theproblem For II, SA, and 2P0. they are the statespace, he nelgh-bors function. and the cost function2.2.1. State Space

Each state m query optmuzahon corresponds o an accessplan (strategy) of the query to be optmuzed Hence, m the sequel,we use the terms state and strategy m&stmgmshably Usmg theheunstlcs of performmg selections and projections as early aspos-sible and excludmg unnecessaryCartesianproducts [Seh79], wecan ehmmate certam subophmal strategies to mcresse theefficiency of the optmuzahon Thus. we reduce the goal of thequery optumzer to finding the best JOT order, together with thebest OT method for each OUI In dus case,each strategy can berepresentedas a Join processrng ree, 1e , a tree whose leaves arebase relations. m temal nodes are Jam operators, and edges mdl-cate the flow o f data. If all mtemal no&s of such a tree have atleast one leaf as a chdd, then the tree IS called hear Otherwise,it is called bushy Most Jam methods d&mgmsh the two JOT

313


3/10

operands. one bemg the outer relation and the other bemg theinner relabon An outer linear jorn processrng tree (left-deeptree) IS a hnear Jam processmg ree whose mne~ elations of allJOISIS are base relations In our study, the strategy spacemcludesall possibleJam processing rees,1e , both linear and bushy ones2.2.2. Neighbors Function

The neighbors of a state, which IS a Jam processmg ree(I e , a strategy), are determmedby a set of transformauon rulesEach rule 1s apphed to one or two Internal no&s of the state,replaces them by one or two new nodes, and usually leaves therest of the nodes of the state unchanged With A, B, and C bemgarbitrary JOUI processmg ormulas, the set of transformation rulesthat we used m our study 1sgiven below(1) Jam method chorce A wnurlloqB + A w,,,,~, B(2) Jam comutatrvrty AwB+BwA(3) Jorn ammamrty (AwB)wCt,Aw(BwC)(4) L.t$join exchange (AwB)oaC+(AwC)wB(5) RqhtJoln exchange Aw(BwC)-+Bw(AwC)

Rule (1) changes he Jam method of a Jam operator, e g ,from nested-loops to merge-scan Together with the algebrrucrules (2) and (3), it ts enough for the spaceof all mterestmgstra-tegles to be connected Each of the addItiona exchange rules 1sequivalent to applymg rules (2). (3), and (2) m that order Theiradvantage IS that they avold the use of JOUI commutatn%yApplymg JOIII commutativity does not change the state cost forsome Jam methods, e g , merge-scan.which tends to create pla-teaux m the state space The algonthms do not usually use theprecise defimuon of a local mmunum to recognize one but useapproxlmatlons, and plateaux can be mistaken for local mmuna.Havmg many such false local mlmma m the state spacedegradesthe output quality o f randomlzed algonthms. especially II TheJam exchange rules reduce the number of plateaux by addmgdirect paths hat bypass hem

Most of the time, applymg one of the above rules on someJam nodes of a state does not affect the cost of the remammgnodes of the state Thus, the cost of a nerghbor can be evaluatedfrom the cost of the ongmal state m constant tune, takmg intoaccount the local changesperformed by the transformattons Anunavoidable exception occurs when mterestmgsort order changes[Seh79], and the change must be propagated o the ancestorsofthe transformednodes2.2.3. Cost Function

The cost fun&on that we used m our study only accountsfor the I/O required by each strategy The precise formulas arenot presentedhere due to lack o f space They are based on thefollowmg assumphons (a) no pipelmmg, I e , temporary relahonsare created or the mtermedlate esults, (b) mmunal buffenng forall operations, (c) on-the-fly execution of proJecuons,and (d) noduplicate ehrnmatlon on proJections2.3. Implementatron Specific Parameters

Several parametersof randomized optunlzatlon algonthmsare implementation specific These can be tuned to Improve per-formance and/or output quahty The followmg tables summarizeour choices for the parametersof II, SA, and 2FO We amved at

them after some expenmentatlon with various altematlves, andalso basedon past expenence with the algorithms m query optmn-zahon [Ioan87] and other fields [John871

Table 2.1: Implementabon specific parameters or II

next state random neighbortemperature eduction T,, = 0 95*T0uTable 22: Implementation specific parameters or SA

parameter valuestoppmng-conditwn II phase) 10 ocal optmuzatlons~mhalstateSn GA Dhasel mmS of n Dhase~mtml emperi&e ?, (SA phase) ) 0 l*cost(Si)Table 2.3 Implementation specific parameters or 2FO

The only parameter that needs some explanation m the abovetables IS he definmon of a local mmunum for II Because here Sa sign&ant cost mvolved m exhaustively searchmgall neighborsof a strategy (let alone in verifymg the truth of the precisedefimuon of a local muumum), we use an approxnnatlon to Iden-tlfy a local mmnnum In particular, a state IS consldered o be alocal mmunum after n randomly chosen neighbors of It are tested(with repetmon), where n IS the actual number of its neighbors,none of which has lower cost Note that thusdoes not guaranteethat all neighbors are tested, smce some may be chosen multlpletunes A state that sat.&ies the above operatlonal defimhon IScalled an r-local mmmum, to dlstmgursh It from an actual localmmunum Clearly, every local mmunum 1s an r-local mmlmurn,but the converse 1snot true Usmg the ldent&icatlon of an r-localmmunum as he stoppmgcntenon for a local opbmlzation unpliesthat some dotill moves may be occasionally missed, and astate may be falsely consldered as a local mmunum We chum,however, that the savmgsm execuuon tune by usmg this approxt-mation far outwelgh the potential nusses of rea l local mlmmaThis claun was venfied m a hrmted number of experiments thatwe performed3. PERFORMANCE EVALUATION

In this sechon, we report on an expemnental evaluation ofthe performanceand behavior of SA, II, and 2FOon query optun-lzatlon First, we descnbe he testbed hat we used for our expen-ments, and then we discuss he obtamed esults3.1. Testbed

We expenmentedonly with tree queries [Ullm82] contam-mg only equahty moms he to known dlfficulties m then optum-zatton [Ono88], specific attention was given to star quenes Treeand star quenes were generated andomly The query size rangedfrom 5 to 100 Jams Each query was tested m conJuncuon withthree different types of relation catalogs. I e , dtiferent relaboncardmahues and Jam selectlvltles We made the usual assump-tions about umform &smbutlon of values and mdependenceof

314


4/10

values m the Jam attnbutes [Seh79.Whan85] Because of theseassumptions, we used the number of umque values m the Joinattrlbutes to control Jam selectlvltles The catalogs that we usedare summarizedm Table 3 1

~1

Table 3.1: Relation catalogsAs an example for the meanmg of the enmes m Table 3 1, m therelcat3 catalog, relauon cardmahtles were randomly chosenbetween 1000 and 1OOOOOuples. and the number of umquevalues m the Jam columns was randomly chosen between 10%and 100% of the cardmahty of the correspondmg elmon Thecatalogs were selectedso that we could test quenes with differentdegreesof variance m the relation cardmahtles and pm selecnvl-hes, and thus, with different cost dlsmbutlons m the state spaceThis degree of variance mcreasesas we move from relcatl torelcat3

Each relation page contamed 16 tuples All relations hadfour atmbutes and were clustered on one of them There was aB+-tree or hashmg pnmary mdex on the clustered atmbute. or therelation was physically sorted on it These altemauves wereequlprobable The other attributes had a secondary mdex withprobability l/2, and agam there was a random choice between aB -tree and hashmg secondarymdex

Fmally. the JO~ methods that we consldered were nested-loops and merge-scan3.2. Experiment Profile

We vnplemented all algonthms m C. and tested hem on aSun-4 workstation when no-one else was usmg the machme Weallowed the query size to grow up to 100 JOT Twenty differentqueries were tested for each size up to 40 JOHIS, and five weretested for larger sizes For each query and relation catalog, SAand 2P0 were run five times. except for the caseswhere each runof SA would require more than two hours, for which no experl-ments were conducted with SA Thus, we have no results on SAfor both types of quenes with more than 60 JOUIS for catalogs rel-cat2 and relcat3. and for star quenes with more than 40 momsfor catalog relcat3 This declsmn was basedon the expectationthat the behavior of SA compared to II and 2P0 for the moreexpensive queries w111 e slmllar to that for the less expensiveones For each problem mstance. I was also run five tunes. eachrun havmg as much hme as the average Ume taken by a SA or2P0 run on the same query for the same catalog, dependmgonwhether there were SA runs or not respectively3.3. Behavior as a Function of Time

As part of the expenments, we recordedhow the mmunumcost found changedover tune durmg the course of the algorithmsexecution The typical behavior IS shown m Figure 3 1 The par-ticular example 1s or a 40-Join tree query with the relcatl cata-log The y-axis represents he raho of the strategy cost over themlmmum strategy cost found for the query among all runs of allalgonthms Clearly, there are slgmficant differences between SAand II On the one hand, after a few local optimlzatlons. II

reachesa stateof cost that IS close to the mmmum cost found bya complete run of SA The improvement that this cost representsover the uut~al andom statecost IS several orders of magmtude,mgeneral After that, II makes only small unprovements On theother hand, m the early stages,SA wanders around statesof veryhigh cost LIurmg the later stages.however, it reaches statesofcosts sunllar to those found by II after a few local optmuzatmns,and most often, It eventually finds a better state Tlus observatmnm&cates that SA 1sperformmg useful work only after It reacheslow cost statesand the temperature1s ow, smce at Hugh empera-ture, it only vlslts high cost states This fac t 1swhat mouvated themtmducUon of 2P0 The iirst phase of 2P0 produces a low coststate rom which the secondphasecan start with low temperatureIndeed, we observe that m&ally DO mprovea as qmckly as II.but soon It surpassest, and eventually converges o its final solu-tlon much more rapldly than SA

260 I

0 180 360 540 720 900Tune (Seconds)Figure 3 1: Muumum cost found over tune3.4. Output Quality

In general, we observed no slgmficant quahtative dlffer-ence m the relative output of the algonthms between ree and starquenes Becauseof thus and the lack of space,we only presentthe results for relcat2 for tree quenes, whereaswe present he700- --. SA5 80. ,-----. II --*- 2Po ,---I

scaled 4 60 tIcost ,:340, I

.0 20 40 60 80 100Number of JomsFigure 3.2: Average scaledcost of the output strategyfor tree quenes and the relcat2 catalog

315


5/10

2001 80

scaled1 6ocost140

700 700- - SA*--___ II 5 80 t: 5 80- 2PO I.*I-.* I 4 60, Scaled 4 60 Scaled tI cost : cost I

340 340 ,*- -4,*-- II1 20 220 220loo 100 100.0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100

Number of Jams Number of Jams Number of Jams( relcatl ) ( relcat2 ) ( relcat3 )Fpre 3.3. Average scaledcost of the output strategy of SA, JI, and 2P0 for star queries

211Scaledcost 111

oo---+ SA80 .--..-+ II- 2PO60 .,40 ..I..

20 ,I----4

,* A,,* / . .I00 * d

7 00I

5 80460Scaledcost 340

0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100Number of Jams Number of Jams Number of Jams(relcatl ) ( relcat2 ) ( relcat3 )Figure 3 4 Best scaledcost of the output strategyof SA, II, and 2P0 for star quenes

13200- 13200- 13200----.. SA10800 10800* 2PO 10800

Rumung 8400 Runmng 8400 Runmng 8400Tme Tlllle Time 1(se4 aoo. ; (set) (joc)c). (se4 ~0.

3600 1/ 3600 3600.1200. 1200

01 00 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100Number of Jams Number of Joms Number of Ioms

(a) tree quenes and relcatl (b) reequenes nd elcat2 (c) starquenes nd relcatlFigure 3 5. Average runnmg tune of SA and 2P0

316


6/10

complete set of results for star queries, smce hey are the hardesttree queries to optimize [On0881 The cos t of the output strategiesproduced by the algonthms for the various relatton catalogs as afunction of query size IS shown m Rgures 3 2 for tree quenes andFigure 3 3 for star queries Agam. the y-axis represents scaledcost. i e , the ratio of the output strategy cost over the mmunumcost found for the query among all runs of all algonduns Foreach size, the averageover all queries of that size. of the averagescaledcost over all five runs of each query IS shownWe discuss how the results change as we move along twodlmenslons of mterest query size and vanance m catalog parame-ters (1) Query size For small quertes.with 5 or 10 OUS, there ISno difference among the three algonthms. regardlessof the cata-log type Almost all runs of the three algonthms find stateswiththe samecost In general, as query size grows, the output of 2P0improves compared o that of SA. whch unproves compared othat of II At the samehme. however, the averageoutput strategycost of all algorithms becomes ess stable. 1e , the averagescaledcost moves farther from 1 This means hat there are more casesm which algonthms miss the optunum Of the three algonthms,however, 2P0 1s he relatively most stable one (u) Vartance mcatalog pummeters The output &fference between he algonthmsmcreaseswith higher vanance m the relahon cardmahtlesand theJoin selectivihes, i e , as we move from relcatl to re lcat3Interestmgly, there 1snot much &fference between relcat2 andrelcat3 In fact, occasionally, relcat2 gives rise to lugherdtfferences between 2P0 and lI/SA than relcat3 This showsthat variance m relation size has a more slgmficant impact on theoutput of the two algorithms than vanance m selectivity factors +It IS also mterestmg to compare the best output foundamong the five runs of each algonthm for each problem mstanceWe show the averageof that over all star queries of a given size mFigure 3 4 Clearly, when the best of five runs for each algorithm1sconsidered, 2P0 not only outperforms the other algonthms mall cases,but it also becomesvery stable, 1e , the best scaled costof five runs of 2P0 is very close, if not equal, to 1 ns suggeststhat 2P0 1s he algorithm of choice for large queries, particularlyIf It 1s un a small number of hmes for stability We should also

observe, however, that the performance of SA becomes verystable as well To the contrary, II 1s sttll not very stable, rarelyoutperformmg SA The effect of query size and vanance m cata-log parameters n the relative output quality of SA, II, and 2P0 mthis case1ssumlar to that for the averageof five runs, and we ela-borate on it no more3.5. Query Optimization Time

The average unnmg time of 2P0 and SA as a function ofquery size for vanous mterestmg cases 1s shown m Figure 3 5(Recall that II was given the sameamount of ttme as SA or 2PO.depending on whether SA was run or not.) Agam due to lack ofspace, we only show some of the relevant graphs Those thatcorrespond o the remammgcasesare slmdar Below, we discussthe effect of query size, variance m catalog parameters, nd querytype on query opttmtzatlon time (1) Query size Clearly, 2P0needs ess tune than SA m all cases As expected, the absolutedifference m runnmg time mcreaseswith query size To the con-trary, the relauve difference increases with query size for treequeries (It reaches a factor of 4 for NO-Jam tree queries with

Note that we used a different scale for the y-axis for relcatl thanfor the other catalogs

relcatl). but decreaseswith query stze for star quenes Thelatter tend to be less regular than tree quenes. and therefore, endto need more ttme m phase 2 of 2P0 (ii) Vurrmce m catalogpumneters The cost steadtly mcreasesrom relcatl to relcat3,as expected Thuscan be seen, or example, m Figures 3 5 (a) and3 5 (b). which correspond to tree quenes for relcatl and rel-cat2 (m) Query rype As expected [OnoSS], star quenes tookstgmficantly more optmuzanon tune than random tree quenes Inaddlhon, the improvement of 2P0 over SA for star quenes wasmuch smaller than for star quenes Thts 1s shown m Its moredramattc mstanttatmn m Figures 3 5 (a) and 3 5 (c), whichcorrespond o the two query types or relcatl 4. STATE SPACE ANALYSIS *

To understand the results of the performance evaluahonand the cause for the behavior of SA . II, and 2P0 that waspresentedm the previous section, we studied the shapeof the costfunction over the state space that the three algorithms had tosearch The outcome of 0~s study IS reported m this section4.1. Shape of Cost Function over the Strategy Space

The size of the strategy space1sprohibluve of any attemptof an exhaustive search of It Hence, m order to study the costfunction shape, andomlzabon was employed agam, and the fol-lowmg types of expenments were performed(1) Random generation of 10000 strategiesand calculahon ofthen costs(19 Random generation of 10000 ocal mmuna and calcula-bon of their costs ms was achieved by performmg alocal optmnzahon from each strategy generatedm expen-ment (1)(14 Random walks m areasof low cost strategies Our pur-pose was to get a feelmg for the number of good localmmuna that exst and then mutual distance For eachquery tested, we performed 5 random walks, each one ofwhch started rom a low cost local mlmmum Each walkwas a sequenceof 2000 smaller parts Each part consistedof a senesof uphill moves followed by a series of downhdlmoves Each senes of uphll moves ended when the s&a-tegy cost exceededa prespeclfied hmlt, which ensured hatthe search remamed m areasof low cost strategies (Thehmit was equal to 5 tunes the average ocal mm imum costfound m experunent (u)) Each senes of dotill movesended m a local mmlfnum Thus, a total of 10000 localmmuna were visited in expenment (iii) also

We tested en 20-Jam and ten 40-Jam ree queries, and anequal number of the same sizes of star queries Each query wastested with all three relahon catalogs For experunents (u) and(m). we used a better approxunatton for a local mmunum than ther-local mmlTnum that we used m the Implementatmn of II. toimprove the accuracy of the analysis In particular, we used theapproxunatton of the p-local mmwnum, which 1s defined as a

* The expenmentseportedm tius section or the state spaceanalysis were conducted usmg Condor [LIIz~~] Condor IS a faahty forexecutmg UNIX fobs on a pool of cooperatmg worksta tmns Jobs arequeued and executed remotely on workstatIons at rimes when those works-tauons would otherwise be idle Our expemnents are very tune-consummg Without Condor It would be. very dtificult to collect all thenecessary data m a reasonable ume

317


7/10

strategynone of whose neighbors has a lower cost Note that pla-teaux can stfi be nustaken as local mmuna with this defimtumUnless otherwise noted, any reference to a local muumum 111 ussubsection efers to a p-local muumumThe results of expenmenta (I) and (u) are summarized mFigure 4 1 for star quenes We do not show the results of the ran-dom tree quenes tested,because hey are very smular to those ofFigure 4 1 Quenes 1 to 10 have 20 JOUIS and quenes 11 to 20have 40 moms There IS no sigmficance m the order of placement

of the various quenes on the x-ax=, except that they are groupedmto those with 20 JOIIIS and those with 40 JOIIIS For each query,we show the scaled cost of the average strategy (expenment (I))and the scaled cost of the average ocal mmm mm (experunent (II))that corresponds o the query Agam, the scaled cost 1s he raboof a strategy cost over the lowest local mmunum cost found mexpenment (II) for the correspondmg query The average ocalmuuma costs are several orders of magmtude ower than the aver-age state costs As the query size grows, the difference remamsrelatively stable for quenes with the same catalog, although theabsolute scaled costs mcrease On the other hand. the &fferenceseems o mcrease as the catalog changes from relcatl to rel-cat3 Fmally, compared o the averagecost of random strategies,the average cost of local mmnna 1s relanvely close to the bestlocal nummum cost The speck ratio of averagevs best localnummum cost IS affected by the vanance m the catalog parsme-ters and by the pzutlcular query itself In some cases, t representscost differences as high as two orders of magmtude (e g , do-JOIIIqueries with relcat3) Even m these cases,however, that costdifference 1smslgmficant compared o the tiference between theaverage strategy cost and the best local mmmwm cost, whch 1shigher than five orders of magmtude Thus, we can conclude thatmost ocal mmuna are not only far better than the average andomstate,but there s also relatively small variance 111heir costs

Durmg expenment (u), we also measured he number ofdownhdl moves taken by the local optumzat~ons Figure 4 2shows the averagenumber of dotill moves for each star queryTlus number of downhrll moves IS higher for star queries than fortree quenes Moreover, It mcreases s he query size grows and asthe catalog changes rom relcatl to relcat3, with a maxunum

for relcat2 The general conclusion from the results m Figure4 2 1s hat sta rtmg at a random state many downlull moves areneeded o reach a ocal mmunum4001 20 JO&--360320.Number f 280?EF 240

200.160.120.

0- b........,024681 12 14 16 18 20

--%oJOUlS .

dcal2

ii

relcat3

elcat

l ------- + averagestrategy+-- - -+ averagelocalmm

best ocal mm20JOlIlS +

::

Query NumberFigure 4.2: Number of downlull moves for star queries

In expenment (III), for the sameset of tree and star quenesas before, we counted the number of local mmuna vls~ted hat haddstmct costs This only provides a lower bound on the number ofdlstmct local numma In ad&tton, for each query, we measuredthe average &stance between two consecuttvely visited localmmuna Agam, smce here could be shorter paths between them,dus only provides an upper bound on their dstance These resultsare summarized m Table 4 1 where we show the range of valuesfor both measuredquantlhes for all quenes of both query typesand alI three catalogs

scaledcost

4 I . . . . . . . . . _I. . ._ . . , .0 2 4 6 8 101214161820 0 2 4 6 8 101214161820 0 2 4 6 8 101214161820QueryNumber Query Number Query Number( relcatl ) ( relcat2 ) ( relcat3 )Figure 4 1: Cost dlstnbutlon of random statesand ocal mmuna for star quenes

318


8/10

Table 4 1: Local rnma m the low cost areaThe general concluston from the results m Table 4 11s hat, for aiiqueries, any connected area of relauvely low cost strategtes dueto the prespecrfiedhmrt) contamsmany local numma Moreover,these ocal mmma are relattvely close to each other (the stze ofthe state space1sseveral orders of magmtudehigher than any drs-tance eported m Table 4 1)

All the above observatrons an be summarizedas ollows(1) The average ocal mmtm um has relattvely low cost com-pared o the averagestate (from (1)and (u))(2) The average distance from a random state to a localmmrmum s long (from (u))(3) The number of local mmrma1s arge (from (ui))(4) Many local mmuna are connected hrough low cost stateswtthm short dtstance from (m))In addttton to the above, the overall behavior of the three algo-nthms describedm Sectton 3 ts summanzedbelow(5) Searching among ow cost states hat are connected o eachother produces better results than searchmg m multtple.(potenttally) unconnected,areasof states

The above pomts (l)-(5) lead to the followmg coniectureregardmg he shapeof the cost functton over the strategy spaceThe shapeof the cost functwn resembles a %up,wtih some elatrvely small vanatrons at the bottom

In other words, there is a small area of strategieswith low costs,the cup bottom, surrounded by the remammg strategies wtthmcreasmgly htgher costs There 1s relatrvely small vanattonamong the costs m the cup bottom, but enough to make explora-uon of that area worth whtle The space wtth which we deal 1smultrdunenstonal, so rt IS hard to vtsuaitze For a ldunenstonaicost functton. the correspondmgsttuauon 1sshown m Ftgure 4 3

StSbFigure 4.3 Shapeof wst funcuon

Actually, the reported expenmental results do not excludethe case where the shape of the cost funchon 1s several cupswhose bottoms are at smular cost levels Addmonai experrments,however, mdrcate hat the exrstenceof multtple cups 1sunhkelyIn partrcular, whenever the output strategiesof two runs of 2P0on the same query differed stgmiicantiy m cost, we ran SA wtthlow temperaturestartmg rom one of the strateges The low tem-perature guaranteed hat the algorithm dtd not move mto high coststates. e , that it remamedm the same cup In aii cases ested,SA dtd vlsrt the other strategy,which mdrcates hat the mabritty of2P0 to vrsit the best of the two strategiesm both runs was mosthkely due to the randomness f the algorithm and not due to 2P0searchmgm two d ifferent cups m the two runs4.2. Explanation of Behavior as a Function of Time

Usmg the wmecture of the cup shapedcost functton, weare now m a posmon to explam the typical behavtor of SA, II, and2pO over ttme as shown m Figure 3 1 SA starts from a randomstate, which tends to be at the high wst area Whtle the tempera-ture remamshtgh, due to the large number of uphill moves fromstatesm the high and middle cost area,and the iugh probabthty ofacceptmg uphrii moves, SA tends to spend much tune withouttmprovement. After the temperature1s educed srgmficantly. SAreaches he cup bottom, which rt explores extensrvely, by takmgadvantageof tts abtitty to vrstt many local muuma by acceptmguphtll moves On the other hand, II can reach the cup bottomqmckiy by acceptmg only downhrll moves Smce most localmmuna are there, II can 6nd a relatrvely good one wnhm a fewlocal opttmtzattons Thrs explams why II performs so well m thebegummg stages,while SA performs so poorly As for 2F0, asexpected, It reaches he cup bottom very qmckly (II phase) andthen improves further by searchmgm that area SA phase), imsh-mg m less ttme than either of the other algonthms4.3. Explanation of Output Quality

We can also explam why 2P0 (and usually SA also) out-performs II m terms of output quahty II vtsits relattvely fewlocal mmuna, because6ndmg one 1sexpensive for the followmgreasons (a) for each local optrmtzahon, II has to generate a

-- SAa.__-. II- 2Po

--,II :yJIL; . ..21420 lOlOOi000

-1-1I1scaled costFigure 4.4: Number of vtstts m each wst area

319


9/10

random state and evaluate tts cost, both of which are expenstveoperattons, (b) startmg at a random state, tt takes tune to find alocal mtmm um, because the dtstance between the two IS long(pomt (2) and Figure 4 2). and (c) durmg a local opumtzatton.espectaily when m a low cost state, I tnes many neighbors beforeIt finds a downlull move On the other hand, 2pO and SA spend areasonableamount of ttme at the cup bottom (with low tempera-ture), and are able to explore rt much more thoroughly than II,thus mcreasmg he probabthty that they find the global mmtmumThe above was also verified by measurmg he number of statesvrstted by each aigortthm as a functron of the cost of the stateWe observed hat, although overall II vrsrts more states han SA,whtch vtsns more states han 2PG, the last two vrstt more statesoflow costs han the first A typtcal sttuahon ts shown m Ftgure 4 4The example ts for a 40-Jam ree query with the relcatl catalogSrmtlar behavtor was observed or other quenes5. INCLUDING HASH-JOIN, PIPELINING, ANDBUFFERING

As an extenston to the study described m the prevrous sec-uons, we have conducted a lmnted set of expertmentswrth a stra-tegy space hat mcluded hash-join as an alternative Jam method(m addttton to nested-loops and merge-scan) and wrth a wstfuncuon that parhaily removed assumptrons a) and (b) of Sectton2 2 3, t e , tt allowed prpelmmg for nested-loops and m severalcases took advantage of arbitrary amounts of available bufferspace We expenmented wrth twenty 20-Jam and twenty 40-Jamtree quenes with all three catalogs The set-up of these expen-ments was exactly the one describedm the prevrous sectrons Westudtedboth the behavror of ail three algonthms m the new settmgand the charactensttcs of the shape of the cost functmn Theresults are very smular to those of Sectton 3 and 4 2PG alwaysoutperforms I and SA m terms of both output quahty and runnmgume In addttron, he conJectureof the cup -shaped cost functronremamsvahd This lmphes that our wnclusmns m Sectron3 and4 were not a result of our choice of Jam methods or the specrficrestrtcttons (a) and (b) on the cost functron As a pomt of refer-ence, m Tables 5 1 and 5 2. we show the averageand best scaledcost of the output strategms f five runs for ail three aigonthms forall combmattonsof query srze and catalog As before, the costsare scaled based on the lowest strategy cost found for eachspectfic query and catalog, and averagedover ail quenes wtth thesamecharactensttcs

relcatl relcat2 relcat320 1 40 20 1 40 20 1 40II 101 I 101 125 ] 357 109 ] 142SA 100 100 1 12 136 102 1112P0 100 100 107 129 101 106Table 5 .1. Average scaledcost of the output strategy

relcatl relcat2 relcat320 1 40 20 1 40 20 1 40II loo I100 113 ] 311 105 ] 132

Table 5 .2. Best scaledcost of the output strategy6. RELATED WORK

Query optrmtzatton has been a very active areaof researchfor relattonal database systems The reader 1s referred to the

survey paper by Jarke and Koch [Jark84] and the book by Km,Remer. and Batory [Km861 Regardmg arge JOT queries, thetrophmtzauon was spectficaiiy addressedby Knshnamurthy et al[Kns86]. who proposeda quadrauc algorithm that took advantageof the form of the Mom ost formula. In thts section, we want topnmanly compare our work wrth that of Swam1 and Gupta[Swam88], who conducted a performanceevaluauon of SA and Iithat was srmtlar to our study reported m Sectron 3 There areseveral drstmct drfferencesbetween he two stu~es Fust. Swarmand Guptas strategy space consrsts of left-deep trees only,whereaswe mchtde ail strategms Second, hey exammeonly oneMomalgorithm..namely hash-Jam,whereas we have expenmentedwrth two of them, namely nested-loops and merge-scan Thud,they assumeda mam memory databaseand therefore used cpu-ttme as the cost of a strategy, whereas we used I/G ttme for thatFourth, they used an approxtmatton for a local mmmum m thermplementattonof II that 1s different from the r-local m mmmmthat we used Ftfth, the neighbors of a strategy are determmed ytwo transformatmn ules, namely cychc exchange of the posrtmnof two or three relations m the JOT tree that corresponds o thestrategy, whereas most of our transformatron ules are based onalgebrarcproperues of m oms There are also several other imple-mentatron dtfference between the two studies that seemuNleces-sary to pomt out hereThe results of our study presentedm Sectton 3 contradtctthose of Swami and Gupta [Swam88], whose conclusron was thatSA was never supenor to II, mdependentof the amount of tunethat was gtven to rt In prmcrple, any wmbmatton of the dtffer-ences menttoned above could be. he source for the difference mthe results Inttuttvely. however, we beheve that the pnmary rea-son 1s he difference m the choice of nerghbors or the strategiesmthe strategy space, e , the &fference m the transformatton ulesthat were used m the two studies Swarm and Guptas transforma-tion rules generateneighbors that have large differences m thetacost, which makes the shape of the cost functton much lesssmooth (not a cup) Therefore, SA does not have the opportunttyto spend much ume m a low cost area and performs poorly Onthe other hand II can move down to a local mmtmum m a few

moves and therefore vtstts many of them m the same amount oftrme Thus, the two aigonthms are ordered differently m terms ofoutput quairty m the two studms Although we beheve that mostltkely the above ts the mam reason for the dtfference m wnclu-stons. further mvesttgatton ts required to understand the tssuespreaselyAs a closmg wnunent, we should mentton that our workon mappmg he shapeof the cost functton over the strategy spaceof a query 1sumque, Swami and Gupta included no such study mthen work Moreover, the cup formatron gave the abrhty to use2pO as an opttmtzatton algonthm, which exiubtts supenor perfor-mance Gn the other hand, Swarm proposed and expenmentedwtth a set of heunsttcs. which m general improved the perfor-manceof both II and SA [Swam891

7. CONCLUSIONS AND FUTURE WORKWe have adapted he well-known randomrzedopumtzatmnalgonthms of Stmulated Anneaimg and Iterattve Improvement toopttmizatmn of large JOIII queries and stud& theu performanceWe observed that II performs better than SA mrttally, but rfenough tuue 1s given to SA, rt outperforms II We studted theshape of the wst funcuon over the state space of a query, andexperunentaily venfied that rt resemblesa cup wtth a non-smoothbottom Based on thrs observatton.we explained the behavior ofthe two aigonthms Fmaily, makmg use of the cup shape of the

320


10/10

cost funcbon, we proposed the Two Phase Opttmtzatmn algo-nthm, whose performance 1s supenor to that of the other algo-rithms with respect o both output qualrty and rumung ttmeThe work reported m dus paper s only the begmnmg munderstandmghow randonuzed opmlzahon algonthms performon complex quenes, and also what the shapeof the wst funcuon1s over the space of equivalent strategies or queries There areseveral ssues hat are mterestmgand on which we plan to work m

the future First, we want to mveshgate the senslt~vvlty f dusstudys wncluslons to the spectfic choices that we made for van-ous parameters In particular, we would hke to complete ourexperiments with hash-Jam, ptpelmmg, and various degrees ofbuffer ava labtltty, which all a ffect the cost formulas for m&vi-dual JOISIS and the cost relatlonshlps between neighbors Wewould also hke to experunent wtth non-umformly dlstnbuteddata Second , we want to tdentlfy the key properties that causethe cup formahon of the wst funchon tis will be helpful notonly m understandmg he behavior of the algonthms. but also mprovldmg us wtth cntena for the apphcablllty of Two PhaseOphmlzahon Such results wfi be very helpful m extensiblequery opttmlzers [Care861 md. we want to compare TwoPhaseOptumzatlon and the other randomtzed ophmlzation algo-nthms with the tradttlonal ones, e g , those of System-R [Sell791or Starburst L&m881 tis should lead mto an understandingofthe relative advantages between generatlon-based andtransformation-based query optmuzatlon Fmally, we want toexperiment with other types of relahonal quenes, i e, cyclicqueries and queries that mvoive umon8. REFERENCES[Care86]M Carey and et al, The Architecture of the EXODUSExtensible DBMS, m Proc of the 1986 InternatronalWorkshopon Object-OrrentedDatabaseSystems, &ted byK Dlttnch and U Dayal, Pac&ic Grove, CA, September1986.pp 52-65[Grangl]

J Grant and J Mmker. Optumzahon m Deducbve andConvennonal Relational DatabaseSystems,m AdvancesmData Base Theory, Vol I, e&ted by H Gallare, J Mmker.and J M Ntcolas. Plenum Press,New York, NY, 1981.pp195-234[Ioan87]Y E Ioanmdls and E Wong, Query Opbmtzahon bySlmuiated Amealmg, m Proceedmgs of the 1987 ACM-SIGMOD Conference, San Francisco. CA, June 1987. pp9-22[ &irk841M Jarke and J Koch, Query Optlmlzabon m DatabaseSystems, ACM Computrng Surveys 16 (June 1984), pp111-152[ John871D S Johnson,C R Aragon, L A McGeoch, and C Sche-von, Optrmrzatwn by StmulatedAnnealrng An Experunen-tal Evaluatron (Part I), unpubhshedmanuscript. June 1987[Klm86]W Kim, D Remer, and D Batory, Query Processrng m

DatabaseSystems,Spnnger Veriag, New York, NY, 1986[Kuk83]S Knkpatnck. C D Gelatt, Jr, and M P Vecch, Ophml-zatton by Simulated Annealmg, Scrence 220, 4598 (May1983),pp 671-680[ KllS861R Knshnamurthy, H Boral, and C Zamoio, Ophmlzabonof Nonrecurslve Queries, 111 roceedrngs f the 12th Inter-

natwnal VLDB Conference, Kyoto, Japan, August 1986.pp 128-137[Lttz88]M J Lttzkow, M Llvny, and M W Mutka, Condor - AHunter of Idle Workstatmns, m The 8th InternatronulCoYlference n Dtstrtbuted Computmg Systems,1988, pp104-111[L&m881G M L&man, Grammar-hke Funcnonal Rules forRepresentmgQuery Optmnzatlon Alternatives, m Proceed-mgs of the 1988 ACM-SIGMOD Conference,Chicago, IL,June 1988,pp 18-27[Naha86]S Nahar, S Sal-m. and E Shragowltz, Sunulated Anneai-mg and Combmatortal Optmnzatlon, m Proceedmgs f the23rd DestgnAutomatwn Conference, 1986,pp 293-299[ono88]K One and G M L&man, Extensrble Enumeratron ofFeasible Jams for Relatronal Query Optumzation. IBMResearchReport RJ6625.December1988[ Rome851F Romeo and A Sanglovanm-Vmcentelh, ProbablhsbcHill Chmbmg Algorithms Properhes and Apphcattons,Proceedingsof IEEE Conference on VLSI, 1985, pp 393-417[ Se11791P Selmger, M M Astrahan, D D Chamberim, R A

Lone, and T G Price, Access Path Selection m a Rela-uonal DatabaseManagementSystem,111 roceedingsof theI979 ACM-SIGMOD Cor$erence,Boston, MA, June 1979,pp 23-34[Se11881T K Selhs, Multtpie Query Optumzabon, ACM Transac-twns on DatabaseSystems13.1 (March 1988),pp 23-52[Swam881A Swarm and A Gupta, Optmuzatlon of Large JomQueries, ~fl Proceedmgs of the 1988 ACM-SIGMODCo$erence, Chicago. IL, June 1988,pp 8-17[Swam891A Swarm,Optunlzahon of Large Jom Queries Combmm gHeunsttcs and Combmatonal Techmques, m Proceedmgs

of the 1989 ACM-SIGMOD Conference, Portland, OR,June 1989,pp 367-376[Uilm82]J D Ullman, Prrnctples of Database Systems,ComputerScmwe Press,Rockvdle, MD, 1982[Whan85]K Y Whang. Query Optrmrzatron m Office-by-Example,IBM Researchreport,RC11571,1985

321

Documents

5. Randomized Algorithms for Optimizing Large Join Queries - Ioannidis