Embed Size (px)
Citation preview
15
Outline
v Selection
v Sorting routine
v Join
v Projection
v Set operators
v Group By aggregation
16
Why Sort?
v Important utility in DBMS:§ Request data in sorted order (e.g., ORDER BY)
• e.g., find students in decreasing order of gpa
§ Sort-merge join algorithm involves sorting.§ Eliminate duplicates in a collection of records (e.g., SELECT
DISTINCT)§ Sorting is first step in bulk loading B+ tree index.
v Problem: sort 1TB of data with 1GB of RAM § Limited memory à key is to minimize # I/Os!§ Methodology for algorithm design: from simple to complex
17
2-Way Sort: Requires 3 Buffersv Pass 1: Read a page, sort it, write it as a sorted subfile
§ only one buffer page (from the memory) is used
v Pass 2, 3, …, etc.: Merge two sorted subfiles§ three buffer pages are used
Main memory buffers
INPUT 1
INPUT 2
OUTPUT
DiskDisk
18
Two-Way External Merge Sort
A file of N pages:Pass 1: N sorted runs of 1 page eachPass 2: N/2 sorted runs of 2 pages
eachPass 3: N/4 sorted runs of 4 pages
each…Pass P+1: 1 sorted run of 2P pages
2P ³N à P ³ log2N
Input file
1-page runs
2-page runs
4-page runs
8-page runs
PASS 1
PASS 2
PASS 3
PASS 4
9
3,4 6,2 9,4 8,7 5,6 3,1 2
3,4 5,62,6 4,9 7,8 1,3 2
2,34,6
4,78,9
1,35,6 2
2,34,46,78,9
1,23,56
1,22,33,44,56,67,8
v Divide and conquer:sort subfiles (runs) and then merge
19
Two-Way External Merge Sort
• Each pass, read + write N pages in file à 2N.
• Number of passes is:
• So total cost is:
! " 1log2 +N
! "( )1log2 2 +NN
Input file
1-page runs
2-page runs
4-page runs
8-page runs
PASS 1
PASS 2
PASS 3
PASS 4
9
3,4 6,2 9,4 8,7 5,6 3,1 2
3,4 5,62,6 4,9 7,8 1,3 2
2,34,6
4,78,9
1,35,6 2
2,34,46,78,9
1,23,56
1,22,33,44,56,67,8
v Divide and conquer: sort subfiles (runs) and then merge
20
General External Merge Sort
v Pass 1: Use B buffer pages. Produce éN/Bù sorted runs of B pages each.
v Pass 2, 3…, etc.: Merge B-1 runs.
B Main memory buffers
INPUT 1
INPUT B-1
OUTPUT
DiskDisk
INPUT 2
. . . . . .. . .
Given B (>3) buffer pages. How can we utilize them?
21
Cost of External Merge Sort
v Number of passes = 1 + élog B-1 éN/Bù ùCost = 2N * (1 + élog B-1 éN/Bù ù )
v E.g., with 5 (B) buffer pages, sort 108 (N) page file:
Pass 1 é108/5ù = 22 sorted runs of 5 pages each (last run is only 3 pages)
éN/Bù sorted runs of B pages each
Pass 2 é22/4ù = 6 sorted runs of 20 pages each (last run is only 8 pages)
éN/Bù /(B-1) sorted runs of B(B-1) pages each
Pass 3 2 sorted runs, 80 pages and 28 pages éN/Bù /(B-1)2 sorted runs of B(B-1)2 pages
Pass 4 Sorted file of 108 pages éN/Bù /(B-1)3 sorted runs of B(B-1)3 (³N) pages
22
Number of Passes of External Sort
N B=3 B=5 B=9 B=17 B=129 B=257100 7 4 3 2 1 11,000 10 5 4 3 2 210,000 13 7 5 4 2 2100,000 17 9 6 5 3 31,000,000 20 10 7 5 3 310,000,000 23 12 8 6 4 3100,000,000 26 14 9 7 4 41,000,000,000 30 15 10 8 5 4
23
Output(1 buffer)
124
3
5
28
10
Input(1 buffer)
Current Set(B-2 buffers)
Replacement Sortv Replacement Sort: Produce initial sorted runs as long as
possible. Used in Pass 1 of sorting.
v Organize B available buffers:§ 1 buffer for input§ B-2 buffers for current set§ 1 buffer for output
6
24
Output(1 buffer)
124
3
5
28
10
Input (1 buffer)
Current Set(B-2 buffers)
Replacement Sort
v Pick tuple r in the current set (CS) such that r is the smallest value in CS that is ³ largest value in output, e.g. 8, to extend the current run.
v Write output buffer out if full, extending the current run.
v Fill the space in current set by adding tuples from input.
v Current run terminates if every tuple in the current set is < the largest tuple in output.
6
25
Output(1 buffer)
124
3
5
28
10
Input (1 buffer)
Current Set(B-2 buffers)
Replacement Sort
v Good empirical results: When used in Pass 1 for sorting, write out sorted runs of size 2B on average.
v Affects calculation of the number of passes accordingly.
6
30
Outline
v Selection
v Sorting routine
v Join
v Projection
v Set operator
v Group By aggregation
31
Equality Joins With One Join Column
v R wv S, natural join. Very common operation!v Semantics: cross product (´) followed by selection (s)
§ If R ´ S is large, R ´ S followed by a selection is inefficient.§ Must be carefully optimized.
v Cost metric: # of I/Os. Ignore output cost in analysis.§ R: M pages, pR tuples per page § S: N pages, pS tuples per page.
SELECT *FROM Reserves R, Sailors SWHERE R.sid = S.sid
32
Schema for Examples
v Sailors:§ Each tuple is 50 bytes long, § 80 tuples per page, § 500 pages.
v Reserves:§ Each tuple is 40 bytes long, § 100 tuples per page, § 1000 pages.
v Cost metric: # I/Os
Sailors (sid: integer, sname: string, rating: integer, age: real)Reserves (sid: integer, bid: integer, day: date, rname: string)
33
1) Page-Oriented Nested Loops Joinv A baseline approach:
v Cost: M + M * N = 1000 + 1000*500 = 501,000 I/Os.§ 2M random I/Os; others are sequential I/Os.
v How many buffer pages do we need?
foreach page of R doforeach page of S do
write out each matching pair <r, s>//r is in R-page, s is in S-page
34
2) Block Nested Loops Join
v How can we utilize additional buffer pages?§ If the smaller reln, say R, fits in memory, use R as outer, read
the inner S only once.§ Otherwise, read a big chunk of R each time, hence reducing #
times of reading S. v Block Nested Loops Join:
§ The smaller reln R as outer, the other S as inner.§ Buffer allocation:
• 1 buffer for scanning the inner S • 1 buffer for output • All remaining buffers for holding a “block” of outer R
35
Block Nested Loops Join (Contd.)
. . .
. . .
R & SBlock of R(hash table, size £ B-2 pages)
Input buffer for S Output buffer
. . .
Join Result
foreach block in R dobuild a hash table on R-blockforeach page in S do
foreach matching tuple r in R-block, s in S-page doadd <r, s> to result
36
Cost of Block Nested Loops Join
v Cost: Scan of outer + #outer blocks * scan of inner§ B buffer pages available. If M < N§ Cost = M+ éM/ B-2ù * N
v E.g. B=102, Sailors S = 500 pages, Reserves R = 1000 pages. § What is the cost if S is outer, R is inner?
• A block = B-2 = 100 pages• Cost = 500 + é500/100ù * 1000 = 5,500 I/Os.
§ What is the cost if we swap R and S? • Cost = 1000 + é1000/100ù * 500 = 6,000 I/Os.
§ Which relation should be the outer for smaller cost?
38
3) Index Nested Loops Joinv Given an index on the join column of one relation, say S:
v Cost: M + ( M * pR * cost of finding matching S tuples) 1) Cost of search in S index:
• Hash index: about 1 I/O to search + extra pages for matches• B+ tree: 2-4 I/O�s to search + extra pages for matches.
2) Cost of retrieving matching S tuples (assuming Alt. 2 or 3):• Clustered index: one or a few I/O�s (typical).• Unclustered: up to 1 I/O per matching S tuple.
foreach tuple r in R doforeach tuple s in S where r == s (via index lookup) do
add <r, s> to result
41
4) Sort-Merge (R S) for Equi-Join
v Sort R and S on join column using external sorting. v Merge R and S on join column, output result tuples.
▹◃
Repeat until either R or S is finished:§ Scanning:
• Advance scan of R until current R-tuple >=current S tuple, • Advance scan of S until current S-tuple>=current R tuple; • Do this until current R tuple = current S tuple.
§ Matching: • Match all R tuples and S tuples with same value (called R-
group and S-group of the current value). • Output <r, s> for all pairs of such tuples.
42
Example of Sort-Merge Join
v Cost: Sorting_cost(R) + Sorting_cost(S) + Merging_cost§ Merging_cost Î[M+N, M*N]§ M+N: foreign key join with the referenced reln. as inner.§ M*N: uncommon but possible. When?
v What is the I/O pattern in the sort-merge join?v How many buffers are needed in the merge phase?
sid sname rating age22 dustin 7 45.028 yuppy 9 35.031 lubber 8 55.544 guppy 5 35.058 rusty 10 35.0
sid bid day rname28 103 12/4/96 guppy28 103 11/3/96 yuppy31 101 10/10/96 dustin31 102 10/12/96 lubber31 101 10/11/96 lubber58 103 11/12/96 dustin
43
Refinement of Sort-Merge Joinv When can we achieve a 2-pass algorithm for foreign key-
primary key join join?
v Observed repeated merging phases:§ Sorting of R and S has respective merging phases.§ Join of R and S also has a merging phase.§ Combine all these merging phases!
B memory buffer pages
INPUT 1
INPUT B-1
OUTPUT
DiskDisk
INPUT 2
. . . . . .. . .
44
Merging in Two-Pass Sort-Merge
B memory buffer pages
Run1 of R
RunK of R
OUTPUT
Join ResultsRun2 of R
. . .
Relation R
. . .
Run1 of S
RunK of S
Run2 of S
Relation S
. . .
45
Merging in Two-Pass Sort-Merge
B memory buffer pages
3 3 2 1 1
5 4 4 3 1
OUTPUT
Join Results7 6 5 2 2
. . .
Relation R
. . .
7 6 3 2
12 11 10 4
9 8 5 1
Relation S
. . .
46
Two-Pass Sort-Merge Join
v Pass 1 Sorting: sort subfiles of R and S individuallyv Pass 2 Merging: merge sorted runs of R and S
§ merge sorted runs of R, § merge sorted runs of S, and § compare R and S tuples using the join condition.
v There exists a linear (2-pass) algorithm for foreign key-primary key join if certain conditions hold
48
v Memory requirement for two-pass sort-merge:
§ Sorting pass produces sorted runs of size up to 2B. So,Number of runs = (M+N)/2B.
§ Merging pass holds sorted runs of both relations and an output buffer. So, (M+N)/2B + 1 <= B ® B >
§ Sometimes, can see a looser bound B > , U=max(M,N)
v Cost: read & write each relation in sorting pass+ read each relation in merging pass
= 3 ( M+N ) !(+ writing result tuples, ignored here)
Memory Requirement and Cost
(M + N ) / 2
U
49
v Idea: For an Equi-Join, partition both R and S using a hash function s.t. R tuples will only match S tuples in partition i.
5) Hash-Join for Equi-Join
B memory buffer pages DiskDisk
Original Relation OUTPUT
2INPUT
1
hashfunction
h B-1
Partitions
1
2
B-1
. . .
v Phase 1 Partitioning: Partition both relations using hash function h (Ri tuples will only match with Si tuples).
50
Hash-Joinv Phase 2 Probing:
§ Read in partition Ri, build hash table using h2 (<> h!). § Scan partition Si, one page at a time, search for matches.
Partitionsof R & S
Input bufferfor Si
Hash table for partitionRi (k £ B-2 pages)
B memory buffer pagesDisk
Output buffer
Disk
Join Result
hashfnh2
h2
51
Memory Requirement
v Partitioning: # partitions in memory = B-1, Probing: to fit each Ri in memory, size of partition ≤ B-2. § A little more memory needed to build hash table, but ignored here.
v Assuming uniformly sized partitions, L = min(M, N): § L / (B-1) £ (B-2) à B > + 1§ Use the smaller relation as the building relation in probing phase.
v What if hash fn h does not partition uniformly? § One or more R partitions may not fit in memory. § Can apply hash-join recursively to this R-partition and the
corresponding S-partition. Higher cost, of course…
v What is the I/O pattern in the hash join?
L
52
Cost of Hash-Joinv Partitioning: reads+writes both relns; 2(M+N).
Probing: reads both relns; M+N I/Os. Total cost = 3(M+N).§ In our running example, a total of 4500 I/Os using hash join
(compared to 501,000 I/Os w. Page NLJ).
v Sort-Merge Join vs. Hash Join:§ Given a minimum amount of memory (what is this, for each?) both
have a cost of 3(M+N) I/Os. § Hash Join superior on this count if relation sizes differ greatly. § Sort-Merge less sensitive to data skew; result is sorted.
57
Rough Comparisons of Join Methods
Buffer size B
# I/Os
3 L U L U L+U
L+U
3(L+U)
L+L*U
2Llog2L+2Ulog2U+3(L+U)
(Hybrid) Hash Join (no skew): [3(L+U), L+U]
Block NLJ: L+LU/(B-2)Sort Merge: [2LlogB-1L+2UlogB-1U+3(L+U),
L+U]
? ?
58
General Join Conditionsv Equalities over several attributes (e.g., R.sid=S.sid AND
R.rname=S.sname):ü Block NL works fine.ü For Index NL,
• use index on <sid, sname> if available; or • use an index on sid or sname, check the other predicate on the fly.
ü For Sort-Merge and Hash Join, sort/partition on combination of the two join columns.
59
General Join Conditionsv Inequality conditions (e.g., R.rname < S.sname):
ü For Index NL, need B+ tree index.• Range probes on inner; number of matches likely to be much
higher than for equality joins. • Clustered index is much preferred.
ü Block NL often works well.û Hash Join, Sort Merge Join not applicable.
67
Outline
v Selection
v Sorting routine
v Join
v Projection
v Set operators
v Group By aggregation
68
Aggregate Operations (AVG, MIN, etc.)
v Aggregation without grouping§ File scan: in general, requires scanning the relation.§ Index only scan: if a tree index�s search key includes all
attributes in the SELECT and WHERE clauses. • e.g. B+tree on <rating, age>
SELECT min(S.age)FROM Sailors SWHERE S.rating = 10
69
Aggregate Operations (contd.)
v Aggregation with grouping (GROUP BY)§ Single-relation sorting: sort by group-by attribute(s); compute
aggregate for each group in last merging phase.§ Single-relation hashing: hash on group-by attribute(s): compute
aggregate using in-memory hash table for each partition.§ Index only scan: if a tree index�s search key includes all
attributes in SELECT, WHERE and GROUP BY clauses. • e.g. B+tree on <rating, age>
SELECT min(S.age)FROM Sailors SWHERE S.rating > 5GROUP BY S.rating
SELECT count(distinct C.userid)FROM Clicks CGROUP BY C.url
70
A Baseline Sorting Alg. for Group by
q Step 1: sort the entire relation by R.aq Step 2: scan the sorted relation, find
each group on the fly, compute aggregate on R.b
SELECT sum(R.b)FROM RGROUP BY R.a
v The above algorithm needs to scan R many times.v Is it possible to limit the algorithm to 1 pass?
§ When memory size B >= relation size N
v Is it possible to limit the algorithm to 2 passes?§ See next.
71
Two-Pass Sorting for Group Byv Pass 1: Produce éN/Bù sorted runs of B pages each.v Pass 2: Merge B-1 runs to produce a final sorted file directly.
Find each group on the fly and compute the aggregate.v To finish in 2 passes, we have éN/Bù<=B-1, or B>sqrt(N)
B Main memory buffers
INPUT 1
INPUT B-1
OUTPUT
DiskDisk
INPUT 2
. . . . . .. . .
Find groups &aggregate
72
v Idea: when |R|=N > B, partition R using a hash function s.t. the i_th partition can be processed in memory later.
Two-Pass Hashing for Group By
B memory buffer pages DiskDisk
Original Relation OUTPUT
2INPUT
1
hashfunction
h B-1
Partitions
1
2
B-1
. . .
v Phase 1 (Partitioning): Partition R into B-1 buckets using hash function h
73
Two-Pass Hashing for Group Byv Phase 2 (Probing):
§ Read in partition Ri, find groups (using in-memory sorting or building a hash table using h2 <> h!), compute the aggregate in each group
§ To enable Ri to fit in memory, we have N/(B-1)<=B-1, or B>sqrt(N)
Partitionsof R
Hash table for partitionRi (k £ B-1 pages)
B memory buffer pagesDisk
Output buffer
hashfnh2
v IO cost of two-phase hashing: 3N (ignore the output cost)
74
Comparison of Hash Algorithms
Buffer size B
# I/Os
3 N N
N
3N
Two-phase hash: 3N
Can we do better than 3N given more memory than ?
One-phase hash: N
N
