Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1

Principles of Data Management

Lecture #7 (External Sorting)

Instructor: Mike Carey mjcarey@ics.uci.edu

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 2

Today’s Headlines

v  Project 2 (Relation Manager) involves finishing RBF and building RM on top §  Description now online (wiki) §  Partnerships working? (Just checking…)

v  Spring course options (a quick advertisement) §  CS 223 (Transaction Processing and Distributed

Data Management) §  CS 295? (“Mentored Projects in Big Data”)

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 3

Why Sort?

v  A classic problem in computer science! (J) v  Data sometimes requested in sorted order.

§  E.g., find students in decreasing gpa order.

v  Sorting is first step in bulk loading B+ tree index. v  Sorting useful for eliminating duplicate copies (i.e.,

SELECT DISTINCT) in a collection of records. (Why?) v  Sort-merge join algorithm involves sorting. v  Problem: sort 1 TB of data with 1 GB of RAM.

§  Q: Why not just use virtual memory?

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 4

2-Way Sort: Requires 3 Buffers

v  Pass 1: Read a page, sort it, write it out §  only one buffer page is used

v  Pass 2, 3, …, etc.: Read and merge pairs of runs §  three buffer pages are used:

Main memory buffers

INPUT 1

INPUT 2

OUTPUT

Disk Disk

... ...

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 5

Sorting N=2k Pages of Data v  Pass 0:

§  Read, sort, write à 2k 1-page runs (subfiles)

v  Pass 1: §  Read+merge 1-page pairs, write à 2k-1 2-page runs

v  Pass 2: §  Read+merge 2-page pairs, write à 2k-2 4-page runs

���

v  Pass k-1: §  Read+merge 2k-2 -page pairs, write à 2 2k-1 -page runs

v  Pass k: §  Read+merge 2k-1-page pairs, write à 1 2k –page result

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 6

Two-Way External Merge Sort v  Each pass we read + write

each page in file. v  N pages in the file => the

number of passes v  So total I/O cost is: v  Idea: Divide and conquer:

sort subfiles and merge v  Q: See any room to do

better, w/just 3 pages?

⎡ ⎤= +log2 1N

⎡ ⎤( )2 12N Nlog +

Input file

1-page runs

2-page runs

4-page runs

8-page runs

PASS 0

PASS 1

PASS 2

PASS 3

9

3,4 6,2 9,4 8,7 5,6 3,1 2

3,4 5,6 2,6 4,9 7,8 1,3 2

2,3 4,6

4,7 8,9

1,3 5,6 2

2,3 4,4 6,7 8,9

1,2 3,5 6

1,2 2,3 3,4 4,5 6,6 7,8

(k=3)

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 7

General External Merge Sort

v  To sort a file with N pages using B buffer pages: §  Pass 0: use B buffer pages. Produce sorted runs of B

pages each. §  Pass 2, …, etc.: merge B-1 runs.

⎡ ⎤N B/

B Main memory buffers

INPUT 1

INPUT B-1

OUTPUT

Disk Disk

INPUT 2

. . . . . . . . .

  More than 3 buffer pages. How can we utilize them?

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 8

Cost of External Merge Sort

v  Number of passes: v  Cost = 2N * (# of passes) v  E.g., with 5 buffer pages, to sort 108 page file:

§  Pass 0: = 22 sorted runs of 5 pages each (last run is only 3 pages)

§  Pass 1: = 6 sorted runs of 20 pages each (last run is only 8 pages)

§  Pass 2: 2 sorted runs, 80 pages and 28 pages §  Pass 3: Sorted file of 108 pages

⎡ ⎤⎡ ⎤1 1+ −log /B N B

⎡ ⎤108 5/

⎡ ⎤22 4/

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 9

Number of Passes of External Sort

N B=3 B=5 B=9 B=17 B=129 B=257100 7 4 3 2 1 11,000 10 5 4 3 2 210,000 13 7 5 4 2 2100,000 17 9 6 5 3 31,000,000 20 10 7 5 3 310,000,000 23 12 8 6 4 3100,000,000 26 14 9 7 4 41,000,000,000 30 15 10 8 5 4

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 10

B-2 buffers, organized as a pair of heaps (“current set” and “next set”) to give fast access to min-record ≥ max

20, “Wu, X.”, …

Output Buffer

(max = 36)

Input buffer

49, “Smith, J.”, …

62, “Jones, F.”, …

. . .

30, “Ridge, B.”, …

36, “Beck, J.”, …

70, “Kim, J.”, …

25, “Wagner, R.”, …

88, “Lee, M.”, …

A More Optimized Initial Pass: Replacement Sort

v  Used to read unsorted input file page by page

v  Next page read in when all of this page’s records have been added to one of the heaps

v  Used to write pages of current output run

v  Flushed when full (same run) or when all records in current set are < max (new run)

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 11

Replacement Sort (cont.)

v  Fact: average run length will now be ~2B §  The “snowplow” analogy

v  Worst-Case: §  What is min length of a run? §  How does this arise?

v  Best-Case: §  What is max length of a run? §  How does this arise?

v  Quicksort is faster for CPU, but ... §  (Also simpler for memory management!)

B

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 12

I/O for External Merge Sort

v  … longer runs often means fewer passes! v  So far, we’ve done I/O a page at a time v  Could read a block of pages sequentially!

§  Goal is to reduce the number of seeks.

v  Suggests we should make each buffer (input/output) be a block of pages. §  But this will reduce fan-out during merge passes! §  In practice, most files still sorted in 2-3 passes.

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 13

Number of Passes of Optimized Sort

N B=1,000 B=5,000 B=10,000100 1 1 11,000 1 1 110,000 2 2 1100,000 3 2 21,000,000 3 2 210,000,000 4 3 3100,000,000 5 3 31,000,000,000 5 4 3

  Block size = 32, initial pass yields runs of size ~2B.

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 14

Double Buffering v  To reduce wait time for I/O request to

complete, can prefetch into `shadow block’. §  Potentially, more passes; in practice, most files still

sorted in 2-3 passes.

OUTPUT

OUTPUT'

Disk Disk

INPUT 1

INPUT k

INPUT 2

INPUT 1'

INPUT 2'

INPUT k'

block size b

B main memory buffers, k-way merge

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 15

Sorting Records! (Mid-1990’s)

v  Sorting became a blood sport! §  Parallel sorting became the name of the game ...

v  Datamation: Sort 1M records of size 100 bytes §  Typical DBMS: 15 minutes §  World record: 3.5 seconds

•  12-CPU SGI machine, 96 disks, 2GB of RAM

v  New benchmarks proposed: §  Minute Sort: How many can you sort in 1 minute? §  Dollar Sort: How many can you sort for $1.00? ($0.01)

v  Today: http://sortbenchmark.org/

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 16

Sort Benchmarks Today

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 17

Sorting Summary

v  External sorting is important; DBMS may dedicate part of buffer pool for sorting!

v  External merge sort minimizes disk I/O cost: §  Pass 0: Produce sorted runs of size B (# buffer pages),

or of size B-1 if we are handling variable-length records. Passes > 0: Merge runs (until just one run is produced).

§  # of runs merged at a time depends on B and block size. §  Larger block size means less I/O cost per page of data. §  Larger block size means fewer runs merged per step. §  In practice, # of passes needed rarely more than 2 or 3.

Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 18

Sorting Summary, cont.

v  Choice of in-memory sort algorithm used in the initial sort pass, a.k.a. Pass 0, matters: §  Quicksort: Quick, but run lengths ≤ memory. §  Replacement sort: Slower (2x CPU) and messier

to implement, but can get 2x longer runs if used. Good to know about, but not commonly used in today’s DBMSs.

v  The best sorts today are amazingly speedy: §  Despite 40+ years of research, still improving!