1
Design and Analysis of Algorithms
演算法設計與分析
Lecture 2March 4, 2010洪國寶
2
Homework # 1
1. 2.1-3 (p. 22)2. 2.2-3 (p. 29)3. 2.3-6 (p. 37)4. 2-2 (p. 40)
Due March 11, 2010
3
Outline
• Review• Introduction to design• Recurrences
4
Course information (5/5)
• Grading (Tentative)Homework 25%
(You may collaborate when solving the homework, however when writing up the solutions you must do so on your own. Handwritten only.)
Midterm exam 30% (Open book and notes)Final exam 30% (Open book and notes)Class participation 15%
5
Review: Definition of algorithms
• An algorithm is any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output.
• The goal of this course is to explore some paradigms of designing efficient and correctalgorithms for computational problems.
6
Motivation (6/6)
• The importance of efficient algorithms– Total system performance depends on choosing
efficient algorithms as much as on choosing fast hardware.
– A faster algorithm running on a slower computer will always win for sufficiently large instances.
7
Review: Correct algorithms
• An algorithm is said to be correct if for every input instance, it halts with correct output.
• Correctness is not obvious.
8
Review: well-specified computational problem
• Example of a well-specified computational problem: sorting problem (p. 5)
Input: sequence ⟨a1, a2, …, an⟩ of numbers.Output: permutation ⟨a'1, a'2, …, a'n⟩ such
that a'1 ≤ a'2 ≤ … ≤ a'n .
9
Insertion sort
10
Review: Loop invariant
• A Method for Proving the Correctness of Iterative Algorithms (that is, algorithms contain loop structures)
• Similar to mathematical induction
Loop Invariants
• A loop invariant states a condition that is true immediately before every iteration of a loop.• The invariant should be true also immediately after the last iteration.
Time (# of computation steps)
It. 1 It. 2 It. 3 It. 4 · · · It. k
Loop Invariants• We have to show:–Initialization:
The loop invariant holds before the first iteration
–Maintenance:If the loop invariant holds before the i-th iteration, it holds
before the (i+1)-st iteration.
–Termination:If the loop invariant holds after the last iteration, the
algorithm does what we want.
A Loop Invariant for Insertion Sort
• Invariant: Before the iteration of the outer loop where j = k + 1, array A[1..k] stores the k elements initially in A[1..k] in sorted order. Array A[k+1..n]is unaltered.
14
Review: Algorithm analysis
• Analysis: predict the cost of an algorithm in terms of resources and performance
• Resource: time, space, bandwidth, random bits, etc.
15
Review: Algorithm analysis• Machine Model
Generic Random Access Machine (RAM)Executes operations sequentiallyEach memory access takes exactly 1 stepSet of primitive operations:
-- Arithmetic. Logical, Comparisons, Function calls
Loop and subroutine calls are not primitive ops.• Simplifying assumption: all ops cost 1 unit
16
Review: Algorithm analysis
• Under the RAM model, the running time of an algorithm depends mainly on input size.
• Input size: depends on the problem- number of items (sorting problem)- number of bits (number theoretic problems)- number of edges and vertices (graph problems)
17
Review: Algorithm analysis
• Reasons for worst case analysis- upper bound- worst case occurs fairly often- average case is often roughly as bad as worst case
18
Notations
f(n) >> c g(n)f(n) = ω(g(n))Little Omega
f(n) << c g(n)f(n) = o(g(n))Little Oh
f(n) ≥ c g(n)f(n) = Ω(g(n))Omega
f(n) ≤ c g(n)f(n) = O(g(n))BigOh
f(n) ≈ c g(n)f(n) = θ(g(n))Theta
19
Upper Bound Notation• We say InsertionSort’s run time is O(n2)
– Properly we should say run time is in O(n2)– Read O as “Big-O” (you’ll also hear it as
“order”)• In general a function
– f(n) is O(g(n)) if there exist positive constants cand n0 such that f(n) ≤ c ⋅ g(n) for all n ≥ n0
• Formally– O(g(n)) = f(n): ∃ positive constants c and n0
such that f(n) ≤ c ⋅ g(n) ∀ n ≥ n0
20
Asymptotic Tight Bound
• A function f(n) is Θ(g(n)) if ∃ positive constants c1, c2, and n0 such that
c1 g(n) ≤ f(n) ≤ c2 g(n) ∀ n ≥ n0
• Theorem– f(n) is Θ(g(n)) iff f(n) is both O(g(n)) and
Ω(g(n))
21
Introduction to Analysis
• Asymptotic notationsExamples: use blackboard
22
Introduction to Analysis• Asymptotic notation properties: (pp. 48-50)
For the following, assume that f(n) and g(n) are asymptotically positive. – Transitivity:
f (n) = Θ(g(n)) and g(n) = Θ(h(n)) imply f (n) = Θ(h(n)), f (n) = O(g(n)) and g(n) = O(h(n)) imply f (n) = O(h(n)), f (n) = Ω(g(n)) and g(n) = Ω(h(n)) imply f (n) = Ω(h(n)),
– Reflexivity: f (n) = Θ(f(n)), f(n) = O(f(n)), f(n) = Ω(f(n)), – Symmetry: f(n) = Θ(g(n)) if and only if g(n) = Θ(f(n)).
• Although any two real numbers can be compared, not all functions are asymptotically comparable.
– That is, for two functions (n) and g(n), it may be the case that neither f (n) = O(g(n)) nor f(n) = Ω(g(n)) holds.
Q: Is Θ an equivalence relation?How about O and Ω?
23
Outline
• Review• Introduction to design• Recurrences
24
Introduction to Design (1/)
• There are many ways to design algorithms- induction (incremental approach):insertion sort
- divide and conquer:merge sort
25
Introduction to Design (2/)
• divide and conquer:1. divide the problem into a number of
subproblems2. conquer the subprobelms (recursively)3. combine the solutions to the subproblems
into the solution of the original problem
26
Introduction to Design (3/)
• merge sort1. divide the sequence into two
subsequences of n/2 elements each2. sort the two subsequences recursively
using merge sort3. merge the two sorted subsequences
27
Introduction to Design (4/)
• merge sortTo sort the two subsequences recursivelyusing merge sort, we need to specify the indexes of the subsequences.
Merge-Sort (A, p, r)
28
Merge-Sort (A, p, r)INPUT: a sequence of n numbers stored in
array AOUTPUT: an ordered sequence of n numbers
1. if p < r2. then q ← [(p+r)/2]3. Merge-Sort (A, p, q)4. Merge-Sort (A, q+1, r)5. Merge (A, p, q, r)
divide
conquer
combine
29
Merge (A, p, q, r)INPUT: two sorted sequences A[p..q], A[q+1..r]OUTPUT: an ordered sequence A[p..r]
1. Create new arrays L and R2. Copy A[p..q] to L and copy A[q+1..r] to R3. Copy the elements in L and R back to A[p,r]
in order
30
.
31
The Merge Procedure: Repeatedly copy the smallest
element from L and R to A:10 i← 111 j← 112 for k = p to r13 do if L[i] ≤ R[j]14 then A[k] ← L[i]15 i← i + 116 else A[k] ← R[j]17 j← j + 1
Copy A[p..q] and A[q + 1..r] to temporary arrays L and R:
1 n1← q – p + r2 n2← r – q3 Create array L and R4 for i = 1 to n15 do L[i] ← A[p + i – 1]6 for j = 1 to n27 do R[j] ← A[q + j]
Create two artificial endmarkers that are never copied to
A:8 L[n1 + 1] ←∞9 R[n2 + 1] ←∞ }Why?
Merge ProcedureRunning Time
Lemma: Procedure Merge takes at most cn time, wheren = r – p + 1 and c is an appropriate constant.
Merge ProcedureCorrectness
Lemma:If arrays A[p..q] and A[q + 1..r] are sorted before invoking procedure Merge,
array A[p..r] is sorted after procedure Merge finishes.
Q: What is the most appropriate loop invariant for the merge procedure?
35Let’s take a closer look at Figure 2.3.
Merge ProcedureCorrectness
Lemma: If arrays A[p..q] and A[q + 1..r] are sorted beforeinvoking procedure Merge, array A[p..r] is sorted after procedure Merge finishes.
Loop invariant: Before every iteration of the last for-loop, array A[p..k – 1] stores the k – p smallest elements in L and R. Moreover, L[i] and R[j] are the smallest elements of their arrays that have not been copied back into A
37
Correctness of the Merge procedure
• Use blackboard
38
Introduction to Design (17/)
• Merge sort(A,p,r)1. if p < r2. then q ← [(p+r)/2]3. Merge-Sort (A, p, q)4. Merge-Sort (A, q+1, r)5. Merge (A, p, q, r)• To sort a sequence of n numbers stored in
array A, run merge-sort(A,1,n).Example (Figure 2.4)
39
Merge sort(A,1,8)
Merge sort(A,1,4) Merge sort(A,5,8)
Merge sort(A,1,2)
p=r
40
Introduction to Design (19/)
• merge sort (remaining issues)- correctness- running time- compared to insertion sort
What are the advantages of merge sort?What are the advantages of insertion sort?
41
Correctness of merge sort
• By induction on length s of a segment of the array A wheres = r – p + 1
Merge-sort(A,p,r)1. if p < r2. then q ← [(p+r)/2]3. Merge-Sort (A, p, q)4. Merge-Sort (A, q+1, r)5. Merge (A, p, q, r)
42
Correctness of merge sort
• Use blackboard
43
Merge-SortTo sort a sequence of n numbers stored in array A, run merge-sort(A, 1, n).
if 1 < nthen q ← [(1+n)/2]
Merge-Sort (A, 1, q)Merge-Sort (A, q+1, n)Merge (A, 1, q, n)
How do we find the running time?
44
Analysis of Merge Sort• Divide: computing the middle takes Θ(1)• Conquer: solving 2 subproblems takes 2T(n/2)• Combine: merging n elements takes Θ(n)• Total:
T(n) = Θ(1) if n = 1T(n) = 2T(n/2) + Θ(n) if n > 1
⇒ T(n) = Θ(n lg n) (Chapter 4)
Sorting (Recap)
• We have studied two sorting algorithms:– Insertion Sort takes c1 n2 time.– Merge Sort takes c2 n lg n time.• Even if c2≫ c1, Merge Sort will ultimately outperform Insertion Sort– Example (use blackboard)
Sort n = 106 numbers, c1 = 2, and c2 = 50
46
Motivation (6/6)
• The importance of efficient algorithms– Total system performance depends on choosing
efficient algorithms as much as on choosing fast hardware.
– A faster algorithm running on a slower computer will always win for sufficiently large instances.
47
Sorting (Recap)
• Two different paradigms:– Insertion Sort is an incremental algorithm:
• Maintain a solution for part of the input• Augment/correct the solution as we take more and more input
elements into account.– Merge Sort uses the divide-and-conquer paradigm:
• Divide the problem into two (or more) subproblems of approximately equal size/complexity
• Solve (conquer) the subproblems recursively• Combine the solutions for the subproblems into a solution for
the whole problem
48
Outline
• Review• Introduction to analysis (cont.)• Introduction to design• Recurrences
49
Recurrence Relations• Describe functions in terms of their values on
smaller inputs• Solution Methods (Chapter 4)
– Substitution Method– Iteration Method– Master Method
• Arise from Divide and ConquerT(n) = Θ(1) if n ≤ cT(n) = a T(n/b) + D(n)+C(n) otherwise
50
Recurrence Relations• Remarks
- T(n) is defined when n is an integer (the size of the input is always an integer)
- Generally, T(n) = O(1) for sufficiently small n
- When we state and solve recurrences, we often omit floors, ceilings, and boundary conditions.
51
Recurrence Relations• Describe functions in terms of their values on
smaller inputs• Solution Methods (Chapter 4)
– Substitution Method– Iteration Method– Master Method
52
Substitution Method• Guess the form of the solution, then use
mathematical induction to show that it works
• Works well when the solution is easy to guess
• No general way to guess the correct solution
53
An ExampleTo Solve: T(n) = 3T(n/3) + n • Guess: T(n) = O(n lg n) ≤ c n lg n • Calculate:
T(n) ≤ 3c n/3 lg n/3 + n≤ c n lg (n/3) + n = c n lg n - c n lg3 + n= c n lg n - n (c lg 3 - 1)≤ c n lg n
(The last step is true for c ≥ 1 / lg3.)
Q: Is this mathematical induction?
54
Making a Good Guess• Guess a similar solution to one that you have seen
before– T(n) = 3T(n/3 + 5) + n and T(n) = 3T(n/3) + n
When n is large, the difference between n/3 and (n/3 + 5) is insignificant
• Another way is to prove loose upper and lower bounds on the recurrence and then reduce the range of uncertainty.– Start with T(n) = Ω(n) & T(n) = O(n2) ⇒ T(n) = Θ (n log n)
55
Subtleties• When the math doesn’t quite work out in the induction,
try to adjust your guess with a lower-order term. For example:
– We guess T(n) = O(n) ≤ c n for T(n) = 3T(n/3)+ 4, but we have T(n) ≤ 3c n/3 + 4 = c n + 4
56
Subtleties• When the math doesn’t quite work out in the induction,
try to adjust your guess with a lower-order term. For example:
– We guess T(n) = O(n) ≤ c n for T(n) = 3T(n/3)+ 4, but we have T(n) ≤ 3c n/3 + 4 = c n + 4
– New guess is T(n) ≤ c n - b, where b ≥ 0T(n) ≤ 3(c n/3 - b)+4 = c n - 3b + 4 = c n - b - (2b-4)Therefore, T(n) ≤ c n - b, if 2b - 4 ≥ 0 or if b ≥ 2
57
Changing Variables• Use algebraic manipulation to turn an unknown
recurrence into one similar to what you have seen before. – Consider T(n) = 2T(n1/2) + lg n – Rename m = lg n (that is, n=2m) and we have
T(2m) = 2T(2m/2) + m – Set S(m) = T(2m) and we have
S(m) = 2S(m/2) + m ⇒ S(m) = O(m lg m)– Changing back from S(m) to T(n), we have
T(n) = T(2m) = S(m) = O(m lg m) = O(lg n lg lg n)
58
Avoiding Pitfalls• Be careful not to misuse asymptotic notation. For
example:
– We can falsely prove T(n) = O(n) by guessing T(n) ≤ c nfor T(n) = 2T(n/2) + n
T(n) ≤ 2c n/2 + n≤ c n + n = O(n) ⇐ Wrong!
– The error is that we haven’t proved T(n) ≤ c n
59
Recurrence Relations• Describe functions in terms of their values on
smaller inputs• Solution Methods (Chapter 4)
– Substitution Method– Iteration Method– Master Method
60
Iteration Method• Expand (iterate) the recurrence and express it as a
summation of terms dependent only on n and the initial conditions
• The key is to focus on 2 parameters– the number of times the recurrence needs to be iterated to
reach the boundary condition– the sum of terms arising from each level of the iteration
process• Techniques for evaluating summations can then be used
to provide bounds on solution.
61
An Example• Solve: T(n) = 3T(n/4) + n
T(n) = n + 3T(n/4)= n + 3[ n/4 + 3T(n/16) ]= n + 3[n/4] + 9T(n/16)= n + 3[n/4] + 9 [n/16] + 27T(n/64)
T(n) ≤ n + 3n/4 + 9n/16 + 27n/64 + … + 3log4 nΘ(1)≤ n ∑ (3/4)i + Θ(nlog43)= 4n+ o(n)= O(n)
62
Another Example• Insertion sort (recursive version)
T(n) = T(n-1) + n• Solve: T(n) = T(n-1) + n
T(n) = T(n-1) + n = (T(n-2) + n-1)+ n = T(n-3) + n -2 + (n-1) + n = 1+2+3+ … + n
= n(n+1)/2= O(n2)
Insertion-sort(A,n)if n=1 then return A[n]
else Insertion-sort(A,n-1)Insert(A, n-1, A[n])
63
Recursion Trees
• Keep track of the time spent on the subproblems of a divide and conquer algorithm
• A convenient way to visualize what happens when a recursion is iterated
• Help organize the algebraic bookkeeping necessary to solve the recurrence
64
Merge Sortn
n/2
n/4 n/4
n/2
n/4 n/4
Running times to merge two sublists
Running time to sort the left sublist
65
Running Timen n=n
n/2
n/4 n/4
n/2
n/4 n/4
2x(n/2) = n
lg n4x(n/4) = n
Total: n lg n
66
Recursion Trees and Recurrences• Useful even when a specific algorithm is not specified
– For T(n) = 2T(n/2) + n2, we have
67
Recursion Trees
T(n) = Θ(n2)
68
Recursion Trees• For T(n) = T(n/3) + T(2n/3) + n
T(n) = O(n lg n)
69
Merge sort revisited• In merge sort, we can divide the sequence into
two subsequences of n/3 and 2n/3 elements each.• Then we will result in the following recurrence
T(n) = T(n/3) + T(2n/3) + n.• The previous slide shows that the asymptotic
running time remains the same.
70
Recurrence Relations• Describe functions in terms of their values on
smaller inputs• Solution Methods (Chapter 4)
– Substitution Method– Iteration Method– Master Method
71
Master Method• Provides a “cookbook” method for solving
recurrences of the formT(n) = a T(n/b) + f(n)
• Assumptions:– a ≥ 1 and b ≥ 1 are constants
– f(n) is an asymptotically positive function
– T(n) is defined for nonnegative integers– We interpret n/b to mean either n/b or n/b
72
The Master Theorem• With the recurrence T(n) = a T(n/b) + f(n) as in the
previous slide, T(n) can be bounded asymptotically as follows:
1. If f(n)=O(nlogba-ε) for some constant ε > 0, then T(n)= Θ(nlogba).
2. If f(n) = Θ(nlogba), then T(n) = Θ(nlogba lg n).3. If f(n) = Ω ( nlogba+ε ) for some constant ε > 0,
and if a f(n/b) ≤ c f(n) for some constant c < 1and all sufficiently large n, then T(n)= Θ(f(n)).
73
Simplified Master TheoremLet a ≥ 1 and b > 1 be constants and let T(n) be the
recurrenceT(n) = a T(n/b) + c nk
defined for n ≥ 0.1. If a > bk, then T(n) = Θ( nlogba ).2. If a = bk, then T(n) = Θ( nk lg n ).3. If a < bk, then T(n) = Θ( nk ).
74
Proof of the master theorem
75
Proof of the master theorem
• Use blackboard
76
The Master Theorem (Recap)
• Given: a divide and conquer algorithm– An algorithm that divides the problem of size n
into a subproblems, each of size n/b– Let the cost of each stage (i.e., the work to
divide the problem + combine solved subproblems) be described by the function f(n)
• Then, the Master Theorem gives us a cookbook for the algorithm’s running time.
77
The Master Theorem (Recap)
• if T(n) = aT(n/b) + f(n) then
( )
( )
( )
( )
( )
( )
<>
<Ω=
Θ=
=
Θ
Θ
Θ
=
+
−
10
largefor )()/( AND )(
)(
)(
)(
log)(
log
log
log
log
log
c
nncfbnafnnf
nnf
nOnf
nf
nn
n
nT
a
a
a
a
a
b
b
b
b
b
ε
ε
ε
78
Examples• T(n) = 16T(n/4) + n
– a = 16, b = 4, thus nlogba = nlog416 = Θ(n2)– f(n) = n = O(nlog416 - ε ) where ε = 1 ⇒ case 1.– Therefore, T(n) = Θ(nlogba ) = Θ(n2)
• T(n) = T(3n/7) + 1– a = 1, b=7/3, and nlogba = nlog 7/3 1 = n0 = 1– f(n) = 1 = Θ(nlogba) ⇒ case 2.– Therefore, T(n) = Θ(nlogba lg n) = Θ(lg n)
79
Examples (Cont.)• T(n) = 3T(n/4) + n lg n
– a = 3, b=4, thus nlogba = nlog43 = O(n0.793)– f(n) = n lg n = Ω(nlog43 + ε ) where ε ≈ 0.2 ⇒ case 3.– Therefore, T(n) = Θ(f(n)) = Θ(n lg n)
• T(n) = 2T(n/2) + n lg n– a = 2, b=2, f(n) = n lg n, and nlogba = nlog22 = n– f(n) is asymptotically larger than nlogba, but not
polynomially larger. The ratio lg n is asymptotically less than nε for any positive ε. Thus, the Master Theoremdoesn’t apply here.
80
Homework # 1
1. 2.1-3 (p. 22)2. 2.2-3 (p. 29)3. 2.3-6 (p. 37)4. 2-2 (p. 40)
Due March 11, 2010