Asymptotic Notation, Recurrences · Asymptotic growth rates Let g : N ! R . O -notation: O (g ) is...

Asymptotic Notation, Recurrences

Mary Cryan

24th September, 2013

ADS (2013/14) – Lecture 2 – slide 1

Asymptotic growth rates

Let g : N→ R.

O-notation: O(g) is the set of all functions f : N→ R for which there areconstants c > 0 and n0 ≥ 0 such that

0 ≤ f (n) ≤ c · g(n), for all n ≥ n0.

“Rate of change of f (n) is at most that of g(n)”

Ω-notation: Ω(g) is the set of all functions f : N→ R for which there areconstants c > 0 and n0 ≥ 0 such that

0 ≤ c · g(n) ≤ f (n), for all n ≥ n0.

“Rate of change of f (n) is at least that of g(n)”

Θ-notation: Θ(g) is the set of all functions f : N→ R for which there areconstants c1, c2 > 0 and n0 ≥ 0 such that

0 ≤ c1 · g(n) ≤ f (n) ≤ c2 · g(n), for all n ≥ n0.

“Rate of change of f (n) and g(n) are about the same”

Examples

I Let f (n) = 0.01 · n2 and g(n) = n. Then g = O(f ).

I Let f (n) = ln(n) and g(n) = n. Then g = Ω(f ).

I Let f (n) = 10n + ln(n) and g(n) = n. Then g = Θ(n).

Sometimes O(. . .) appears within a formula, rather than simplyforming the right hand side of an equation. We make sense ofthis by thinking of O(. . .) as standing for some anonymous (butfixed) function from the set of the same name.For example, h(n) = 2O(n) means ∃c > 0, n0 ∈ N such that

h(n) ≤ 2cn for all n > n0.

Consequences

Suppose f (n) = O(g(n)) AND g(n) = O(f (n)). What can we say?

What if f (n) = O(g(n)) AND f (n) = Ω(g(n))?

Various consequences of the above conventions:

Θ(n)×Θ(n2) = Θ(n3),

Θ(n) +Θ(n2) = Θ(n2),

Θ(n) +Θ(n) = Θ(n).

Reminder of InsertionSort

Algorithm Insertion-Sort(A)

1. for j ← 2 to length[A]

2. do key ← A[j ]3. Insert A[j ] into the sorted sequence A[1 . . . j − 1]

4. i ← j − 15. while i > 0 and A[i ] > key6. do A[i + 1]← A[i ]7. i ← i − 18. A[i + 1]← key

Array A is indexed from j = 1 to n = length[A] (different from Java).

running-time of InsertionSort

I The for-loop on line 1 is iterated n − 1 times

I For each execution of the for, the while does ≤ j iterations;

I Each of the comparisons/assignments requires only O(1) basic steps;

I Therefore the total number of steps (=time) is at most

n∑j=1

j = O(1)n(n + 1)

2= O(n2).

I This is essentially tight - sorting the list n, n − 1, n − 1, . . . , 3, 2, 1takes Ω(n2) time.

reminder of MergeSort

Input: A list A of natural numbers, p, r : 1 ≤ p ≤ r ≤ n.Output: A sorted (increasing order) permutation of A[p . . . r ].

Algorithm Merge-Sort(A, p, r)

1. if p < r then2. q ← bp+r

2 c3. Merge-Sort(A, p, q)

4. Merge-Sort(A, q + 1, r)5. Merge(A, p, q, r)

reminder of Merge

Input: A list A of whole numbers, 3 indices p, q, r : 1 ≤ p ≤ q < r , whereA[p . . . q] and A[q + 1 . . . r ] are individually sorted.Output: A sorted (increasing order) permutation of A[p . . . q].

(i) Define an output array B of length r − p + 1.(ii) Perform the merge, consisting of

I between minq − p + 1, r − q and r − p + 1 key comparisons, andbetween 2 minr − q, q − p + 1, r − p + 1 and 2(r − p + 1) indexcomparisons.

I exactly r − p + 1 copy operations (each key copied to B),I 2(r − p + 1) updates of the indices i , j , k as we run through A[p . . . q],

A[q + 1 . . . r ] and B[p . . . r ].I Another r − p + 1 copying operations copying B back into A.

The running-time of Merge is linear in the length of the array, specifically:

4(r−p+1)+3 minq−p+1, r−q, r−p+1 ≤ TMerge(p, q, r) ≤ 7(r−p+1)

Taking n = r − p + 1, we have

4n ≤ TMerge(n) ≤ 7n

Running-time of MergeSort

Putn = r − p + 1.

Running time TMS(n) satisfies:

TMS(n) =

Θ(1) if n = 1,

TMS(dn/2e) + TMS(bn/2c) +Θ(n) if n > 1.

The Θ(n) is from analysis of Merge on the previous slide. Analysis ofMergeSort gives bn+1

2 c and dn−12 e as the subarray sizes - these are

same as bn2c and dn2e.

Solving recurrences

Methods for deriving/verifying solutions to recurrences:

Induction Guess the solution and verify by induction on n.

Lovely if your recurrence is “NICE” enough that you canguess-and-verify. Rare.

Unfold and sum “Unfold” the recurrence by iterated substitution on the“neat” values of n (often power of 2 case). At some pointa pattern emerges. The “solution” is obtained byevaluating a sum that arises from the pattern.Since the pattern is just a guess for special n, the methodis rigorous only if we verify the solution (e.g., by a directinduction proof, or by the Master Theorem).

Often the only way to do the PROOF neatly is to RELATEto “neat” values of n . . . sometimes powers-of-2

“Master Theorem” Match the recurrence against a template. Read offthe solution from the Master Theorem.

Upper bound for MergeSort

Proof by “first principles”When working from first principles, need to replace “extra work” terms(Θ(n) for MergeSort) by terms with explicit constants.So check slide 8.

TMS(n) ≤

1 if n = 1,

TMS(dn/2e) + TMS(bn/2c) + 7n if n > 1.(1)

Unfold-and-sum will give a “guess” for the upper bound:

TMS(n) ≤ 7n lg(n) + n.

TMS(n) ≤

1 if n = 1,

TMS(n) ≤

1 if n = 1,

Upper bound for MergeSort (n a power-of-2)

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

An equality version of (1). Clearly TMS(n) ≤ T ′MS(n) for n ∈ N.

claim (powers of 2): T ′MS(n) ≤ 7n lg(n) + n if n = 2k , for some k ∈ N

Proof (for powers of 2):Base case k = 0: direct from algorithm (1 comparison).Induction Hypothesis (IH): Upper bound holds for n = 2k−1.Induction Step: Now consider n = 2k and apply the recurrence:

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

= 7n(lg(n/2) + 1) + n = 7n lg(n) + n (by lg rules),

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

Proof (for powers of 2):

Base case k = 0: direct from algorithm (1 comparison).Induction Hypothesis (IH): Upper bound holds for n = 2k−1.Induction Step: Now consider n = 2k and apply the recurrence:

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

Proof (for powers of 2):Base case k = 0: direct from algorithm (1 comparison).

Induction Hypothesis (IH): Upper bound holds for n = 2k−1.Induction Step: Now consider n = 2k and apply the recurrence:

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

Proof (for powers of 2):Base case k = 0: direct from algorithm (1 comparison).Induction Hypothesis (IH): Upper bound holds for n = 2k−1.

Induction Step: Now consider n = 2k and apply the recurrence:

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

T ′MS(n) =

1 if n = 1,

T ′MS(dn/2e) + T ′

MS(bn/2c) + 7n if n > 1.(2)

T ′MS(n) = T ′

MS(d2k−1e) + T ′MS(b2k−1c) + 7n

= 2 · T ′MS(2k−1) + 7n

= 2 · 2k−1(7 lg(2k−1) + 1) + 7n ((IH))

= n · 7 lg(n/2) + n + 7n

AS REQUIRED.

Upper bound for MergeSort (general n)

Note: Argument above WON’T imply TMS(n) ′ ≤ 7n lg(n) + n for general n.

Non power-of-2 bound: We have n ∈ N, n not an power of 2.

STEP 1: Prove that TMS(n) ′ is monotone increasing.

Induction Hypothesis (IH):If n ∈ N then T ′MS(n) < T ′

MS(m) for all n < m.

Base Case (n = 1): T ′MS(1) = 1. For m > 2, T ′

MS(m) ≥ m ≥ 2 > T ′MS(1).

Induction Step (n): Suppose true for all k ∈ N, k < n. Consider n. By n ≥ 2,the recurrence holds for n, so

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

For m > n, apply the recurrence to get

TMS(m) = T ′MS(dm/2e) + T ′

MS(bm/2c) + 7m.

By b n2c, d n

2e < n (need strict <), we can apply the (IH) with k = b n2c or k = d n

2e.By b n

2c ≤ bm2 c and d n

2e ≤ dm2 e our (IH) tells us that 1st two terms for T ′

MS(m)are ≥ than the partner terms for T ′

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

So by Induction, STEP 1 is proved.

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

By b n2c ≤ bm

2 c and d n2e ≤ dm

2 e our (IH) tells us that 1st two terms for T ′MS(m)

are ≥ than the partner terms for T ′MS(n).

But also 7n < 7m . . . ⇒ T ′MS(n) < T ′

MS(m).So by Induction, STEP 1 is proved.

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).

But also 7n < 7m . . . ⇒ T ′MS(n) < T ′

MS(m).So by Induction, STEP 1 is proved.

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

MS(m) ≥ m ≥ 2 > T ′MS(1).

T ′MS(n) = T ′

MS(dn/2e) + T ′MS(bn/2c) + 7n. (3)

MS(bm/2c) + 7m.

By b n2c, d n

2e.By b n

MS(n).But also 7n < 7m . . . ⇒ T ′

MS(n) < T ′MS(m).

Upper bound for MergeSort (general n) cont’d.

STEP 2: Choose a “power of 2” to relate to n.

I Want an upper bound, so need a power of 2 greater than n.

I So define n = 2dlg(n)e (this will be “m”).

I We know n ≤ n but n < 2n.

I Monotonicity property from STEP 1 tells us T ′MS(n) ≤ T ′

I Proof of Upper bound for powers of 2 tells us T ′MS(n) ≤ 7n lg(n) + n.

I By n < 2n, we get

T ′MS(n) ≤ T ′

MS(n) = 7n(lg(n)) + n < 7(2n) lg(2n) + 2n = 14n lg(n) + 16n.

So for any n ∈ N we have T ′MS(n) ≤ 14n lg(n) + 16n.

The “truth” is quite a bit better, but not well-suited to an inductive proof.

I By n < 2n, we get

T ′MS(n) ≤ T ′

I By n < 2n, we get

T ′MS(n) ≤ T ′

I By n < 2n, we get

T ′MS(n) ≤ T ′

I By n < 2n, we get

T ′MS(n) ≤ T ′

I By n < 2n, we get

T ′MS(n) ≤ T ′

I By n < 2n, we get

T ′MS(n) ≤ T ′

I By n < 2n, we get

T ′MS(n) ≤ T ′

I By n < 2n, we get

T ′MS(n) ≤ T ′

Notes on the proof

Our “first principles” proof is essentially a direct proof of a sub-case ofthe Master Theorem.

For “first principles” O(·) proofs where b·c, d·e is involved, there are a fewcommon steps:

I Consider an equality version T ′(·) of the recurrence T (·) such thatT (n) ≤ T ′(n) holds for all n ∈ N.

I prove the result for T ′ for the “NEAT” case (power-of-2 here, butwould be power-of-d if bn/dc, dn/de was involved)

I STEP 1: Prove T ′(n) is monotonically increasing with n forgeneral n.

I STEP 2: Consider the closest power-of-d greater than n, say n, for anon-neat n ∈ N. Then exploit T (n) ≤ T ′(n) (by definition),T ′(n) ≤ T ′(n) (by STEP 1), and then substitute in the provenupper bound for T ′(n) (because n is “NEAT”).

Notes on the proof

Our “first principles” proof is essentially a direct proof of a sub-case ofthe Master Theorem.

For “first principles” O(·) proofs where b·c, d·e is involved, there are a fewcommon steps:

I Consider an equality version T ′(·) of the recurrence T (·) such thatT (n) ≤ T ′(n) holds for all n ∈ N.

I prove the result for T ′ for the “NEAT” case (power-of-2 here, butwould be power-of-d if bn/dc, dn/de was involved)

I STEP 1: Prove T ′(n) is monotonically increasing with n forgeneral n.

I STEP 2: Consider the closest power-of-d greater than n, say n, for anon-neat n ∈ N. Then exploit T (n) ≤ T ′(n) (by definition),T ′(n) ≤ T ′(n) (by STEP 1), and then substitute in the provenupper bound for T ′(n) (because n is “NEAT”).

Reading and Working

Reading Assignment

Inf2B ADS Lecture Notes 2 and 8.

[CLRS] Sections 2.1, 2.2 and 2.3 (of 3rd or 2nd edition). Also Section 3.1(omitting the bits on the little-o and little-ω notation at the end).

(all this material should be familiar from Inf2B and your math classes)

Tutorials (will start week 3, on Tuesday)

Fill the Doodle poll TODAY linked-to from the ADS course webpage (ifyou haven’t yet)

Asymptotic Notation, Recurrences · Asymptotic growth rates Let g : N ! R . O -notation: O (g ) is...

Documents

Asymptotic Notation (O, Ω, ) Describes the behavior of the time or space complexity for large instance characteristics Common asymptotic functions

Runtime Efficiency and Asymptotic Analysisinformationtechnology.pk/wp-content/uploads/2015/04/Lecture13.pdf · asymptotic upper bound big-O notation • An asymptotic bound, as function

Asymptotic Analysis · (f(n)): Big Theta -asymptotic tight bound The “big-Omega” -Notation asymptotic lower bound f(n) is (g(n)) if there exists constants c and n 0, s.t. c g(n)

SOFTENG 211 Asymptotic notation (2004)...SOFTENG 211 Asymptotic notation (2004) - 3 - improvement that gets more and more marked as the size of the instance increases. For this reason,

Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II

CSE 2010: Algorithms and Data Structuresmy.fit.edu/.../cse2010_notation/02-algorihtms_complexity.pdfAsymptotic Notation 3.1 Asymptotic notation 45 (a) (b) (c) n n n n 0 n 0 n 0 f.n/D

5 Asymptotic Notation - TUM...5 Asymptotic Notation We are usually not interested in exact running times, but only in an asymptotic classiﬁcation of the running time, that ignores

CS 2133: Data Structures Mathematics Review and Asymptotic Notation

Complexity Analysis: Asymptotic Analysis. Recall What is the measurement of algorithm? How to compare two algorithms? Definition of Asymptotic Notation

Ch7: Asymptotic Notations - Naresuan University · Ch7: Asymptotic Notations 305233, 305234 Algorithm Analysis and Design JirapornPooksook NaresuanUniversity. What is asymptotic notation?

DAA ADEG Asymptotic Notation Minggu 4 11

Asymptotic Notation, Recurrences: Substitution, …sourav/Lecture-02.pdfAsymptotic Notation, Recurrences: Substitution, Iteration, Master Method Lecture 2 L2.2 Solving recurrences

13 February 2016 Asymptotic Notation, Review of Functions & Summations

UNIT - I ALGORITHM ANALYSIS 1.1 Introduction · notation – Removing condition from the conditional asymptotic notation – Properties of Big-oh notation – Recurrence equations

Advanced algorithms asymptotic notation, graphs and their … · 2011-10-05 · Advanced algorithms asymptotic notation, graphs and their representation in computers Jiří Vyskočil,

Growth of Functions and Aymptotic Notation · Asymptotic Notation 14 Asymptotic Bounds and Algorithms • In all of the examples so far, we have assumed we knew the exact running

geekztrainerblog.files.wordpress.com - I ALGORITHM ANALYSIS Algorithm analysis – Time space tradeoff – Asymptotic notations – Conditional asymptotic notation – Removing condition

SNU IDB Lab. Ch3. Asymptotic Notation © copyright 2006 SNU IDB Lab

Asymptotic moment boundedness of the numerical solutions ...personal.strath.ac.uk/x.mao/talks/numerical_bd.pdf · Notation Outline 1 Introduction Some facts Notation 2 Asymptotic

DAA JK - Asymptotic Notation - Minggu 4_16