goddard/Courses/CSCE310J Fundamentals of Algorithm Analysis Dr. Steve Goddard [email protected] CSCE 310J: Data Structures &

http://www.cse.unl.edu/~goddard/Courses/CSCE310J

Fundamentals of Algorithm Analysis

Dr. Steve [email protected]

CSCE 310J: Data Structures & Algorithms

Most of the slides in this lecture were originally created by

Jeff Edmonds

from York University

Relevant Mathematics

Jeff Edmonds

York University

Classifying FunctionsThe Time Complexity of an AlgorithmAdding Made EasyRecurrence Relations

Start With Some Math

Input Size

Tim

eClassifying Functions

f(i) = n(n)

Recurrence Relations

T(n) = a T(n/b) + f(n)

Adding Made Easy∑i=1 f(i).

Time Complexityt(n) = (n2)

Assumed Knowledge

• Logarithms and Exponentials

• Existential and Universal Quantifiers, g b Loves(b,g)

b g Loves(b,g)

Quantifiers: all, some

• “for all x” x P(x) is true iff P(x) is true for all x– universal quantifier (universe of discourse)

• “there exist x” x P(x) is true iff P(x) is true for some value of x– existential quantifier

x A(x) is logically equivalent to x(A(x)) x A(x) is logically equivalent to x(A(x)) x (A(x) B(x))

“For all x such that if A(x) holds then B(x) holds”

Classifying Functions

Giving an idea of how fast a function grows without going into too much detail.

Which are more alike?


Mammals



Dogs

Classifying Animals

Vertebrates

Birds

Mam

mals

Reptiles

Fish

Dogs

Giraffe


T(n) 10 100 1,000 10,000

log n 3 6 9 13 amoeba

n1/2 3 10 31 100 bird

10 100 1,000 10,000 human

n log n 30 600 9,000 130,000 my father

n2 100 10,000 106 108 elephantn3 1,000 106 109 1012 dinosaur2n 1,024 1030 10300 103000 the universe

Note: The universe contains approximately 1050 particles.

n


n1000 n2 2n


Polynomials

n1000 n2 2n


1000n2 3n2 2n3


Quadratic

1000n2 3n2 2n3

Classifying Functions?

Functions


Functions

Constant

Logarithm

ic

Poly L

ogarithmic

Polynom

ial

Exponential

Exp

Double E

xp

(log n)5 n5 25n5 log n5 2n5 25n

2

Classifying Functions?

Polynomial


Polynomial

Linear

Quadratic

Cubic

?

5n25n 5n3 5n4

Logarithmic

• log10n = # digits to write n

• log2n = # bits to write n = 3.32 log10n

• log(n1000) = 1000 log(n)

Differ only by a multiplicative constant.

Poly Logarithmic

(log n)5 = log5 n

Logarithmic << Polynomial

For sufficiently large n

log1000 n << n0.001

Linear << Quadratic


10000 n << 0.0001 n2

Polynomial << Exponential


n1000 << 20.001 n

Ordering Functions

Functions

Constant

Logarithm

ic

Poly L

ogarithmic

Polynom

ial

Exponential

Exp

Double E

xp

(log n)5 n5 25n5 log n5 2n5 25n

2

<< << << << << <<

<< << << << << <<

Which Functions are Constant?

• 5• 1,000,000,000,000• 0.0000000000001• -5• 0• 8 + sin(n)

YesYesYes

No

YesYes

Which Functions are “Constant”?

• 5• 1,000,000,000,000• 0.0000000000001• -5• 0• 8 + sin(n)

YesYesYesNo

NoYes

Lie in between

7

9

The running time of the algorithm is a “Constant” It does not depend significantly

on the size of the input.

Which Functions are Quadratic?

• n2

• … ?


• n2

• 0.001 n2

• 1000 n2

Some constant times n2.


• n2

• 0.001 n2

• 1000 n2

• 5n2 + 3n + 2log n


• n2

• 0.001 n2

• 1000 n2

• 5n2 + 3n + 2log n

Lie in between


• n2

• 0.001 n2

• 1000 n2

• 5n2 + 3n + 2log n

Ignore low-order termsIgnore multiplicative constants.

Ignore "small" values of n. Write θ(n2).

Which Functions are Polynomial?

• n5

• … ?


• nc

• n0.0001

• n10000

n to some constant power.


• nc

• n0.0001

• n10000

• 5n2 + 8n + 2log n • 5n2 log n • 5n2.5


• nc

• n0.0001

• n10000

• 5n2 + 8n + 2log n • 5n2 log n • 5n2.5

Lie in between

Which Functions are Polynomials?• nc

• n0.0001

• n10000

• 5n2 + 8n + 2log n • 5n2 log n • 5n2.5

Ignore low-order termsIgnore power constant.

Ignore "small" values of n. Write nθ(1)

Which Functions are Exponential?

• 2n

• … ?


• 2n

• 20.0001 n

• 210000 n

n times some constant powerraises to the power of 2.


• 2n

• 20.0001 n

• 210000 n

• 8n

• 2n / n100 •2n · n100

too small?too big?


• 2n

• 20.0001 n

• 210000 n

• 8n

• 2n / n100 •2n · n100

= 23n

> 20.5n

< 22n

Lie in between


• 2n

• 20.0001 n

• 210000 n

• 8n

• 2n / n100 •2n · n100

= 23n

> 20.5n

< 22n

20.5n > n100

2n = 20.5n · 20.5n > n100 · 20.5n

2n / n100 > n0.5n


Ignore low-order termsIgnore base.

Ignore "small" values of n. Ignore polynomial factors.

Write 2θ(n)

• 2n

• 20.0001 n

• 210000 n

• 8n

• 2n / n100 •2n · n100


Functions

Constant

Logarithm

ic

Poly L

ogarithmic

Polynom

ial

Exponential

Exp

Double E

xp

(log n)θ(1) nθ(1) 2θ(n)θ(log n)θ(1) 2nθ(1) 2θ(n)

2

Notations

Theta f(n) = θ(g(n)) f(n) ≈ c g(n)

BigOh f(n) = O(g(n)) f(n) ≤ c g(n)

Omega f(n) = Ω(g(n)) f(n) ≥ c g(n)

Little Oh f(n) = o(g(n)) f(n) << c g(n)

Little Omega f(n) = ω(g(n)) f(n) >> c g(n)

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

f(n) = θ(g(n))

Definition of Theta

f(n) is sandwiched between c1g(n) and c2g(n)

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

f(n) = θ(g(n))

Definition of Theta

f(n) is sandwiched between c1g(n) and c2g(n)

for some sufficiently small c1 (= 0.0001)

for some sufficiently large c2 (= 1000)

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

f(n) = θ(g(n))

Definition of Theta

For all sufficiently large n

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

f(n) = θ(g(n))

Definition of Theta

For all sufficiently large n

For some definition of “sufficiently large”

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

f(n) = θ(g(n))

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

c1·n2 3n2 + 7n + 8 c2·n2? ?

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·n2 3n2 + 7n + 8 4·n23 4

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·12 3·12 + 7·1 + 8 4·121False

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·72 3·72 + 7·7 + 8 4·727False

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·82 3·82 + 7·8 + 8 4·828True

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·92 3·92 + 7·9 + 8 4·929True

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·102 3·102 + 7·10 + 8 4·10210True

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·n2 3n2 + 7n + 8 4·n2?

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·n2 3n2 + 7n + 8 4·n2n 88

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·n2 3n2 + 7n + 8 4·n2

Truen 8

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·n2 3n2 + 7n + 8 4·n2

7n + 8 1·n2

7 + 8/n 1·n

n 8

True

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n2)

3·n2 3n2 + 7n + 8 4·n23 4 8 n 8

True

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

n2 = θ(n3)

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

n2 = θ(n3)

c1 · n3 n2 c2 · n3? ?

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

n2 = θ(n3)

0 · n3 n2 c2 · n30True, but ?

Definition of Theta

c c n n n1 0 2 0 0 0, , , , . . .

n2 = θ(n3)

0 Constants c1 and c2 must be positive!

False

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n)

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n)

c1·n 3n2 + 7n + 8 c2·n? ?

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n)

3·n 3n2 + 7n + 8 100·n3 100

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n)

3·n 3n2 + 7n + 8 100·n?

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n)

3·100 3·1002 + 7·100 + 8 100·100100

False

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n)

3·n 3n2 + 7n + 8 10,000·n3 10,000

Definition of Theta

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

3n2 + 7n + 8 = θ(n)

3·10,000 3·10,000 2 + 7·10,000 + 8 100·10,00010,000

False

Definition of Theta

3n2 + 7n + 8 = θ(n)

What is the reverse statement?

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

Understand Quantifiers!!!

Sam Mary

Bob Beth

JohnMarilin Monro

Fred Ann

Sam Mary

Bob Beth

JohnMarilin Monro

Fred Ann

b, Loves(b, MM)

b, Loves(b, MM)

b, Loves(b, MM)]

b, Loves(b, MM)]

Definition of Theta

c c n n n c g n f n o r f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( ) ( )

3n2 + 7n + 8 = θ(n)

The reverse statement

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

Definition of Theta


3n2 + 7n + 8 = θ(n)

3n2 + 7n + 8 > 100·n3 100 ?

Definition of Theta


3n2 + 7n + 8 = θ(n)

3·1002 + 7·100 + 8 > 100·1003 100

True100

Definition of Theta


3n2 + 7n + 8 = θ(n)

3n2 + 7n + 8 > 10,000·n3 10,000 ?

Definition of Theta


3n2 + 7n + 8 = θ(n)

3·10,0002 + 7·10,000 + 8 > 10,000·10,0003 10,000

True10,000

Definition of Theta


3n2 + 7n + 8 = θ(n)

3n2 + 7n + 8 > c2·nc1 c2 ?

Definition of Theta


3n2 + 7n + 8 = θ(n)

3·c22 + 7 · c2 + 8 > c2·c2c1 c2

Truec2

Definition of Theta


3n2 + 7n + 8 = θ(n)

3n2 + 7n + 8 > c2·nc1 c2 ?n0

Definition of Theta


3n2 + 7n + 8 = θ(n)

3·c22 + 7 · c2 + 8 > c2·c2c1 max(c2,n0 )

Truec2 n0

True

Definition of Theta

3n2 + 7n + 8 = g(n)θ(1)

c1, c2, n0, n n0 g(n)c1 f(n) g(n)c2

3n2 + 7n + 8 = nθ(1)

Definition of Theta

3n2 + 7n + 8 = nθ(1)

nc1 3n2 + 7n + 8 nc2

c1, c2, n0, n n0 g(n)c1 f(n) g(n)c2

? ?

Definition of Theta

3n2 + 7n + 8 = nθ(1)

n2 3n2 + 7n + 8 n3

c1, c2, n0, n n0 g(n)c1 f(n) g(n)c2

2 3

Definition of Theta

3n2 + 7n + 8 = nθ(1)

n2 3n2 + 7n + 8 n3

c1, c2, n0, n n0 g(n)c1 f(n) g(n)c2

2 3 ? n ?

Definition of Theta

3n2 + 7n + 8 = nθ(1)

n2 3n2 + 7n + 8 n3

c1, c2, n0, n n0 g(n)c1 f(n) g(n)c2

2 3 5 n 5

Definition of Theta

3n2 + 7n + 8 = nθ(1)

n2 3n2 + 7n + 8 n3

True

True

c1, c2, n0, n n0 g(n)c1 f(n) g(n)c2

2 3 5 n 5

Order of Quantifiers

n n n c c c g n f n c g n0 0 1 2 1 2, , ( ) ( ) ( ), ,

f(n) = θ(g(n))

?

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

g b lo ves b g, , ( , ) b g lo ves b g, , ( , )

Understand Quantifiers!!!

One girl Could be a separate girl for each boy.

Sam Mary

Bob Beth

JohnMarilin Monro

Fred Ann

Sam Mary

Bob Beth

JohnMarilin Monro

Fred Ann

Order of Quantifiers

No! It cannot be a different c1 and c2

for each n.

n n n c c c g n f n c g n0 0 1 2 1 2, , ( ) ( ) ( ), ,

f(n) = θ(g(n))

c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )

Other Notations

Theta f(n) = θ(g(n)) f(n) ≈ c g(n)

BigOh f(n) = O(g(n)) f(n) ≤ c g(n)

Omega f(n) = Ω(g(n)) f(n) ≥ c g(n)

Little Oh f(n) = o(g(n)) f(n) << c g(n)

Little Omega f(n) = ω(g(n)) f(n) >> c g(n)

BigOh Notation

• n2 = O(n3)

• n3 = O(n2)

BigOh Notation

• O(n2) = O(n3)

• O(n3) = O(n2)Odd Notation

f(n) = O(g(n)) Standard f(n) ≤ O(g(n)) Stresses one function dominating another.

f(n)O(g(n)) Stress function is member of class.

The Time Complexity of an Algorithm

Specifies how the running time depends on the size of the input.

Purpose?

Purpose

• To estimate how long a program will run.

• To estimate the largest input that can reasonably be given to the program.

• To compare the efficiency of different algorithms.

• To help focus on the parts of code that are executed the largest number of times.

• To choose an algorithm for an application.

Time Complexity Is a Function


A function mapping

“size” of input

“time” T(n) executed .

Definition of Time?

Definition of Time

• # of seconds (machine dependent).

• # lines of code executed.

• # of times a specific operation is performed (e.g., addition).

Which?

Definition of Time

• # of seconds (machine dependent). • # lines of code executed. • # of times a specific operation is performed

(e.g., addition).

These are all reasonable definitions of time, because they are within a constant factor of

each other.

Size of Input Instance?

83920

Size of Input Instance

• Size of paper• # of bits• # of digits• Value

- n = 2 in2

- n = 17 bits

- n = 5 digits

- n = 83920

2’’

83920

5

1’’

Which are reasonable?



- n = 2 in2

- n = 17 bits

- n = 5 digits

- n = 83920

• Intuitive

2’’

83920

5

1’’



- n = 2 in2

- n = 17 bits

- n = 5 digits

- n = 83920

• Intuitive• Formal

2’’

83920

5

1’’



- n = 2 in2

- n = 17 bits

- n = 5 digits

- n = 83920

• Intuitive• Formal• Reasonable

# of bits =

3.32 * # of digits

2’’

83920

5

1’’



- n = 2 in2

- n = 17 bits

- n = 5 digits

- n = 83920

• Intuitive• Formal• Reasonable• Unreasonable

# of bits = log2(Value) Value = 2# of bits

2’’

83920

5

1’’

Two Example Algorithms

• Sum N array entries: A(1) + A(2) + A(3) + …

• Factor value N: Input N=5917 & Output N=97*61.

Algorithm N/2, N/3, N/4, …. ?

Time?





Time = N

Time = N





Is this reasonable?

Time = N

Time = N





No!

Time = N

Time = N

One is considered fast and the other slow!










size n = N

size n = log N





Time as function of input size?

size n = N

size n = log N





Time = N = n

Time = N = 2n

size n = N

size n = log N





Linear vs Exponential Time!

Time = N = n

Time = N = 2n

size n = N

size n = log N


14,23,25,30,31,52,62,79,88,98


• # of elements - n = 10 elements

14,23,25,30,31,52,62,79,88,98

10


• # of elements - n = 10 elements

14,23,25,30,31,52,62,79,88,98

10

Is this reasonable?


• # of elements - n = 10 elements ~ Reasonable

14,23,25,30,31,52,62,79,88,98

10

If each element has size c

# of bits = c * # of elements


• # of elements - n = 10 elements Reasonable

14,23,25,30,31,52,62,79,88,98

10

If each element is in [1..n]

each element has size log n

# of bits = n log n ≈ n

Time Complexity Is a Function


A function mapping

# of bits n needed to represent the input

# of operations T(n) executed .

Which Input of size n?

There are 2n inputs of size n.Which do we consider

for the time T(n)?


Typical Input

Average Case

Worst Case


Typical Input But what is typical?

Average Case For what distribution?

Worst Case •Time for all inputs is bound.

•Easiest to compute

Time Complexity of Algorithm

O(n2): Prove that for every input of size n, the algorithm takes no more than cn2 time.

Ω(n2): Find one input of size n, for which the algorithm takes at least this much time.

θ (n2): Do both.

The time complexity of an algorithm isthe largest time required on any input of size n.

Time Complexity of Problem

O(n2): Provide an algorithm that solves the problem in no more than this time.

Ω(n2): Prove that no algorithm can solve it faster.θ (n2): Do both.

The time complexity of a problem is the time complexity of the fastest algorithm that solves the problem.

Adding The Classic Techniques

Evaluating ∑i=1 f(i).n

Gauss ∑i=1..n i = 1 + 2 + 3 + . . . + n

= ?

Arithmetic Sum

1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

2

1)(n n S

1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

Let’s restate this argument using a

geometric representation

Algebraic argument

1 2 . . . . . . . . n

= number of white dots.1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

1 2 . . . . . . . . n

= number of white dots

= number of yellow dots

n . . . . . . . 2 1

1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

n+1 n+1 n+1 n+1 n+1



n

n

n

n

n

n

There are n(n+1) dots in the grid

1 + 2 + 3 + . . . + n-1 + n = S

n + n-1 + n-2 + . . . + 2 + 1 = S

(n+1) + (n+1) + (n+1) + . . . + (n+1) + (n+1) = 2S

n (n+1) = 2S

n+1 n+1 n+1 n+1 n+1



n

n

n

n

n

n

2

1)(n n S

Note = (# of terms · last term))

Gauss ∑i=1..n i = 1 + 2 + 3 + . . . + n

= (# of terms · last term)

Arithmetic Sum

True when ever terms increase slowly

∑i=0..n 2i = 1 + 2 + 4 + 8 +. . . + 2n

= ?

Geometric Sum

Geometric Sum

∑i=0..n 2i = 1 + 2 + 4 + 8 +. . . + 2n

= 2 · last term - 1

Geometric Sum

∑i=0..n ri = r0 + r1 + r2 +. . . + rn

= ?

Geometric Sum

S

Sr r r r ... r r

S Sr 1 r

Sr 1

r 1

2 3 n n 1

n 1

n 1

1 r r r ...2 3 rn

Geometric Sum

∑i=0..n rir 1

r 1

n 1

Geometric SumWhen r>1?

∑i=0..n rir 1

r 1

n 1

Geometric Sum

θ(rn)Biggest TermWhen r>1

∑i=0..n ri = r0 + r1 + r2 +. . . + rn

= (biggest term)

Geometric Increasing

True when ever terms increase quickly

∑i=0..n ri

Geometric SumWhen r<1?

1 r

1 r

n 1

∑i=0..n ri

Geometric Sum

1 r

1 r

n 1

θ(1)When r<1

Biggest Term

∑i=0..n ri = r0 + r1 + r2 +. . . + rn

= (1)

Bounded Tail

True when ever terms decrease quickly

∑i=1..n 1/i

= 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + …+ 1/n

= ?

Harmonic Sum

f(i) = 1∑i=1..n f(i) = n

n

Sum of Shrinking Function

f(i) = 1/2i

∑i=1..n f(i) = 2


f(i) = 1/i∑i=1..n f(i) = ?

n


Harmonic Sum

∑i=1..n 1/i

= 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + …+ 1/n

= (log(n))

Harmonic Sum

Approximating Sum by Integrating

The area under the curveapproximates the sum

∑i=1..n f(i) ≈ ∫x=1..n f(x) dx


∫x=1..n xc dx = 1/c+1 nc+1∑i=1..n ic ≈

Arithmetic Sums

θ(# of terms · last term)True when ever terms increase slowly

= θ(nc+1)


∫x=1..n bx dx = 1/ln(b) bn∑i=1..n bi ≈

Geometric Sums

θ(last term)True when ever terms increase quickly

= θ(bn)

Harmonic Sum



Problem: Integrating may be hard too.

Outline

II) Math proofs of sums

V) Ability to know the answer fast

I) What is the sumEg. ∑i=1..n f(i) = ∑i=1..n 1/i

0.9999 =θ(n0.0001)

III) Pattern of sums ∑i=1..n f(i) = θ(n · f(n))

IV) Intuition why this is the sum

I flash it upIf you follow great.If not don’t panic.

Adding Made Easy

We will now classify (most) functions f(i)

into four classes:

–Geometric Like–Arithmetic Like–Harmonic–Bounded Tail

Adding Made Easy

We will now classify (most) functions f(i)

into four classes

For each class, we will give

an easy rule for approximating

it’s sum

θ( ∑i=1..n f(i) )

Classifying Animals

Vertebrates

Birds

Mam

mals

Reptiles

Fish

Dogs

Giraffe

Mammal Complex social networks Dog Man’s best friend


Functions

Poly.

Exp.

2θ(n) ∑i=1..n f(i) = θ(f(n))

8·2n / n100 + n3

θ(2n / n100)

8·2n + n3

nθ(1) 2θ(n)

θ(2n)

f(n) = 8·2n / n100 + n3

θ(2n / n100) ∑i=1..n f(i) = θ(2n / n100)

20.5n < < 2n8· 2n / n100 + n3

Significant Less significant

Irrelevant

Adding Made Easy

11

1

2

34

Four Classes of Functions

Adding Made Easy

If the terms f(i) grow sufficiently quickly, then the sum will be

dominated by the largest term.

Silly example:1+2+3+4+5+ 1,000,000,000 ≈ 1,000,000,000

f(i) 2Ω(n) ∑i=1..n f(i) = θ(f(n))Geometric Like:



Classic example:

?




Classic example:∑i=1..n 2i = 2n+1-1 ≈ 2 f(n)




For which functions f(i) is this true?

How fast and how slow can it grow?


r 1r 1

n 1

θ( )r n Last Term

when r>1.

∑i=1..n ri 1 r r r . . .2 3 r n

θ(f(n))

∑i=1..n (1.0001)i ≈ 10,000 (1.0001)n

= 10,000 f(n)Lower Extreme:

Upper Extreme: ∑i=1..n (1000)i ≈ 1.001(1000)n

= 1.001 f(n)

Recall, when f(n) = ri = 2θ(n)


Because the constant is sooo big, the statement is just barely true.

∑i=1..n (1.0001)i ≈ 10,000 (1.0001)n



= 1.001 f(n)


∑i=1..n (1.0001)i ≈ 10,000 (1.0001)n



= 1.001 f(n)

Even bigger?


2n2i

∑i=1..n 22 ≈ 22 = 1f(n)

No Upper Extreme:

Even bigger?

∑i=1..n (1.0001)i ≈ 10,000 (1.0001)n



2n2i

∑i=1..n 22 ≈ 22 = 1f(n)

No Upper Extreme:

Functions in between?

∑i=1..n (1.0001)i ≈ 10,000 (1.0001)n



∑i=1..n 2i

= 2·2n –2

= 21 + 22 + 23 + 24 +…+ 2n

8·[ ]

8· 8· 8· 8· 8·

8·

= θ( 2n )

1100 2100 3100 4100 n100

i100

n100

+ i3

+ 13 + 23 + 33 + 43 + n3

+ n3

Dominated by the largest term.

= θ(f(n))

f(n) = 2n8·n100

+ n3 = θ( 2n )n100


n100≈


f(n) = 2n8·n100

∑i=1..n f(i)f(n) c·f(n)f(i) ·f(n)f(i) ·f(i+1)

Easy Hard

(i+1)100

i1002

= (1+1/i)100

12 1.9

=8·2(i+1)

(i+1)100f(i+1)f(i) 8·2i

i100

=

1/.51

·f(i+1)f(i)


f(n) = 2n8·n100

∑i=1..n f(i)f(n) c·f(n)f(i) ·f(n)

·f(i+2)

·f(i+1)f(i)

·f(i+1)f(i) ·f(i+3).51(n-i)·f(i+(n-i)).51(n-i)·f(n)

f(i) .51(n-i)·f(n)

Eg. i = n, f(n) .51(n-n)·f(n)1·f(n)

i = 0, f(0) .51n·f(n)


f(n) = 2n8·n100

∑i=1..n f(i)f(n) c·f(n)

∑i=1..n f(i) ∑i=1..n .51(n-i)·f(n)

= θ(1) · f(n)

f(i) ·f(n)·f(i+1)f(i)

f(i) .51(n-i)·f(n)

= [∑i=1..n .51(n-i)] · f(n)

Functions

Poly.

Exp.

2θ(n) ∑i=1..n f(i) = θ(f(n))

8·2n / n100 + n3

θ(2n / n100)

8·2n + n3

nθ(1) 2θ(n)

θ(2n)

f(n) = 8·2n / n100 + n3

θ(2n / n100) ∑i=1..n f(i) = θ(2n / n100)

20.5n < < 2n8· 2n / n100 + n3

Significant Less significant

Irrelevant


If f(n) = c ban nd loge (n)c ?a ?b ?d ?e ?

f(n) = Ω, θ ?

then ∑i=1..n f(i) = θ(f(n)).

2Ω(n)

> 0 0 1

(-,)(-,)



All functions in 2Ω(n)? Maybe not.


Functions that oscillate continually do not have the property.


Functions expressed with +, -, , , exp, log

do not oscillate continually.

They are well behaved for sufficiently large n.

These do have the property.


Adding Made Easy

Done

Adding Made Easy

If most of the terms f(i) have roughly the same value, then the sum is roughly the

number of terms, n, times this value.

Silly example:1,001 + 1,002 + 1,003 + 1,004 + 1,005

≈ 5 · 1,000

f(i) = nθ(1)-1 ∑i=1..n f(i) = θ(n·f(n))Arithmetic Like:



Another silly example:∑i=1..n 1 = n · 1




Is the statement true for this function?

∑i=1..n i = 1 + 2 + 3 + . . . + n





∑i=1..n i = 1 + 2 + 3 + . . . + n

The terms are not roughly the same.


But half the terms are roughly the same.

∑i=1..n i =

1 + . . . + n/2 + . . . + n


But half the terms are roughly the same

and the sum is roughly the number

terms, n, times this value

∑i=1..n i =

1 + . . . + n/2 + . . . + n

∑i=1..n i = θ(n · n)



∑i=1..n i2 = 12 + 22 + 32 + . . . + n2

Even though, the terms are not roughly the same.


Again half the terms are roughly the same.

∑i=1..n i =

12 + . . . + (n/2)2 + . . . + n2

1/4 n2


Again half the terms are roughly the same.

∑i=1..n i =

12 + . . . + (n/2)2 + . . . + n2

1/4 n2

∑i=1..n i2 = θ(n · n2)


∑i=1..n f(i) ≈ area under curve


area of small square ∑i=1..n f(i)

≈ area under curve area of big square


n/2 · f(n/2)

= area of small square ∑i=1..n f(i)


= n · f(n)


θ(n · f(n)) = n/2 · f(n/2)



= n · f(n)


θ(n · f(n)) = n/2 · f(n/2)



= n · f(n)

?


∑i=1..n i2 = 12 + 22 + 32 + . . . + n2


f(i) = n2

f(n/2) = θ(f(n))

The key property is

= ?

The key property is

f(n/2) = (n/2)2 = 1/4 n2 = θ(n2) = θ(f(n))

∑i=1..n i2 = 12 + 22 + 32 + . . . + n2


f(i) = n2

= θ(n·f(n)) = θ(n · n2)

∑i=1..n f(i) = ?


f(i) = nθ(1)

∑i=1..n ir = 1r + 2r + 3r + . . . + nr


f(i) = nθ(1)

f(n/2) = θ(f(n))

The key property is

The key property is

f(n/2) = (n/2)r = 1/2r nr = θ(nr) = θ(f(n))

∑i=1..n ir = 1r + 2r + 3r + . . . + nr


= θ(n·f(n)) = θ(n · nr)

f(i) = nθ(1)

∑i=1..n 2i = 21 + 22 + 23 + . . . + 2n


f(i) = nθ(1)

f(n/2) = θ(f(n))

The key property is

The key property is

∑i=1..n 2i = 21 + 22 + 23 + . . . + 2n

f(n/2) = 2(n/2) = θ(2n) = θ(f(n))


f(i) = nθ(1)

= θ(n·f(n)) = θ(n · 2n)

∑i=1..n 1 = n · 1 = n · f(n)

Middle Extreme:

Upper Extreme: ∑i=1..n i1000 = 1/1001 n1001

= 1/1001 n · f(n)

All functions in between.


Adding Made Easy

Half done

f(i) = 1∑i=1..n f(i) = n

n


f(i) = ?∑i=1..n f(i) = n1/2

n


1/i1/2

f(i) = 1/i∑i=1..n f(i) = log n

n


f(i) = 1/2i

∑i=1..n f(i) = 2




Does the statement hold for functions f(i) that shrink?




∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n

Does the statement hold for the Harmonic Sum?



∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n

∑i=1..n f(i) = ?

θ(n · f(n))



∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n

∑i=1..n f(i)

θ(n · f(n))

= ∑i=1..n 1/i = ?



∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n

∑i=1..n f(i)

θ(n · f(n))

= ∑i=1..n 1/i = θ(log n)



∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n

∑i=1..n f(i)

= θ(n · f(n))

= ∑i=1..n 1/i = θ(log n)

?



∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n

∑i=1..n f(i)

= θ(n · f(n))

= ∑i=1..n 1/i = θ(log n)

θ(1) = θ(n · 1/n)



∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n

∑i=1..n f(i)

= θ(n · f(n))

= ∑i=1..n 1/i = θ(log n)

θ(1) = θ(n · 1/n)≠

No the statement does not hold!


Adding Made Easy

not included

Does the statement hold for the almost Harmonic Sum?

∑i=1..n 1/i0.9999

Shrinks slightly slower than harmonic.



∑i=1..n 1/i0.9999

∑i=1..n f(i)

= θ(n · f(n))

= ∑i=1..n 1/i0.9999

= ?



∑i=1..n 1/i0.9999

∑i=1..n f(i)

= θ(n · f(n))

= ∑i=1..n 1/i0.9999

= θ(n0.0001)



∑i=1..n 1/i0.9999

∑i=1..n f(i)

= θ(n · f(n))

= ∑i=1..n 1/i0.9999

= θ(n0.0001)

?



∑i=1..n 1/i0.9999

∑i=1..n f(i)

= θ(n · f(n))

= ∑i=1..n 1/i0.9999

= θ(n0.0001)

θ(n0.0001) = θ(n · 1/n0.9999)



∑i=1..n 1/i0.9999

∑i=1..n f(i)

= θ(n · f(n))

= ∑i=1..n 1/i0.9999

= θ(n0.0001)

θ(n0.0001) = θ(n · 1/n0.9999)

=

The statement does hold!


∑i=1..n 1 = n · 1 = n · f(n)

Middle Extreme:

Lower Extreme: ∑i=1..n 1/i0.9999

= θ(n0.0001) = θ(n · f(n))



∑i=1..n 1 = n · 1 = n · f(n)

Middle Extreme:

Lower Extreme: ∑i=1..n 1/i0.9999

= θ(n0.0001) = θ(n · f(n))

Upper Extreme: ∑i=1..n i1000 = 1/1001 n1001

= 1/1001 n · f(n)


Arithmetic Like


f(n) = Ω, θ ?

then ∑i=1..n f(i) = θ(n · f(n)).

n-1+θ(1)

> 0 0

or b 1> -1

(-,)

(For +, -, , , exp, log functions f(n))

Conclusion

Adding Made Easy

Done

Adding Made Easy

Harmonic

Harmonic Sum

∑i=1..n 1/i

= 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + …+ 1/n

= (log(n))

Adding Made Easy

If the terms f(i) decrease towards zerosufficiently quickly,

then the sum will be a constant.

The classic example ∑i=0..n 1/2i = 1 + 1/2 + 1/4 + 1/8 + … = 2.

f(n) n-1-Ω(1) ∑i=1..n f(i) = θ(1)Bounded Tail:

If f(i) decays even faster, then the tail of the sum is more bounded.

2i

∑i=1..n 22 = θ(1).

1


Upper Extreme: ∑i=1..n 1/i1.0001

= θ(1)


Upper Extreme: ∑i=1..n 1/i1.0001

= θ(1)

No Lower Extreme:2i

∑i=1..n 22 = θ(1).

1



Bounded Tail


f(n) = Ω, θ ?

then ∑i=1..n f(i) = θ(1).

> 0

(-,)

(-,)

Conclusion

or b 1

c nd loge (n)

c a b

ban

Bounded Tail

If f(n) = c nd loge (n)c ?a b = 1d ?e ?

f(n) = Ω, θ ?

then ∑i=1..n f(i) = θ(1).

nθ(1)-1

> 0

< -1(-,)

Conclusion

Adding Made Easy

Done

Adding Made EasyMissing Functions

n n

n nlo g

1/nlogn

logn/n

Adding Made Easy • Geometric Like: If f(n) 2Ω(n), then ∑i=1..n f(i) = θ(f(n)).

• Arithmetic Like: If f(n) = nθ(1)-1, then ∑i=1..n f(i) = θ(n · f(n)).

• Harmonic: If f(n) = 1/n , then ∑i=1..n f(i) = logen + θ(1).

• Bounded Tail: If f(n) n-1-Ω(1), then ∑i=1..n f(i) = θ(1).

(For +, -, , , exp, log functions f(n))

This may seem confusing, but it is really not.

It should help you compute most sums easily.

Recurrence Relations

T(1) = 1

T(n) = a T(n/b) + f(n)

Recurrence Relations Time of Recursive Program

procedure Eg(int n)

if(n1) then

put “Hi”

else

loop i=1..f(n)

put “Hi”

loop i=1..a

Eg(n/b)

Recurrence relationsarise from the timing of recursive programs.

Let T(n) be the # of “Hi”s on an input of “size” n.

procedure Eg(int n)

if(n1) then

put “Hi”

else

loop i=1..f(n)

put “Hi”

loop i=1..a

Eg(n/b)

Given size 1,the program outputs

T(1)=1 Hi’s.


procedure Eg(int n)

if(n1) then

put “Hi”

else

loop i=1..f(n)

put “Hi”

loop i=1..a

Eg(n/b)

Given size n,the stackframe outputs

f(n) Hi’s.


procedure Eg(int n)

if(n1) then

put “Hi”

else

loop i=1..f(n)

put “Hi”

loop i=1..a

Eg(n/b)

Recursing on a instances of size n/bgenerates T(n/b) “Hi”s.


procedure Eg(int n)

if(n1) then

put “Hi”

else

loop i=1..f(n)

put “Hi”

loop i=1..a

Eg(n/b)

Recursing a times

generates a·T(n/b) “Hi”s.


procedure Eg(int n)

if(n1) then

put “Hi”

else

loop i=1..f(n)

put “Hi”

loop i=1..a

Eg(n/b)

For a total of T(1) = 1 T(n) = a·T(n/b) + f(n)

“Hi”s.


X = 2133Y = 2312ac = bd = (a+b)(c+d) = XY =

MULT(X,Y):

If |X| = |Y| = 1 then RETURN XY

Break X into a;b and Y into c;d

e = MULT(a,c) and f =MULT(b,d)

RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f

Stack of Stack Frames

Stack Frame: A particular execution of one routine on one particular input instance.

X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = bd = (a+b)(c+d) = XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23

ac = ?bd = (a+b)(c+d) = XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23

ac = ?bd = (a+b)(c+d) = XY =

X = 2Y = 2XY=

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23

ac = ?bd = (a+b)(c+d) = XY =

X = 2Y = 2XY = 4

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = 4bd = (a+b)(c+d) = XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = 4

bd = ?(a+b)(c+d) = XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = 4

bd = ?(a+b)(c+d) = XY =

X = 1Y = 3XY = 3

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = 4bd = 3(a+b)(c+d) = XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = 4bd = 3

(a+b)(c+d) = ?XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = 4bd = 3

(a+b)(c+d) = ?XY =

X = 3Y = 5XY = 15

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = 4bd = 3(a+b)(c+d) = 15

XY = ?

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312

ac = ?bd = (a+b)(c+d) = XY =

X = 21Y = 23ac = 4bd = 3(a+b)(c+d) = 15XY = 483

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312ac = 483bd = (a+b)(c+d) = XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312ac = 483

bd = ?(a+b)(c+d) = XY =

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312ac = 483

bd = ?(a+b)(c+d) = XY =

X = 33Y = 12

ac = ?bd = (a+b)(c+d) = XY = 15

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312ac = 483

bd = ?(a+b)(c+d) = XY =

X = 33Y = 12

ac = ?bd = (a+b)(c+d) = XY = 15

X = 3Y = 1XY = 3

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312ac = 483

bd = ?(a+b)(c+d) = XY =

X = 33Y = 12ac = 3

bd = ?(a+b)(c+d) = XY = 15

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312ac = 483

bd = ?(a+b)(c+d) = XY =

X = 33Y = 12ac = 3

bd = ?(a+b)(c+d) = XY = 15

X = 3Y = 2XY = 6

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312ac = 483

bd = ?(a+b)(c+d) = XY =

X = 33Y = 12ac = 3bd = 6

(a+b)(c+d) = ?XY = 15

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


X = 2133Y = 2312ac = 483

bd = ?(a+b)(c+d) = XY =

X = 33Y = 12ac = 3bd = 6

(a+b)(c+d) = ?XY = 396

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f


An so on ….

X = 2133Y = 2312ac = 483bd = 396(a+b)(c+d) = 1890XY = 4931496

X = 21Y = 23ac = 4bd = 3(a+b)(c+d) = 15XY = 483

X = 33Y = 12ac = 3bd = 6(a+b)(c+d) = 18XY = 396

X = 54Y = 35ac = 15bd = 20(a+b)(c+d) = 72XY = 1890

X = 2Y = 2XY=4

X = 1Y = 3XY=3

X = 3Y = 5XY=15

X = 3Y = 1XY=3

X = 3Y = 2XY=6

X = 6Y = 3XY=18

X = 5Y = 3XY=15

X = 4Y = 5XY=20

X = 9Y = 8XY=72

MULT(X,Y):




RETURN

e 10n + (MULT(a+b, c+d) – e - f) 10n/2 + f

Tree of Stack Frames

Solving Technique 1Guess and Verify

•Recurrence Relation:

T(1) = 1 & T(n) = 4T(n/2) + n

•Guess: G(n) = 2n2 – n

•Verify: Left Hand Side Right Hand Side

T(1) = 2(1)2 – 1

T(n)

= 2n2 – n

1

4T(n/2) + n

= 4 [2(n/2)2 – (n/2)] + n

= 2n2 – n


T(1) = 1 & T(n) = 4T(n/2) + n

•Guess: G(n) = an2 + bn + c


T(1) = a+b+c

T(n)

= an2 +bn+c

1

4T(n/2) + n

= 4 [a (n/2)2 + b (n/2) +c] + n

= an2 +(2b+1)n + 4c

Solving Technique 2Guess Form and Calculate Coefficients


T(1) = 1 & T(n) = 4T(n/2) + n

•Guess: G(n) = an2 + bn + 0


T(1) = a+b+c

T(n)

= an2 +bn+c

1

4T(n/2) + n

= 4 [a (n/2)2 + b (n/2) +c] + n

= an2 +(2b+1)n + 4c


c=4cc=0


T(1) = 1 & T(n) = 4T(n/2) + n

•Guess: G(n) = an2 - 1n + 0


T(1) = a+b+c

T(n)

= an2 +bn+c

1

4T(n/2) + n

= 4 [a (n/2)2 + b (n/2) +c] + n

= an2 +(2b+1)n + 4c


b=2b+1b=-1


T(1) = 1 & T(n) = 4T(n/2) + n

•Guess: G(n) = an2 - 1n + 0


T(1) = a+b+c

T(n)

= an2 +bn+c

1

4T(n/2) + n

= 4 [a (n/2)2 + b (n/2) +c] + n

= an2 +(2b+1)n + 4c


a=a


T(1) = 1 & T(n) = 4T(n/2) + n

•Guess: G(n) = 2n2 - 1n + 0


T(1) = a+b+c

T(n)

= an2 +bn+c

1

4T(n/2) + n

= 4 [a (n/2)2 + b (n/2) +c] + n

= an2 +(2b+1)n + 4c


a+b+c=1a-1+0=1a=2


T(1) = 1 & T(n) = aT(n/b) + f(n)

Solving Technique 3Approximate Form and

Calculate Exponent

which is bigger?

Guess


T(1) = 1 & T(n) = aT(n/b) + f(n)

•Guess: aT(n/b) << f(n)

•Simplify: T(n) f(n)

Solving Technique 3Calculate Exponent

In this case, the answer is easy. T(n) = (f(n))


T(1) = 1 & T(n) = aT(n/b) + f(n)

•Guess: aT(n/b) >> f(n)

•Simplify: T(n) aT(n/b)


In this case, the answer is harder.


T(1) = 1 & T(n) = aT(n/b)

•Guess: G(n) = cn


T(n)

= cn

aT(n/b)

= a [c (n/b) ]

= c a b n


(log a/log b)= cn

1 = a b-

b = alog b = log a = log a/log b


T(1) = 1 & T(n) = 4T(n/2)

•Guess: G(n) = cn


T(n)

= cn

aT(n/b) + f(n)

= c [a (n/b) ]

= c a b n


1 = a b-

b = alog b = log a = log a/log b

(log a/log b)= cn(log 4/log 2)= cn

2= cn


T(1) = 1 & T(n) = aT(n/b) + f(n)


If bigger then

T(n) = (f(n))

If bigger then(log a/log b)T(n) = (n )

And if aT(n/b) f(n) what is T(n) then?

Technique 4: Decorate The Tree

• T(n) = a T(n/b) + f(n)

f(n)

T(n/b) T(n/b) T(n/b) T(n/b)

T(n) =

T(1)

•T(1) = 1

1=

a

f(n)

T(n/b) T(n/b) T(n/b) T(n/b)

T(n) =a

f(n)

T(n/b) T(n/b) T(n/b)

T(n) =a

f(n/b)

T(n/b2)T(n/b2)T(n/ b2)T(n/ b2)

a

f(n)T(n) =a

f(n/b)


af(n/b)


af(n/b)


af(n/b)


a

11111111111111111111111111111111 . . . . . . 111111111111111111111111111111111

f(n)T(n) =a

f(n/b)


af(n/b)


af(n/b)


af(n/b)


a

Evaluating: T(n) = aT(n/b)+f(n)

Level

0

1

2

i

h


LevelInstance

size

0

1

2

i

h


LevelInstance

size

0 n

1

2

i

h


LevelInstance

size

0 n

1 n/b

2

i

h


LevelInstance

size

0 n

1 n/b

2 n/b2

i

h


LevelInstance

size

0 n

1 n/b

2 n/b2

i n/bi

h n/bh


LevelInstance

size

0 n

1 n/b

2 n/b2

i n/bi

h n/bh


LevelInstance

size

0 n

1 n/b

2 n/b2

i n/bi

h n/bh = 1

base case


LevelInstance

size

0 n

1 n/b

2 n/b2

i n/bi

h n/bh = 1


LevelInstance

size

0 n

1 n/b

2 n/b2

i n/bi

h = log n/log b n/bh = 1bh = nh log b = log nh = log n/log b


LevelInstance

size

Work

in stack

frame

0 n

1 n/b

2 n/b2

i n/bi

h = log n/log b 1


LevelInstance

size

Work

in stack

frame

0 n f(n)

1 n/b f(n/b)

2 n/b2 f(n/b2)

i n/bi f(n/bi)

h = log n/log b 1 T(1)


LevelInstance

size

Work

in stack

frame

# stack frames

0 n f(n)

1 n/b f(n/b)

2 n/b2 f(n/b2)

i n/bi f(n/bi)

h = log n/log b n/bh T(1)


LevelInstance

size

Work

in stack

frame

# stack frames

0 n f(n) 1

1 n/b f(n/b) a

2 n/b2 f(n/b2) a2

i n/bi f(n/bi) ai

h = log n/log b n/bh T(1) ah


LevelInstance

size

Work

in stack

frame

# stack frames

0 n f(n) 1

1 n/b f(n/b) a

2 n/b2 f(n/b2) a2

i n/bi f(n/bi) ai



LevelInstance

size

Work

in stack

frame

# stack frames

0 n f(n) 1

1 n/b f(n/b) a

2 n/b2 f(n/b2) a2

i n/bi f(n/bi) ai


ah = a

= n

log n/log b

log a/log b


LevelInstance

size

Work

in stack

frame

# stack frames

Work in Level

0 n f(n) 1

1 n/b f(n/b) a

2 n/b2 f(n/b2) a2

i n/bi f(n/bi) ai

h = log n/log b n/bh T(1) nlog a/log b


LevelInstance

size

Work

in stack

frame

# stack frames

Work in Level

0 n f(n) 1 1 · f(n)

1 n/b f(n/b) a a · f(n/b)

2 n/b2 f(n/b2) a2 a2 · f(n/b2)

i n/bi f(n/bi) ai ai · f(n/bi)

h = log n/log b n/bh T(1) nlog a/log b n · T(1)

log a/log b

Total Work T(n) = ∑i=0..h aif(n/bi)


= ∑i=0..h aif(n/bi)

If a Geometric Sum∑i=0..n xi θ(max(first term, last term))


LevelInstance

size

Work

in stack

frame

# stack frames

Work in Level

0 n f(n) 1 1 · f(n)

1 n/b f(n/b) a a · f(n/b)

2 n/b2 f(n/b2) a2 a2 · f(n/b2)

i n/bi f(n/bi) ai ai · f(n/bi)

h = log n/log b n/bh T(1) nlog a/log b n · T(1)

log a/log b

Dominated by Top Level or Base Cases

Evaluating: T(n) = aT(n/b)+f(n)Is the sum Geometric?

Simplify by letting f(n) = nc logkn

T(n) = ∑i=0..h aif(n/bi)

= ∑i=0..h ai(n/bi)c logk(n/bi)

= nc ∑i=0..h 2[log a - c log b] · i logk(n/bi)

Geometrically IncreasingArithmetic SumGeometrically Decreasing

Sum = θ(last term)

= θ( )

Sum = θ(any term # terms)

= θ(f(n) log n)

Sum = θ(first term)

= θ(f(n)).

log a - c log b < 0log a - c log b = 0log a - c log b < 0

c < log a/log b

c = log a/log b & k<-1c = log a/log b & k>-1c > log a/log b

nlog a/log b

(n/..)c = ncai = 2[log a]·i(../bi)c = 2[-c log b]·i

= nc ∑i=0..h 2-d i logk(n/bi)







Sum = θ(last term)

= θ( )


= θ(f(n) log n)


= θ(f(n)).


c < log a/log b


nlog a/log b

k=-1 logk(n/bi) = 1/(logn - i logb)

1/i (backwards)= nc ∑i=0..h 20 i logk(n/bi)

(back wards)

de







Sum = θ(last term)

= θ( )


= θ(f(n) log n)


= θ(f(n)).


c < log a/log b


nlog a/log b

= nc ∑i=0..h 2+d i logk(n/bi)

Evaluating: T(n) = 4T(n/2)+ n3/log5n

Time for top level?:


Time for top level: f(n) = n3/log5n

Time for base cases?:


Time for top level: f(n) = n3 /log5n

Time for base cases:


θ(n ) =log a/log b θ(n )

log ?/log ?

Time for top level: f(n) = n3 /log5n



θ(n ) =log a/log b θ(n ) = ?

log 4/log 2



Dominated?: c = ?


θ(n ) =log a/log b θ(n ) = θ(n2)

log 4/log 2

log a/log b = ?



Dominated?: c = 3 > 2 = log a/log b



log 4/log 2

Hence, T(n) = ?



Dominated?: c = 3 > 2 = log a/log b



log 4/log 2

Hence, T(n) = θ(top) = θ(f(n)) = θ(n3/log5n).

Time for top level: f(n) = 2n


Dominated?: c = ? > 2 = log a/log b

Evaluating: T(n) = 4T(n/2)+ 2n


log 4/log 2

Hence, T(n) = θ(top) = θ(f(n)) = θ(2n).

bigger

>big

The time is even more dominated by the top level of recursion

Time for top level: f(n) = n log5n


Dominated?: c = ? 2 = log a/log b

Evaluating: T(n) = 4T(n/2)+ n log5n


log 4/log 2

Hence, T(n) = ?

1<

= θ(base cases) = θ(n ) = θ(n2).log a/log b

Time for top level: f(n) = log5n



Evaluating: T(n) = 4T(n/2)+ log5n


log 4/log 2

Hence, T(n) = ?

0<


Time for top level: f(n) = n2



Evaluating: T(n) = 4T(n/2)+ n2


log 4/log 2

Hence, T(n) = ?

2=

= θ(f(n) log(n) ) = θ(n2 log(n)).

Time for top level: f(n) = n2 log5n


Dominated?: c = 2 = 2 = log a/log b

Evaluating: T(n) = 4T(n/2)+ n2 log5n


log 4/log 2

k = ?5 ?

Hence, T(n) = ?

> -1

= θ(f(n) log(n) ) = θ(n2 log6(n)).






log 4/log 2

k = ?

Hence, T(n) = ?

-5 ?< -1


Time for top level: f(n) = n2/log n



Evaluating: T(n) = 4T(n/2)+ n2/log n


log 4/log 2

k = ?

Hence, T(n) = ?

-1 ?= -1

= θ(nc loglog n) = θ(n2 loglog n).Did not do before.

Sufficiently close to: T(n) = 4T(n/2)+ n3 = θ(n3). T2(n) = ?

Evaluating: T2(n) = 4 T2(n/2+n1/2)+ n3

θ(n3).

Evaluating: T(n) = aT(n-b)+f(n)

h = ? n-hb

n-ib

n-2b

n-b

T(0)

f(n-ib)

f(n-2b)

f(n-b)

f(n)n

Work in Level

# stack frames

Work

in stack

frame

Instance

size

a

ai

a2

a

1

n/b a · T(0)

ai · f(n-ib)

a2 · f(n-2b)

a · f(n-b)

1 · f(n)

n/b

|base case| = 0 = n-hb

h = n/b

h = n/b

i

2

1

0

Level

Likely dominated by base cases Exponential

Evaluating: T(n) = 1T(n-b)+f(n)

h = ?

Work in Level

# stack frames

Work

in stack

frame

Instance

size

1

1

1

1

1

f(0)

f(n-ib)

f(n-2b)

f(n-b)

f(n)

n-b

n

n-hb

n-ib

n-2b

T(0)

f(n-ib)

f(n-2b)

f(n-b)

f(n)

h = n/b

i

2

1

0

Level

Total Work T(n) = ∑i=0..h f(b·i) = θ(f(n)) or θ(n·f(n))

Math Review: Exponents

XA XB = XA+B (not XAB !!)

XA / XB = XA-B

(XA )B = XAB

XA + XA = 2XA

2A + 2A = 2A+1

Logarithms

• Definition: XA = B if and only if logXB = A (x is the “base” of the logarithm).– Example: 102 = 100 means log10100 = 2.

• TheoremslogXAB = logXA + logXB

logXA/B = logXA – logXB

logX AB = B logXA

Logarithms…

• Theorem: logAB = logCB / logCAProof: Let X = logCB, Y = logCA, and

Z = logAB. By the definition of logarithm: CX = B, CY = A, and AZ = B.

Thus CX = B = AZ = CYZ , X = YZ, Z = X/Y.

Logarithms…

• The notation for logs can be confusing. There are two alternatives:– log2(x) = log (log x) or– log2(x) = (log x)2

• Generally, we use the 2nd definition.• Note: log2(x) is not (log x2)• Note: log is not a multiplicative entity, it is

a function.

End

Documents

goddard/Courses/CSCE310J Fundamentals of Algorithm Analysis Dr. Steve Goddard [email protected] CSCE 310J: Data Structures &