CSE 5311 Notes 1: Mathematical Preliminaries

(Last updated 1/20/18 12:56 PM) Chapter 1 - Algorithms & Computing Relationship between complexity classes, e.g.

€

logn ,

€

n,

€

n logn ,

€

n2,

€

2n, etc. Chapter 2 - Getting Started Loop Invariants and Design-by-Contract (especially Table 1 on p. 13):

http://dl.acm.org.ezproxy.uta.edu/citation.cfm?doid=2578702.2506375 RAM Model

Assumed Reality Memory Access Arithmetic Word size

Chapter 3 - Asymptotic Notation f(n) = O(g(n)) ⇒ g(n) bounds f(n) above (by a constant factor) f(n) = Ω(g(n)) ⇒ g(n) bounds f(n) below (by a constant factor) f(n) = θ(g(n)) ⇒ f(n) = O(g(n)) and f(n) = Ω(g(n)) Iterated Logarithms logk n = (log n)k

log(k) n = log(log(k-1) n) = log log . . . log n (log(1) n = log n) lg* n = Count the times that you can punch a log key on your calculator before value is ≤ 1 Arises for algorithms that run in “practically” or “nearly” constant time.

Appendix A - Summations Review: Geometric Series (p. 1147) Harmonic Series (p. 1147) Approximation by integrals (p. 1154)

2 Chapter 4 - Recurrences SUBSTITUTION METHOD - Review

1. Guess the bound. 2. Prove using (strong) math. induction.

Exercise:

€

T n( ) = 4T n2"

# $ %

& ' + n

Try

€

Ο n3# $ % &

' ( and confirm by math induction

€

Assume T k( ) ≤ ck3 for k < n

T n2#

$ % &

' ( ≤ c

n3

8

T n( ) = 4T n2#

$ % &

' ( + n

≤ 4c n3

8+ n

=c2n3 + n

= cn3 − c2n3 + n −

c2n3 + n ≤ 0 if n is sufficiently large

≤ cn3

Improve bound to

€

Ο n2# $ % &

' ( and confirm

€

Assume T k( ) ≤ ck2 for k < n

T n2#

$ % &

' ( ≤ c

n2

4

T n( ) = 4T n2#

$ % &

' ( + n ≤ 4c n

2

4+ n = cn2 + n STUCK!

3

€

Ω n3# $ % &

' ( as lower bound:

€

Assume T k( ) ≥ ck3 for k < n

T n2#

$ % &

' ( ≥ c

n3

8

T n( ) = 4T n2#

$ % &

' ( + n

≥ 4c n3

8+ n

=c2n3 + n

= cn3 − c2n3 + n −

c2n3 + n ≥ 0?

≥ cn3 DID NOT PROVE!!!

€

Ω n2# $ % &

' ( as lower bound:

€

Assume T k( ) ≥ ck2 for k < n

T n2#

$ % &

' ( ≥ c

n2

4

T n( ) = 4T n2#

$ % &

' ( + n ≥ 4c n

2

4+ n = cn2 + n ≥ cn2 for 0 < c

What’s going on . . .

€

T 1( ) = dT 2( ) = 4T 1( ) + 2 = 4d + 2

T 4( ) = 4T 2( ) + 4 = 4 4d + 2( ) + 4 =16d + 8 + 4 =16d +12

T 8( ) = 4T 4( ) + 8 = 4 16d +12( ) + 8 = 64d + 48 + 8 = 64d + 56

T 16( ) = 4T 8( ) +16 = 4 64d + 56( ) +16 = 256d + 224 +16 = 256d + 240

Hypothesis : T n( ) = dn2 + n2 − n = d +1( )n2 − n = cn2 − n

4

€

Ο n2# $ % &

' ( :

€

Assume T k( ) ≤ ck2 − k for k < n

T n2$

% & '

( ) ≤ c

n4

2−n2

T n( ) = 4T n2$

% & '

( ) + n ≤ 4 c n

4

2−n2

$

% & &

'

( ) ) + n = cn2 − 2n + n = cn2 − n

€

Ω n2# $ % &

' ( : [This was already proven.]

€

Assume T k( ) ≥ ck2 − k for k < n

T n2$

% & '

( ) ≥ c

n4

2−n2

T n( ) = 4T n2$

% & '

( ) + n ≥ 4 c n

4

2−n2

$

% & &

'

( ) ) + n = cn2 − 2n + n = cn2 − n

Exercise:

€

T n( ) = T n( ) +1

€

Ο logn( ):

€

Assume T k( ) ≤ c lgk for k < n

T n( ) ≤ c lg n = c lgn2

T n( ) = T n( ) +1≤ c lgn2

+1= c lgn − c lgn2

+1

≤ c lgn if c ≥ 2

€

Ο loglogn( )

€

Assume T k( ) ≤ c lglgk for k < n. (Note : loga loga n ∈ Θ logb logb n( ))

T n( ) ≤ c lglg n = c lg lgn2

= c lglgn − c

T n( ) = T n( ) +1≤ c lglgn − c +1

≤ c lglgn if c ≥1

5

€

Ω loglogn( )

€

Assume T k( ) ≥ c lglgk for k < n

T n( ) ≥ c lglg n = c lg lgn2

= c lglgn − c

T n( ) = T n( ) +1≥ c lglgn − c +1

≥ c lglgn if 0 < c ≤1

Substitution Method is a general technique - See CSE 2320 Notes 08 for quicksort analysis RECURSION-TREE METHOD - Review and Prelude to Master Method Convert to summation and then evaluate Example: Mergesort, p. 38

€

T n( ) = 2T n2( ) + n (case 2 for master method)

T(n=2k)=

T(n/2=2k-1)= T(n/2=2k-1)=

T(n/4=2k-2)= T(n/4=2k-2)= T(n/4=2k-2)= T(n/4=2k-2)=

T(n/8=2k-3)=

T(n/16=2k-4)=

T(n/2k=2k-k=1)=

.

.

.

6 Exercise:

€

T n( ) = 4T n2"

# $ %

& ' + n

T(n)⇒n

T(n/2)⇒n/2

T(n/4)⇒n/4

T(n/16)⇒n/16...

T(1)⇒1

n

4n/2=2n

16n/4=4n

64n/8=8n

256n/16=16n

4lg n =nlg 4 =n2

T(n/8)⇒n/8

.

.

.T(2)=T(n/2lg n-1)⇒2 4lg n-1•n/2lg n-1 =2lg n-1•n

lg n + 1levels

Using definite geometric sum formula:

€

cn 2k

k=0

lgn−1∑ + cn2 = cn 2lgn −1

2 −1+cn2 Using xk

k=0

t∑ =

xt+1 −1x −1

x ≠1

= cn n −1( ) +cn2

= cn2 − cn +cn2 =Θ n2& ' ( )

* +

(Case 1 of master method)

7 Exercise: (case 3 of master method)

€

T n( ) = 2T n2"

# $ %

& ' + n3

T(n)⇒n3

T(n/2)⇒n3/8

T(n/4)⇒n3/64

T(n/16)⇒n3/4096...

T(1)⇒1

n3

2n3/8=n3/4

4n3/64=n3/16

T(n/8)⇒n3/512

.

.

.

8n3/512=n3/64

16n3/4096=n3/256

2lg n=n

T(2)=T(n/2lg n-1)⇒8 2lg n-1(n/2lg n-1)3=n3/4lg n-1

lg n + 1levels

Using indefinite geometric sum formula:

€

cn3 1

4kk=0

lgn−1∑ + cn ≤ cn3 1

4kk=0

∞∑ + cn

= cn3 1

1− 14

+ cn From xk

k=0

∞∑ =

11− x

0 < x <1

=43cn3 + cn =O(n3)

Using definite geometric sum formula:

€

cn3 14kk=0

lgn−1∑ + cn = cn3

14( )lgn −1

14 −1

+ cn Using xk

k=0

t∑ =

xt+1 −1x −1

x ≠1

= cn3 n−2 −1− 3

4+ cn = cn31− n−2

34

+ cn =43cn3 1− 1

n2%

& '

(

) * + cn

=Θ n3% & ' (

) *

8 MASTER METHOD/THEOREM (“new”) - CLRS 4.5

€

T n( ) = aT nb( ) + f n( )

a ≥ 1 b > 1 Three mutually exclusive cases (proof sketched in 4.6.1):

1.

€

f n( ) =O nlogb a

nε#

$ %

&

' ( , ε > 0⇒ T n( ) =Θ nlogb a#

$ % &

' ( (leaves dominate)

2.

€

f n( ) =Θ nlogb a# $ % &

' ( ⇒ T n( ) =Θ nlogb a logn#

$ % &

' ( (each level contributes equally)

3.

€

f n( ) =Ω nlogb anε$ % & '

( ) , ε > 0, and af n

b( ) ≤ cf n( ),

c <1 for all sufficiently large n⇒ T n( ) =Θ f n( )( ) (root dominates)

(Problem 4.6-3 shows that

€

ε > 0 in 3. follows from the existence of

€

c and

€

n0 . By taking

€

n = bkn0

and the condition on f,

€

ac( )logb n−logb n0 f n0( ) ≤ f n( ) may be established. Last step is

€

ε ≤ logb 1c( ).)

Example:

€

T n( ) =10T n10"

# $

%

& ' + n

a = 10 b = 10 f(n) = n.5

€

nlog1010 = n Case 1: n.5 = O(n1-ε), 0 < ε ≤ 0.5

€

⇒ T n( ) =Θ n( )

9 Example: (recursion tree given earlier - leaves dominate)

€

T n( ) = 4T n2"

# $ %

& ' + n

a = 4 b = 2 f(n) = n

€

nlog2 4 = n2

Case 1: n = O(n2-ε), 0 < ε ≤ 1

€

⇒ T n( ) =Θ n2$ % & '

( )

Example:

€

T n( ) = 9T n3"

# $ %

& ' + 2n2

a = 9 b = 3 f(n) = 2n2

€

nlog39 = n2 Case 2: 2n2 = Θ(n2)

€

⇒ T n( ) =Θ n2 logn$ % & '

( )

Recursion Tree

T(n)⇒2n2

T(n/3)⇒2n2/9

9

9

T(n/9)⇒2n2/81

9

T(n/27)⇒2n2/729

.

.

.T(1)⇒c

2n2

2n2

2n2

2n2

c9log3 n=cnlog3 9=cn2

€

T n( ) = 2n2 log3 n + cn2 =Θ n2 logn# $ % &

' (

10 Example:

€

T n( ) = 2T n2"

# $ %

& ' + n3

a = 2 b = 2 f(n) = n3 nlg 2 = n Case 3: n3 = Ω(n1+ε), 0 < ε ≤ 2

€

af nb( ) = 2 n3

8"

# $

%

& ' ≤ cn3 = cf n( ), 1

4 ≤ c <1

€

⇒ T n( ) =Θ n3$ % & '

( )

Example:

€

T n( ) = T n2"

# $ %

& ' +

n2

a = 1 b = 2 f(n) = n/2 nlg 1 = n0 = 1 Case 3: n/2 = Ω(n0nε), 0 < ε ≤ 1

€

af nb( ) =1

n22

"

#

$ $

%

&

' '

= n4 ≤ c n2 = cf n( ), 1

2 ≤ c <1

€

⇒ T n( ) =Θ n( ) From CLRS, p. 95:

€

T n( ) = 2T n2"

# $ %

& ' + n lgn

a = 2 b = 2 f(n) = n lg n nlg 2 = n1 = n

€

f n( ) =ω n( ) , but

€

f n( )∉ Ω nnε% & ' (

) * for any

€

ε > 0 . . .

€

af nb( ) = 2 f n

2( ) = 2 n2 lg n2( ) = n lgn − n = cn lgn + 1− c( )n lgn − n

≤ cf n( ) = cn lgn for c ≥1

So master method (case 3) does not support

€

T n( ) =Θ n lgn( )

11

Using exercise 4.6-2 as a hint . . . (or a simple recursion tree with

€

n = 2k)

Substitution method to show

€

T n( ) =Θ n lg2 n# $ % &

' (

€

Ο :

Assume T k( ) ≤ ck lg2 k for k < n

T n2( ) ≤ c n2 lg2 n

2( ) = c n2 lgn −1( )2

T n( ) = 2T n2( ) + n lgn ≤ 2c n2 lgn −1( )2 + n lgn = cn lg2 n − 2cn lgn + cn + n lgn

≤ cn lg2 n for c ≥ 12 + ε

Ω :

Assume T k( ) ≥ ck lg2 k for k < n

T n2( ) ≥ c n2 lg2 n

2( ) = c n2 lgn −1( )2

T n( ) = 2T n2( ) + n lgn ≥ 2c n2 lgn −1( )2 + n lgn = cn lg2 n − 2cn lgn + cn + n lgn

≥ cn lg2 n for 0 < c ≤ 12

PROBABILISTIC ANALYSIS Hiring Problem - Interview potential assistants, always hiring the best available. Input: Permutation of 1 . . . n Output: The number of sequence values that are larger than all previous sequence elements. 3 1 2 4 1 2 3 4 4 3 2 1 Worst-case = n Observation: For any k-prefix of sequence, the last element is the largest with probability ______. Summing for all k-prefixes:

€

lnn < 1kk=1

n∑ = Hn ≤ lnn +1

12 Hiring m-of-n Problem Observation: For any k-prefix of sequence, the last element is one of the m largest with probability

______. ______ if k ≤ m ______ otherwise Sum for all k-prefixes (expected number that are hired)

€

m + mkk=m+1

n∑ = m + mHn −mHm

What data structure supports this application? Coupon Collecting (Knuth) n types of coupon. One coupon per cereal box. How many boxes of cereal must be bought (expected) to get at least one of each coupon type? Collecting the n coupons is decomposed into n steps: Step 0 = get first coupon Step 1 = get second coupon Step m = get m+1st coupon Step n - 1 = get last coupon Number of boxes for step m Let

€

pi = probability of needing exactly i boxes (difficult)

€

pi = mn( )i−1 n−mn , so the expected number of boxes for coupon m + 1 is

€

ipii=1

∞∑

Let

€

qi = probability of needing at least i boxes = probability that previous i - 1 boxes are failures (much easier to use) So,

€

pi = qi − qi+1

13

€

ipii=1

∞∑ = i qi − qi+1( )

i=1

∞∑

= iqii=1

∞∑ − iqi+1

i=1

∞∑

= iqii=1

∞∑ − i −1( )qi

i=2

∞∑

= q1 + iqii=2

∞∑ − iqi

i=2

∞∑ + qi

i=2

∞∑

= qii=1

∞∑

€

q1 =1

q2 = mn

q3 = mn( )2

qk = mn( )k−1

qii=1

∞∑ = m

n( )ii=0

∞∑ = 1

1−mn= nn−m = Expected number of boxes for coupon m + 1

Summing over all steps gives

€

nn−ii=0

n−1∑ = n 1+ 12 + 13 + ...+ 1n( ) ≤ n lnn +1( )

Analyze the following game: n sides on each die, numbered 1 through n. Roll dice one at a time. Keep a die only if its number > numbers on dice you already have. a. What is the expected number of rolls (including those not kept) to get a die numbered n?

€

n−1n( )

i=0

∞∑

i= 11−n−1n

= nn− n−1( )

= n

14 b. What number of dice do you expect (mathematically) to keep? Keep going until an n is encountered. Repeats of a number are discarded. Results in hiring problem

€

Hn ≤ lnn +1( ) Suppose a gambling game involves a sequence of rolls from a standard six-sided die. A player wins $1

when the value rolled is the same as the previous roll. If a sequence has 1201 rolls, what is the expected amount paid out?

Probability of a roll paying $1 = 1/6. 1200/6 = $200 Suppose a gambling game involves a sequence of rolls from a standard six-sided die. A player wins $1

when the value rolled is larger than the previous roll. If a sequence has 1201 rolls, what is the expected amount paid out?

€

Payout for next roll = 16 Expected payout after roll of ii=1

6∑

= 16

56 + 4

6 + 36 + 2

6 + 16 + 0( )

= 1536 = 5

12

1200*5/12 = $500 Suppose a gambling game involves a sequence of rolls from a standard six-sided die. A player wins k

dollars when the value k rolled is smaller than the previous roll. If a sequence has 601 rolls, what is the expected amount paid out?

€

16 1• 56 + 2• 46 + 3• 36 + 4 • 26 + 5• 16 + 6• 06( ) = 5+8+9+8+5

36 = 3536

600*35/36 = $583.33 Suppose a gambling game involves a sequence of rolls from a standard six-sided die. What is the

expected number of rolls until the first pair of consecutive sixes appears?

Expected rolls to get first six = 6 =

€

6−16( )

i=0

∞∑

i= 11−6−16

= 66− 6−1( )

%

&

' '

(

)

* *

Probability that next roll is not a six =

€

56

15

Expected rolls for consecutive sixes =

€

6 +1( ) 56( )

k=0

∞∑

k= 6 +1( ) 1

1−56= 42

Also, see: http://dl.acm.org.ezproxy.uta.edu/citation.cfm?doid=2408776.2408800 and

http://dl.acm.org.ezproxy.uta.edu/citation.cfm?doid=2428556.2428578 along with the book by Grinstead & Snell ( https://math.dartmouth.edu/~prob/prob/prob.pdf ), especially

the first four chapters (and Example 4.6 on p. 136, http://marilynvossavant.com/game-show-problem/ ). GENERATING RANDOM PERMUTATIONS PERMUTE-BY-SORTING (p. 125, skim) Generates randoms in 1 . . . n3 and then sorts to get permutation in Θ(n log n) time. Can use radix/counting sort (CSE 2320 Notes 8) to perform in Θ(n) time. RANDOMIZE-IN-PLACE Array A must initially contain a permutation. Could simply be identity permutation: A[i] = i. for i=1 to n swap A[i] and A[RANDOM(i,n)] Code is equivalent to reaching in a bag and choosing a number to “lock” into each slot. Uniform - all n! permutations are equally likely to occur. Problem 5.3-3 PERMUTE-WITH-ALL for i=1 to n swap A[i] and A[RANDOM(1,n)] Produces

€

nn outcomes, but

€

n! does not divide into

€

nn evenly. ∴ Not uniform - some permutations are produced more often than others. Assume n=3 and A initially contains identity permutation. RANDOM choices that give each

permutation. 1 2 3: 1 2 3 1 3 2 2 1 3 3 2 1 1 3 2: 1 2 2 1 3 3 2 1 2 2 3 1 3 1 1 2 1 3: 1 1 3 2 2 3 2 3 2 3 1 2 3 3 1 2 3 1: 1 1 2 1 3 1 2 2 2 2 3 3 3 1 3 3 1 2: 1 1 1 2 2 1 3 2 2 3 3 3 3 2 1: 1 2 1 2 1 1 3 2 3 3 3 2

16 RANDOMIZED ALGORITHMS Las Vegas Output is always correct. Time varies depending on random choices (or randomness in input). Challenge: Determine expected time.

Classic Examples: Randomized (pivot) quicksort takes expected time in

€

Θ n logn( ) . Universal hashing This Semester: Treaps Perfect Hashing Smallest Enclosing Disk Rabin-Karp Text Search Asides: Ethernet: http://en.wikipedia.org/wiki/Exponential_backoff List Ranking - Shared Memory and Distributed Versions

(http://ranger.uta.edu/~weems/NOTES4351/09notes.pdf) Monte Carlo Correct solution with some (large) probability. Use repetition to improve odds.

Usually has one-sided error - one of the outcomes can be wrong

Example - testing primality without factoring To check N for being prime: Randomly generate some a with 1 < a <

€

N ) If

€

Nmoda = 0, then report composite else report prime One-sided error - prime could be wrong. (Several observations from number theory are needed to make this robust for avoiding Carmichael numbers) Concept: Repeated application of Monte Carlo technique can lead to: Improved reliability Las Vegas algorithm that gives correct result “with high probability”