Lecture3RecurrenceRelationsandtheMasterTheorem!

Announcements!

• HW1isposted!

• DueFriday.

• Sections willbeTuesdays4:30-5:20,room380-381U!

• Theyareoptional.

• Butvaluable!

• SignupforPiazza!

• There’salinkonthecoursewebsite.

• CourseannouncementswillbepostedonPiazza.

Moreannouncements

Lasttime….

• Sorting:InsertionSort andMergeSort

• Analyzingcorrectnessofiterative+recursivealgs

• Via“loopinvariant”andinduction

• Analyzingrunningtimeofrecursivealgorithms

• Bywritingoutatreeandaddingupalltheworkdone.

• Howdowemeasuretheruntimeofanalgorithm?

• Worst-CaseAnalysis

• Big-OhNotation

Today

•RecurrenceRelations!• Howdowemeasuretheruntimearecursivealgorithm?

• LikeIntegerMultiplicationandMergeSort?

• TheMasterMethod• Ausefultheoremsowedon’thavetoanswerthis

questionfromscratcheachtime.

RunningtimeofMergeSort

• Let’scallthisrunningtimeT(n).• whentheinputhaslengthn.

• WeknowthatT(n)=O(nlog(n)).

• Butifwedidn’tknowthat…MERGESORT(A):

n=length(A)

if n≤ 1:

return A

L=MERGESORT(A[:n/2])

R=MERGESORT(A[n/2:])

returnMERGE(L,R)

𝑇 𝑛 ≤ 2 ⋅ 𝑇 𝑛2 + 11 ⋅ 𝑛

Fromlasttime

RecurrenceRelations

• 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 11 ⋅ 𝑛 isarecurrencerelation.

• ItgivesusaformulaforT(n) intermsofT(lessthann)

• Thechallenge:

GivenarecurrencerelationforT(n),findaclosedformexpressionforT(n).

• Forexample,T(n)=O(nlog(n))

TechnicalitiesIBaseCases

• Formally,weshouldalwayshavebasecaseswithrecurrencerelations.

• 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 11 ⋅ 𝑛with𝑇 1 = 1

isnotthesameas

• 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 11 ⋅ 𝑛with𝑇 1 = 1000000000

• However,T(1)=O(1),sosometimeswe’lljustomitit.

WhydoesT(1)=O(1)?

Siggi theStudiousStork

Pluckythe

PedanticPenguin

Onyourpre-lectureexercise

• Youplayedaroundwiththeseexamples(whennisapowerof2):

1. 𝑇 𝑛 = 𝑇 *+ + 𝑛, 𝑇 1 = 1

2. 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 𝑛, 𝑇 1 = 1

3. 𝑇 𝑛 = 4 ⋅ 𝑇 *+ + 𝑛, 𝑇 1 = 1

• Whataretheclosedforms?

[onboard]

Oneapproachforallofthese

Sizen

n/2n/2

n/4

(Size1)

…

n/4n/4n/4

n/2tn/2tn/2tn/2tn/2tn/2t

…

• The“tree”approach

fromlasttime.

• Addupallthework

doneatallthesub-

problems.

Anotherapproach:

Recursivelyapplytherelationshipabunchuntilyouseeapattern.

• 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 11 ⋅ 𝑛

• 𝑇 𝑛 = 2 ⋅ 2 ⋅ 𝑇 *1 + 22⋅*

+ + 11 ⋅ 𝑛• 𝑇 𝑛 = 4 ⋅ 𝑇 *

1 + 22 ⋅ 𝑛

• 𝑇 𝑛 = 4 ⋅ 2 ⋅ 𝑇 *3 + 22⋅*

1 + 22 ⋅ 𝑛• 𝑇 𝑛 = 8 ⋅ 𝑇 *

3 + 33 ⋅ 𝑛

• Followingthepattern…

• 𝑻 𝒏 = 𝒏 ⋅ 𝑻 𝟏 + 𝟏𝟏 ⋅ 𝒍𝒐𝒈 𝒏 ⋅ 𝒏 = 𝑶 𝒏 ⋅ 𝐥𝐨𝐠 𝒏

Formally,thisshould

beaccompanied

withaproofthat

thepatternholds!

Morenexttime.

Thisslideskippedinclass,providedhereincase

thiswaymakesmoresensetoyou.

Moreexamples

• Needlesslyrecursiveintegermultiplication

• T(n)=4T(n/2)+O(n)

• T(n)=O(n2)

• Karatsubaintegermultiplication

• T(n)=3T(n/2)+O(n)

• T(n)=O(𝑛?@AB C ≈ n1.6)

• MergeSort

• T(n)=2T(n/2)+O(n)

• T(n)=O(nlog(n))

T(n)=timetosolveaproblemofsizen.

What’sthepattern?!?!?!?!

Thesetwoare

thesameasthe

onesonyour

pre-lecture

exercise.

Themastertheorem

• A formulathatsolvesrecurrenceswhenallofthesub-problemsarethesamesize.

• We’llseeanexampleWednesdaywhennotallproblemsarethesamesize.

• ”Generalized”treemethod.

JedimasterYoda

A useful

formula it is.

Know why it works

you should.

Themastertheorem

• Suppose𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑂 𝑛H .Then

Many symbols

those are….

Threeparameters:

a:numberofsubproblems

b:factorbywhichinputsizeshrinks

d:needtodond worktocreateallthe

subproblems andcombinetheirsolutions.

Wecanalsotaken/bto

meaneither*F or

*F and

thetheoremisstilltrue.

TechnicalitiesIIIntegerdivision

• Ifnisodd,Ican’tbreakitupintotwoproblemsofsizen/2.

• However(seeCLRS,Section4.6.2),onecanshowthattheMastertheoremworksfineifyoupretendthatwhatyouhaveis:

• Fromnowonwe’llmostlyignorefloorsandceilingsinrecurrencerelations.

𝑇 𝑛 = 𝑇 𝑛2 + 𝑇 𝑛

2 + 𝑂(𝑛)

𝑇 𝑛 = 2 ⋅ 𝑇 𝑛2 + 𝑂(𝑛)

Pluckythe

PedanticPenguin

Examples(detailsonboard)

• Needlesslyrecursiveintegermult.

• T(n)=4T(n/2)+O(n)

• T(n)=O(n2)

• Karatsubaintegermultiplication

• T(n)=3T(n/2)+O(n)

• T(n)=O(nlog_2(3) ≈ n1.6)

• MergeSort

• T(n)=2T(n/2)+O(n)

• T(n)=O(nlog(n))

• Thatotherone

• T(n)=T(n/2)+O(n)

• T(n)=O(n)

𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑂 𝑛H .

a=4

b=2

d=1

a=3

b=2

d=1

a=2

b=2

d=1

a>bd

a>bd

a=bd

✓✓✓

a=1

b=2

d=1

a<bd ✓

Proofofthemastertheorem

• We’lldothesamerecursiontreethingwedidforMergeSort,butbemorecareful.

• Supposethat 𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑐 ⋅ 𝑛H.

Pluckythe

PedanticPenguin

Hangon!ThehypothesisoftheMasterTheoremwas

thetheextraworkateachlevelwasO(nd).That’sNOT

thesameaswork<=cnd forsomeconstantc.

That’strue… we’llactuallyproveaweaker

statementthatusesthishypothesisinsteadof

thehypothesisthat𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑂 𝑛H .

It’sagoodexercisetomakethisproofwork

rigorouslywiththeO()notation.

Siggi theStudiousStork

Recursiontree

Sizen

n/bn/b

(Size1)

…

n/b2

n/btn/btn/btn/btn/btn/bt

…

Level

Amountof

workatthis

level

0

#

problems

1

2

t

logb(n)

1

a

a2

at

𝑎?@AL *

Sizeof

each

problem

n

n/b

n/b2

n/bt

1

…

n/b

n/b2

n/b2n/b2

n/b2

n/b2

n/b2

𝑇 𝑛 = 𝑎 ⋅ 𝑇 𝑛𝑏 + 𝑐 ⋅ 𝑛H

𝑐 ⋅ 𝑛𝑑

𝑎+𝑐 𝑛𝑏+

H

𝑎𝑐 𝑛𝑏H

𝑎O𝑐 𝑛𝑏O

H

𝑎?@AL * 𝑐

Recursiontree

Sizen

n/bn/b

(Size1)

…

n/b2

n/btn/btn/btn/btn/btn/bt

…

Level

Amountof

workatthis

level

0

#

problems

1

2

t

logb(n)

1

a

a2

at

𝑎?@AL *

Sizeof

each

problem

n

n/b

n/b2

n/bt

1

…

n/b

n/b2

n/b2n/b2

n/b2

n/b2

n/b2

𝑇 𝑛 = 𝑎 ⋅ 𝑇 𝑛𝑏 + 𝑐 ⋅ 𝑛H

𝑐 ⋅ 𝑛𝑑

𝑎+𝑐 𝑛𝑏+

H

𝑎𝑐 𝑛𝑏H

𝑎O𝑐 𝑛𝑏O

H

𝑎?@AL * 𝑐

Totalwork(derivationonboard)isatmost:

𝑐 ⋅ 𝑛H ⋅ P 𝑎𝑏H

OQRSL(*)

OTU

Nowlet’scheckallthecases(onboard)

Evenmoregenerally,forT(n)=aT(n/b)+f(n)…

[FromCLRS]

OllietheOver-AchievingOstrich

Figureouthowtoadapt

theproofwegavetoprove

thismoregeneralversion!

UnderstandingtheMasterTheorem

• Whatdothesethreecasesmean?

• Suppose𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑂 𝑛H .Then

Theeternalstruggle

Branchingcausesthenumber

ofproblemstoexplode!

Themostworkisatthe

bottomofthetree!

Theproblemslowerin

thetreearesmaller!

Themostworkisat

thetopofthetree!

Considerourthreewarm-ups

1. 𝑇 𝑛 = 𝑇 *+ + 𝑛

2. 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 𝑛

3. 𝑇 𝑛 = 4 ⋅ 𝑇 *+ + 𝑛

Firstexample:tallandskinnytree

Sizen

n/2

n/4

n/2t

1

1. 𝑇 𝑛 = 𝑇 *+ + 𝑛, 𝑎 < 𝑏H

• Theamountofworkdoneatthe

top(thebiggestproblem)swamps

theamountofworkdoneanywhere

else.

• T(n)=O(workattop)=O(n)

Mostworkatthe

topofthetree!

WINNER

Thirdexample:bushytree

Sizen

n/2

3. 𝑇 𝑛 = 4 ⋅ 𝑇 *+ + 𝑛, 𝑎 > 𝑏H

• ThereareaHUGEnumberofleaves,andthetotalworkis

dominatedbythetimetodoworkattheseleaves.

• T(n)=O(workatbottom)=O(4depthoftree )=O(n2)

n/2n/2n/2

1 11111 111

1 11111 111

1 11111 111

1 11111 111

1 11111 111

1 111 111

1 1111 111

WINNER

Mostworkat

thebottom

ofthetree!

Secondexample:justright

2. 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 𝑛, 𝑎 = 𝑏H

• Thebranchingjustbalances

outtheamountofwork.

Sizen

n/2

n/4

1

n/2

n/4n/4n/4

11111 111

• Thesameamountofwork

isdoneateverylevel.

• T(n)=(numberoflevels)*(workperlevel)

• =log(n)*O(n)=O(nlog(n))

1

TIE!

Recap

• The”MasterMethod” makesourliveseasier.

• Butit’sbasicallyjustcodifyingacalculationwecoulddofromscratchifwewantedto.

NextTime

• Whatifthesub-problemsaredifferentsizes?

• Andwhenmightthathappen?

BEFORE NextTime

• Pre-LectureExercise4!• WhichshouldbeeasierifyoudidPre-LectureExercise3…