CSC401 – Analysis of AlgorithmsCSC401 – Analysis of Algorithms Chapter 1Chapter 1

Algorithm AnalysisAlgorithm AnalysisObjectives:Objectives:

Introduce algorithm and algorithm analysisIntroduce algorithm and algorithm analysisDiscuss algorithm analysis methodologiesDiscuss algorithm analysis methodologiesIntroduce pseudo code of algorithmsIntroduce pseudo code of algorithmsAsymptotic notion of algorithm efficiencyAsymptotic notion of algorithm efficiencyMathematics foundation for algorithm analysisMathematics foundation for algorithm analysisAmortization analysis techniquesAmortization analysis techniques

22

What is an algorithm ?What is an algorithm ?

AlgorithmInput Output

An algorithm is a step-by-step procedure for solving a problem in a finite amount of time.

What is algorithm analysis ?Two aspects:• Running time – How much time is taken to

complete the algorithm execution?• Storage requirement – How much memory

is required to execute the program?

33

Running Time Running Time Most algorithms Most algorithms transform input objects transform input objects into output objects.into output objects.The running time of an The running time of an algorithm typically grows algorithm typically grows with the input size.with the input size.Average case time is Average case time is often difficult to often difficult to determine.determine.We focus on the worst We focus on the worst case running time.case running time.– Easier to analyzeEasier to analyze– Crucial to applications such Crucial to applications such

as games, finance and as games, finance and roboticsrobotics

0

20

40

60

80

100

120

Runnin

g T

ime

1000 2000 3000 4000

Input Size

best caseaverage caseworst case

44

How Is An Algorithm Analyzed?How Is An Algorithm Analyzed?Two methodologiesTwo methodologies– Experimental analysisExperimental analysis

MethodMethod– Write a program implementing the Write a program implementing the

algorithmalgorithm– Run the program with inputs of varying Run the program with inputs of varying

size and compositionsize and composition– Use a method like Use a method like System.currentTimeMillis()System.currentTimeMillis()

to get an accurate measure of the to get an accurate measure of the actual running timeactual running time

– Plot the resultsPlot the results

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

0 50 100

Input Size

Tim

e (m

s)

LimitationsLimitations– It is necessary to implement the algorithm, which may It is necessary to implement the algorithm, which may

be difficultbe difficult– Results may not be indicative of the running time on Results may not be indicative of the running time on

other inputs not included in the experiment. other inputs not included in the experiment. – In order to compare two algorithms, the same In order to compare two algorithms, the same

hardware and software environments must be usedhardware and software environments must be used

55

How Is An Algorithm Analyzed?How Is An Algorithm Analyzed?– Theoretical Analysis Theoretical Analysis

MethodMethod– Uses a high-level description of the algorithm Uses a high-level description of the algorithm

instead of an implementationinstead of an implementation– Characterizes running time as a function of the Characterizes running time as a function of the

input size, input size, nn..– Takes into account all possible inputsTakes into account all possible inputs– Allows us to evaluate the speed of an algorithm Allows us to evaluate the speed of an algorithm

independent of the hardware/software independent of the hardware/software environmentenvironment

CharacteristicsCharacteristics– A description language -- Pseudo codeA description language -- Pseudo code– MathematicsMathematics

66

PseudocodePseudocode

High-level description High-level description of an algorithmof an algorithmMore structured than More structured than English proseEnglish proseLess detailed than a Less detailed than a programprogramPreferred notation for Preferred notation for describing algorithmsdescribing algorithmsHides program design Hides program design issuesissues

Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A

currentMax A[0]for i 1 to n 1 do

if A[i] currentMax thencurrentMax A[i]

return currentMax

Example: find max element of an array

77

Pseudocode DetailsPseudocode DetailsControl flowControl flow– ifif …… thenthen …… [ [elseelse …]…]

– switch switch ……

– whilewhile …… dodo ……

– repeatrepeat …… untiluntil ……

– forfor …… dodo ……

– Indentation replaces Indentation replaces braces braces

Method declarationMethod declarationAlgorithm Algorithm methodmethod ( (argarg [, [, argarg…])…])

InputInput ……

OutputOutput ……

Array indexing: A[i]Array indexing: A[i]

Method callMethod callvar.method var.method ((argarg [, [, argarg…])…])

Return valueReturn valuereturnreturn expressionexpression

ExpressionsExpressions AssignmentAssignment

(like (like in Java) in Java) Equality testingEquality testing

(like (like in Java) in Java)nn22 Superscripts and other Superscripts and other

mathematical mathematical formatting allowedformatting allowed

88

The Random Access Machine (RAM) ModelThe Random Access Machine (RAM) Model

A A CPUCPU

An potentially An potentially unbounded bank of unbounded bank of memorymemory cells, each of cells, each of which can hold an which can hold an arbitrary number or arbitrary number or charactercharacter

01

2

Memory cells are numbered and accessing Memory cells are numbered and accessing any cell in memory takes unit time.any cell in memory takes unit time.

99

Primitive OperationsPrimitive OperationsBasic computations Basic computations performed by an algorithmperformed by an algorithm

Identifiable in pseudocodeIdentifiable in pseudocode

Largely independent from Largely independent from the programming languagethe programming language

Exact definition not Exact definition not important (we will see why important (we will see why later)later)

Assumed to take a constant Assumed to take a constant amount of time in the RAM amount of time in the RAM modelmodel

Examples:Examples:– Evaluating an Evaluating an

expressionexpression– Assigning a value to Assigning a value to

a variablea variable– Performing an Performing an

arithmetic operationarithmetic operation– Comparing two Comparing two

numbersnumbers– Indexing into an Indexing into an

arrayarray– Following an object Following an object

referencereference– Calling a methodCalling a method– Returning from a Returning from a

methodmethod

1010

Counting Primitive OperationsCounting Primitive OperationsBy inspecting the pseudocode, we can determine By inspecting the pseudocode, we can determine the maximum number of primitive operations the maximum number of primitive operations executed by an algorithm, as a function of the executed by an algorithm, as a function of the input sizeinput size

AlgorithmAlgorithm arrayMaxarrayMax((AA, , nn))

# operations# operations

currentMaxcurrentMax AA[0][0] 22forfor ii 11 toto nn 1 1 dodo 11 nn

ifif AA[[ii] ] currentMaxcurrentMax thenthen 2(2(nn 1) 1)currentMaxcurrentMax AA[[ii]] 2(2(nn 1) 1)

{ increment counter { increment counter ii } } 2(2(nn 1) 1)returnreturn currentMaxcurrentMax 11

TotalTotal 7 7nn 2 2

1111

Estimating Running TimeEstimating Running TimeAlgorithm Algorithm arrayMaxarrayMax executes executes 77nn 2 2 primitive primitive operations in the worst case. operations in the worst case.

Define:Define:aa = Time taken by the fastest primitive operation= Time taken by the fastest primitive operation

bb = Time taken by the slowest primitive operation= Time taken by the slowest primitive operation

Let Let TT((nn)) be the worst-case time of be the worst-case time of arrayMax.arrayMax. ThenThen

a a (7(7nn 1) 1) TT((nn)) bb(7(7nn 1) 1)

Hence, the running time Hence, the running time TT((nn)) is bounded by is bounded by two linear functionstwo linear functions

1212

Growth Rate of Running TimeGrowth Rate of Running TimeChanging the hardware/ software environment Changing the hardware/ software environment – Affects Affects TT((nn)) by a constant factor, but by a constant factor, but– Does not alter the growth rate of Does not alter the growth rate of TT((nn))

The linear growth rate of the running time The linear growth rate of the running time TT((nn)) is is an intrinsic property of algorithm an intrinsic property of algorithm arrayMaxarrayMax

Growth rates of Growth rates of functions:functions:– Linear Linear nn– Quadratic Quadratic nn22

– Cubic Cubic nn33

In a log-log chart, the In a log-log chart, the slope of the line slope of the line corresponds to the corresponds to the growth rate of the growth rate of the functionfunction

1E+01E+21E+41E+61E+8

1E+101E+121E+141E+161E+181E+201E+221E+241E+261E+281E+30

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

T(n

)

Cubic

Quadratic

Linear

1313

Constant FactorsConstant FactorsThe growth rate is The growth rate is not affected bynot affected by– constant factors or constant factors or – lower-order termslower-order terms

ExamplesExamples– 101022nn 101055 is a linear is a linear

functionfunction– 101055nn22 10 1088nn is a is a

quadratic functionquadratic function1E+01E+21E+41E+61E+8

1E+101E+121E+141E+161E+181E+201E+221E+241E+26

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

T(n

)

Quadratic

Quadratic

Linear

Linear

1414

Big-Oh NotationBig-Oh NotationGiven functions Given functions ff((nn) ) and and gg((nn)), , we say that we say that ff((nn) ) is is OO((gg((nn)))) if if there are positive constantsthere are positive constantscc and and nn00 such that such that

ff((nn)) cgcg((nn) ) for for n n nn00

Example: Example: 22nn 1010 is is OO((nn))– 22nn 1010 cncn

– ((cc 2) 2) n n 1010

– n n 1010((cc 2) 2)

– Pick Pick c c 3 3 and and nn0 0 1010

Example: the function Example: the function nn22 is is not not OO((nn))– nn22 cncn

– n n cc– The above inequality cannot The above inequality cannot

be satisfied since be satisfied since cc must be must be a constant a constant

1

10

100

1,000

10,000

1 10 100 1,000n

3n

2n+10

n

1

10

100

1,000

10,000

100,000

1,000,000

1 10 100 1,000n

n̂ 2

100n

10n

n

1515

More Big-Oh ExamplesMore Big-Oh Examples7n-27n-2

7n-2 is O(n)7n-2 is O(n)need c > 0 and nneed c > 0 and n00 1 such that 7n-2 1 such that 7n-2 c c•n for n •n for n n n00

this is true for c = 7 and nthis is true for c = 7 and n00 = 1 = 1 3n3n33 + 20n + 20n22 + 5 + 5

3n3n33 + 20n + 20n22 + 5 is O(n + 5 is O(n33)) need c > 0 and nneed c > 0 and n00 1 such that 3n 1 such that 3n33 + 20n + 20n22 + 5 + 5

cc•n•n33 for n for n n n00 this is true for c = 4 and nthis is true for c = 4 and n00 = 21 = 21

3 log n + log log n3 log n + log log n 3 log n + log log n is O(log n)3 log n + log log n is O(log n) need c > 0 and nneed c > 0 and n00 1 such that 3 log n + log log 1 such that 3 log n + log log

n n c c•log n for n •log n for n n n00 this is true for c = 4 and nthis is true for c = 4 and n00 = 2 = 2

1616

Big-Oh and Growth RateBig-Oh and Growth RateThe big-Oh notation gives an upper bound on The big-Oh notation gives an upper bound on the growth rate of a functionthe growth rate of a function

The statement “The statement “ff((nn) ) is is OO((gg((nn))))” means that the ” means that the growth rate of growth rate of ff((nn) ) is no more than the growth is no more than the growth rate of rate of gg((nn))

We can use the big-Oh notation to rank We can use the big-Oh notation to rank functions according to their growth ratefunctions according to their growth rate

ff((nn) ) is is OO((gg((nn)))) gg((nn) ) is is OO((ff((nn))))

gg((nn) ) grows moregrows more YesYes NoNo

ff((nn) ) grows moregrows more NoNo YesYes

Same growthSame growth YesYes YesYes

1717

Big-Oh RulesBig-Oh RulesIf is If is ff((nn)) a polynomial of degree a polynomial of degree dd, then , then ff((nn)) is is OO((nndd)), i.e.,, i.e.,

1.1.Drop lower-order termsDrop lower-order terms

2.2.Drop constant factorsDrop constant factors

Use the smallest possible class of Use the smallest possible class of functionsfunctions

– Say “Say “22nn is is OO((nn))”” instead of “instead of “22nn is is OO((nn22))””

Use the simplest expression of the classUse the simplest expression of the class– Say “Say “33nn 55 is is OO((nn))”” instead of “instead of “33nn 55 is is

OO(3(3nn))””

1818

Asymptotic Algorithm AnalysisAsymptotic Algorithm Analysis

The asymptotic analysis of an algorithm The asymptotic analysis of an algorithm determines the running time in big-Oh notationdetermines the running time in big-Oh notationTo perform the asymptotic analysisTo perform the asymptotic analysis

– We find the worst-case number of primitive operations We find the worst-case number of primitive operations executed as a function of the input sizeexecuted as a function of the input size

– We express this function with big-Oh notationWe express this function with big-Oh notation

Example:Example:– We determine that algorithm We determine that algorithm arrayMaxarrayMax executes at executes at

most most 77nn 2 2 primitive operationsprimitive operations– We say that algorithm We say that algorithm arrayMaxarrayMax “runs in “runs in OO((nn) ) time”time”

Since constant factors and lower-order terms are Since constant factors and lower-order terms are eventually dropped anyhow, we can disregard eventually dropped anyhow, we can disregard them when counting primitive operationsthem when counting primitive operations

1919

Summation:Summation:

Geometric summation:Geometric summation:

Natural summation:Natural summation:

Harmonic number:Harmonic number:

Split summation:Split summation:

SummationsSummations

a

aaaaa

nn

n

i

i

1

11

12

0

2

)1(321

0

nnni

n

i

)()2()1()()( bfafafafifb

ai

nnii

ln1

3

1

2

11

1

0

n

ki

k

i

n

i

ififif110

)()()(

aaaaa n

i

i

1

11 2

0

2020

LogarithmsLogarithms– properties of logarithms:properties of logarithms:

loglogbb(xy) = log(xy) = logbbx + logx + logbbyy

loglogbb (x/y) = log (x/y) = logbbx - logx - logbbyy

loglogbbxa = alogxa = alogbbxx

loglogbba = loga = logxxa/loga/logxxbb

ExponentsExponents– properties of exponentialsproperties of exponentials::

aa(b+c)(b+c) = a = abba a cc

aabcbc = (a = (abb))cc

aabb /a /acc = a = a(b-c)(b-c)

b = a b = a loglogaabb

bbcc = a = a c*logc*logaabb

Floor and ceiling functionsFloor and ceiling functions– the largest integer less than or equal to x– the smallest integer greater than or equal to x

Logarithms and Exponents Logarithms and Exponents

x x

2121

By Example -- counterexampleBy Example -- counterexampleContrapositiveContrapositive– Principle: To justify “if p is true, then q is true”, show “if Principle: To justify “if p is true, then q is true”, show “if

q is not true, then p is not true”.q is not true, then p is not true”.– Example: To justify “if ab is odd, a is odd or b is even”, Example: To justify “if ab is odd, a is odd or b is even”,

assume “’a is odd or b is odd’ is not true”, then “a is assume “’a is odd or b is odd’ is not true”, then “a is even and b is odd”, then “ab is even”, then “’ab is odd’ even and b is odd”, then “ab is even”, then “’ab is odd’ is not true”is not true”

ContradictionContradiction– Principle: To justify “p is true”, show “if p is not true, Principle: To justify “p is true”, show “if p is not true,

then there exists a contradiction”.then there exists a contradiction”.– Example: To justify “if ab is odd, then a is odd or b is Example: To justify “if ab is odd, then a is odd or b is

even”, let “ab is odd”, and assume “’a is odd or b is even”, let “ab is odd”, and assume “’a is odd or b is even’ is not true”, then “a is even and b is odd”, thus even’ is not true”, then “a is even and b is odd”, thus “ab is even” which is a contradiction to “ab is odd”.“ab is even” which is a contradiction to “ab is odd”.

Proof Techniques Proof Techniques

2222

Induction -- To justify S(n) for n>=n0Induction -- To justify S(n) for n>=n0– PrinciplePrinciple

Base cases: Justify S(n) is true for n0<=n<=n1Base cases: Justify S(n) is true for n0<=n<=n1Assumption: Assume S(n) is true for n=N>=n1;Assumption: Assume S(n) is true for n=N>=n1;

or Assume S(n) is true for or Assume S(n) is true for n1<=n<=Nn1<=n<=NInduction: Justify S(n) is true for n=N+1Induction: Justify S(n) is true for n=N+1

– Example: Example: Question: Fibonacci sequence is defined as F(1)=1, Question: Fibonacci sequence is defined as F(1)=1, F(2)=1, and F(n)=F(n-1)+F(n-2) for n>2. Justify F(2)=1, and F(n)=F(n-1)+F(n-2) for n>2. Justify F(n)<2^n for all n>=1F(n)<2^n for all n>=1Proof by induction where n0=1. Let n1=2.Proof by induction where n0=1. Let n1=2.

– Base cases: For n0<=n<=n1, n=1 or n=2. If n=1, Base cases: For n0<=n<=n1, n=1 or n=2. If n=1, F(1)=1<2=2^1. If n=2, F(2)=2<4=2^2. It holds.F(1)=1<2=2^1. If n=2, F(2)=2<4=2^2. It holds.

– Assumption: Assume F(n)<2^n for n=N>=2.Assumption: Assume F(n)<2^n for n=N>=2.– Induction: For n=N+1, F(N+1)=F(N)+F(N-Induction: For n=N+1, F(N+1)=F(N)+F(N-

1)<2^N+2^(N-1)<2 2^N=2^(N+1). It holds.1)<2^N+2^(N-1)<2 2^N=2^(N+1). It holds.

Proof by Induction Proof by Induction

2323

Principle:Principle:– To prove a statement S about a loop is correct, define S in To prove a statement S about a loop is correct, define S in

terms of a series of smaller statements S0, S1, …, Sk, terms of a series of smaller statements S0, S1, …, Sk, wherewhere

The initial claim S0 is true before the loop beginsThe initial claim S0 is true before the loop beginsIf Si-1 is true before iteration i begins, then show that Si If Si-1 is true before iteration i begins, then show that Si is true after iteration i is overis true after iteration i is overThe final statement Sk is trueThe final statement Sk is true

Example: Consider the algorithm Example: Consider the algorithm arrayMaxarrayMax– Statement S: max is the maximum number when finishedStatement S: max is the maximum number when finished– A series of smaller statements:A series of smaller statements:

Si: max is the maximum Si: max is the maximum in the first i+1 elements in the first i+1 elements of the arrayof the array

S0 is true before the loopS0 is true before the loopIf Si is true, then easy to If Si is true, then easy to

show Si+1 is trueshow Si+1 is trueS=Sn-1 is also trueS=Sn-1 is also true

Proof by Loop Invariants Proof by Loop Invariants

AlgorithmAlgorithm arrayMaxarrayMax((AA, , nn))maxmax AA[0] [0] forfor ii 11 toto nn 1 1 dodo

ifif AA[[ii] ] maxmax thenthenmaxmax AA[[ii]]

returnreturn maxmax

2424

Sample space: Sample space: the set of all possible outcomes from the set of all possible outcomes from some experimentsome experiment

Probability space: Probability space: a sample space S together with a a sample space S together with a probability function that maps subsets of S to real numbers probability function that maps subsets of S to real numbers between 0 and 1between 0 and 1

Event: Each subset of A of S called an Event: Each subset of A of S called an eventeventProperties of the probability function PrProperties of the probability function Pr– Pr(Ø)=0Pr(Ø)=0– Pr(S)=1Pr(S)=1– 0<=Pr(A)<=1 for any subset A of S0<=Pr(A)<=1 for any subset A of S– If A and B are subsets of S and AIf A and B are subsets of S and AB=B=, then , then

Pr(APr(AB)=Pr(A)+Pr(B)B)=Pr(A)+Pr(B)

IndependenceIndependence– A and B are independent if Pr(AA and B are independent if Pr(AB)=Pr(A)Pr(B)B)=Pr(A)Pr(B)– AA11, A, A22, …, A, …, Ann are mutually independent if Pr(A are mutually independent if Pr(A11AA22… …

AAnn)=Pr(A)=Pr(A11)Pr(A)Pr(A22)…Pr(A)…Pr(Ann) )

Basic Probability Basic Probability

2525

Conditional probabilityConditional probability– The conditional probability that A occurs, given B, is The conditional probability that A occurs, given B, is

defined as: Pr(A|B)=Pr(Adefined as: Pr(A|B)=Pr(AB)/Pr(B), assuming Pr(B)>0B)/Pr(B), assuming Pr(B)>0

Random variablesRandom variables– Intuitively, Variables whose values depend on the Intuitively, Variables whose values depend on the

outcomes of some experimentoutcomes of some experiment– Formally, a function X that maps outcomes from some Formally, a function X that maps outcomes from some

sample space S to real numberssample space S to real numbers– Indicator random variable: a variable that maps outcomes Indicator random variable: a variable that maps outcomes

to either 0 or 1to either 0 or 1

Expectation:a random variable has random valuesExpectation:a random variable has random values– Intuitively, average value of a random variableIntuitively, average value of a random variable– Formally, expected value of a random variable X is Formally, expected value of a random variable X is

defined as E(X)=defined as E(X)=xxxPr(X=x)xPr(X=x)– Properties:Properties:

Linearity: E(X+Y) = E(X) + E(Y)Linearity: E(X+Y) = E(X) + E(Y)Independence: If X and Y are independent, that is, Independence: If X and Y are independent, that is, Pr(X=x|Y=y)=Pr(X=x), then E(XY)=E(X)E(Y)Pr(X=x|Y=y)=Pr(X=x), then E(XY)=E(X)E(Y)

Basic Probability Basic Probability

2626

Computing Prefix AveragesComputing Prefix AveragesWe further illustrate We further illustrate asymptotic analysis with asymptotic analysis with two algorithms for prefix two algorithms for prefix averagesaveragesThe The ii-th prefix average of -th prefix average of an array an array XX is average of is average of the first the first ((ii 1) 1) elements of elements of XX::AA[[ii]] XX[0] [0] XX[1] [1] … …

XX[[ii])/(])/(ii+1)+1)

Computing the array Computing the array AA of of prefix averages of prefix averages of another array another array XX has has applications to financial applications to financial analysisanalysis

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7

X

A

2727

Prefix Averages (Quadratic)Prefix Averages (Quadratic)The following algorithm computes prefix The following algorithm computes prefix averages in quadratic time by applying the averages in quadratic time by applying the definitiondefinition

AlgorithmAlgorithm prefixAverages1prefixAverages1((X, nX, n))InputInput array array XX of of nn integers integersOutputOutput array array AA of prefix averages of of prefix averages of XX #operations#operations

AA new array of new array of nn integers integers nnforfor ii 00 toto nn 1 1 dodo nn

ss XX[0] [0] nnforfor jj 11 toto ii dodo 1 1 2 2 …… ( (nn 1) 1)

ss ss XX[[jj]] 1 1 2 2 …… ( (nn 1) 1)AA[[ii]] ss ( (ii 1) 1) nn

returnreturn A A 11

2828

Arithmetic ProgressionArithmetic ProgressionThe running time of The running time of prefixAverages1 prefixAverages1 isisOO(1 (1 2 2 ……nn))

The sum of the first The sum of the first nn integers is integers is nn((nn 1) 1) 22– There is a simple visual There is a simple visual

proof of this factproof of this fact

Thus, algorithm Thus, algorithm prefixAverages1 prefixAverages1 runs in runs in OO((nn22) ) time time 0

1

2

3

4

5

6

7

1 2 3 4 5 6

2929

Prefix Averages (Linear)Prefix Averages (Linear)The following algorithm computes prefix The following algorithm computes prefix averages in linear time by keeping a running averages in linear time by keeping a running sumsum

AlgorithmAlgorithm prefixAverages2prefixAverages2((X, nX, n))InputInput array array XX of of nn integers integersOutputOutput array array AA of prefix averages of of prefix averages of XX #operations#operations

AA new array of new array of nn integers integers nnss 0 0 11forfor ii 00 toto nn 1 1 dodo nn

ss ss XX[[ii]] nnAA[[ii]] ss ( (ii 1) 1) nn

returnreturn A A 11Algorithm Algorithm prefixAverages2 prefixAverages2 runs in runs in OO((nn) ) time time

3030

AmortizationAmortizationAmortizationAmortization– Typical data structure supports a wide variety of Typical data structure supports a wide variety of

operations for accessing and updating the elementsoperations for accessing and updating the elements– Each operation takes a varying amount of running timeEach operation takes a varying amount of running time– Rather than focusing on each operationRather than focusing on each operation– Consider the interactions between all the operations by Consider the interactions between all the operations by

studying the running time of a series of these operationsstudying the running time of a series of these operations

– Average the operations’ running timeAverage the operations’ running time Amortized running timeAmortized running time– The amortized running time of an operation within a series The amortized running time of an operation within a series

of operations is defined as the worst-case running time of of operations is defined as the worst-case running time of the series of operations divided by the number of the series of operations divided by the number of operationsoperations

– Some operations may have much higher actual running Some operations may have much higher actual running time than its amortized running time, while some others time than its amortized running time, while some others have much lowerhave much lower

3131

The Clearable Table Data StructureThe Clearable Table Data Structure The clearable table The clearable table – An ADT An ADT

Storing a table of elements Storing a table of elements

Being accessing by their index in the tableBeing accessing by their index in the table

– Two methods:Two methods:add(e)add(e) -- add an element -- add an element ee to the next available cell to the next available cell in the tablein the table

clear()clear() -- empty the table by removing all elements -- empty the table by removing all elements

Consider a series of operations (Consider a series of operations (addadd and and clearclear) performed on a clearable table ) performed on a clearable table SS– Each Each addadd takes O(1) takes O(1)– Each Each clearclear takes O(n) takes O(n)– Thus, a series of operations takes O(nThus, a series of operations takes O(n22), ),

because it may consist of only because it may consist of only clearclearss

3232

Amortization AnalysisAmortization AnalysisTheorem: Theorem: – A series of n operations on an initially empty clearable A series of n operations on an initially empty clearable

table implemented with an array takes O(n) timetable implemented with an array takes O(n) time

Proof:Proof:– Let MLet M00, M, M11, …, M, …, Mn-1n-1 be the series of operations performed be the series of operations performed

on S, where k operations are on S, where k operations are clearclear

– Let MLet Mi0i0, M, Mi1i1, …, M, …, Miik-1k-1 be the k be the k clearclear operations within the operations within the

series, and others be the (n-k) series, and others be the (n-k) addadd operations operations – Define iDefine i-1-1=-1,M=-1,Mijij takes i takes ijj-i-ij-1j-1, because at most i, because at most ijj-i-ij-1j-1-1 -1

elements are added by add operations between Melements are added by add operations between M iij-1j-1 and M and Miijj

– The total time of the series is: The total time of the series is:

(n-k) + ∑(n-k) + ∑k-1k-1j=0j=0(i(ijj-i-ij-1j-1) = n-k + (i) = n-k + (ik-1k-1 - i - i-1-1) <= 2n-k) <= 2n-k

Total time is O(n)Total time is O(n)

Amortized time is O(1)Amortized time is O(1)

3333

Accounting MethodAccounting MethodThe methodThe method– Use a scheme of credits and debits: each Use a scheme of credits and debits: each

operation pays an amount of operation pays an amount of cyber-dollarcyber-dollar– Some operations overpay --> creditsSome operations overpay --> credits– Some operations underpay --> debitsSome operations underpay --> debits– Keep the balance at any time at least 0Keep the balance at any time at least 0Example: the clearable tableExample: the clearable table– Each operation pays two cyber-dollarsEach operation pays two cyber-dollars– addadd always overpays one dollar -- one credit always overpays one dollar -- one credit– clearclear may underpay a variety of dollars may underpay a variety of dollars

the underpaid amount equals the number of the underpaid amount equals the number of addadd operations since last operations since last clear clear - 2 - 2

– Thus, the balance is at least 0Thus, the balance is at least 0– So the total cost is 2n -- may have creditsSo the total cost is 2n -- may have credits

3434

Potential FunctionsPotential FunctionsBased on energy modelBased on energy model– associate a value with the structure, Ø, associate a value with the structure, Ø,

representing the current energy staterepresenting the current energy state– each operation contributes to Ø a certain amount each operation contributes to Ø a certain amount

of energy t’ and consumes a varying amount of of energy t’ and consumes a varying amount of energy tenergy t

– ØØ00 -- the initial energy, Ø -- the initial energy, Øii -- the energy after the -- the energy after the i-th operationi-th operation

– for the i-th operationfor the i-th operationttii -- the actual running time -- the actual running time

t’t’ii -- the amortized running time -- the amortized running time

ttii = t’ = t’ii + Ø + Øi-1i-1 - Ø - Øii – Overall: T=∑(tOverall: T=∑(tii), T’=∑(t’), T’=∑(t’ii))– The total actual time T = T’ + ØThe total actual time T = T’ + Ø00 - Ø - Ønn – As long as ØAs long as Ø00 <= Ø <= Ønn, T<=T’ , T<=T’

3535

Potential FunctionsPotential FunctionsExample -- The clearable tableExample -- The clearable table– the current energy state Ø is defined as the the current energy state Ø is defined as the

number of elements in the table, thus Ø>=0number of elements in the table, thus Ø>=0– each operation contributes to Ø t’=2each operation contributes to Ø t’=2

– ØØ00 = 0 = 0

– ØØii = Ø = Øi-1i-1 + 1, if the i-th operation is + 1, if the i-th operation is addaddadd: t = t’+ Øadd: t = t’+ Øi-1i-1 - Ø - Øii = 2 - 1 = 1 = 2 - 1 = 1

– ØØii = 0, if the i-th operation is = 0, if the i-th operation is clearclearclear: t =t’+ Øclear: t =t’+ Øi-1i-1 = 2 + Ø = 2 + Øi-1i-1

– Overall: T=∑ (t), T’=∑(t’)Overall: T=∑ (t), T’=∑(t’)

– The total actual time T = T’ + ØThe total actual time T = T’ + Ø00 - Ø - Ønn – Because ØBecause Ø00 = 0 <= Ø = 0 <= Ønn, T<=T’ =2n, T<=T’ =2n

3636

Extendable ArrayExtendable ArrayADT: ADT: Extendable array is an array with extendable Extendable array is an array with extendable size. One of its methods is size. One of its methods is addadd to add an element to to add an element to the array. If the array is not full, the element is the array. If the array is not full, the element is added to the first available cell. When the array is added to the first available cell. When the array is full, the full, the addadd method performs the following method performs the following– Allocate a new array with double sizeAllocate a new array with double size– Copy elements from the old array to the new arrayCopy elements from the old array to the new array– Add the element to the first available cell in the new arrayAdd the element to the first available cell in the new array– Replace the old array with the new arrayReplace the old array with the new array

Question: Question: What is the amortization time of What is the amortization time of addadd??– Two situations of Two situations of addadd: :

add -- O(1)add -- O(1)Extend and add -- (n)Extend and add -- (n)

– Amortization analysis:Amortization analysis:Each Each add add deposits 3 dollars and spends 1 dollardeposits 3 dollars and spends 1 dollarEach extension spends k dollars from the size k to 2k Each extension spends k dollars from the size k to 2k (copy)(copy)Amortization time of Amortization time of addadd is O(1) is O(1)

3737

Relatives of Big-OhRelatives of Big-Ohbig-Omegabig-Omega– f(n) is f(n) is (g(n)) if there is a constant c > 0 (g(n)) if there is a constant c > 0

and an integer constant nand an integer constant n00 1 such that 1 such that

f(n) f(n) c c••g(n) for n g(n) for n n n00

big-Thetabig-Theta– f(n) is f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and (g(n)) if there are constants c’ > 0 and c’’ > 0 and

an integer constant nan integer constant n00 1 such that c’ 1 such that c’••g(n) g(n) f(n) f(n) c’’c’’••g(n) for n g(n) for n n n00

little-ohlittle-oh– f(n) is o(g(n)) if, for any constant c > 0, there is an integer f(n) is o(g(n)) if, for any constant c > 0, there is an integer

constant nconstant n00 0 such that f(n) 0 such that f(n) c c••g(n) for n g(n) for n n n00

little-omegalittle-omega– f(n) is f(n) is (g(n)) if, for any constant c > 0, there is an integer (g(n)) if, for any constant c > 0, there is an integer

constant nconstant n00 0 such that f(n) 0 such that f(n) c c••g(n) for n g(n) for n n n00

3838

Intuition for Asymptotic NotationIntuition for Asymptotic NotationBig-OhBig-Oh– f(n) is O(g(n)) if f(n) is asymptotically f(n) is O(g(n)) if f(n) is asymptotically less than or less than or

equalequal to g(n) to g(n)

big-Omegabig-Omega– f(n) is f(n) is (g(n)) if f(n) is asymptotically (g(n)) if f(n) is asymptotically greater than greater than

or equalor equal to g(n) to g(n)big-Thetabig-Theta– f(n) is f(n) is (g(n)) if f(n) is asymptotically (g(n)) if f(n) is asymptotically equalequal to g(n) to g(n)

little-ohlittle-oh– f(n) is o(g(n)) if f(n) is asymptotically f(n) is o(g(n)) if f(n) is asymptotically strictly lessstrictly less

than g(n)than g(n)little-omegalittle-omega– f(n) is f(n) is (g(n)) if is asymptotically (g(n)) if is asymptotically strictly greaterstrictly greater

than g(n)than g(n)

3939

Example Uses of the Relatives of Big-OhExample Uses of the Relatives of Big-Oh 5n2 is (n2)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0 let c = 5 and n0 = 1

5n2 is (n)f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0 let c = 1 and n0 = 1

5n2 is (n)f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0 0 such that f(n) c•g(n) for n n0 need 5n0

2 c•n0 given c, the n0 that satisfies this is n0 c/5 0