Lecture02 Analysis

8/8/2019 Lecture02 Analysis

1/37

Data Structures and Algorithms

Analysis of Algorithms


2/37

Outline! Running time! Pseudo-code! Big-oh notation! Big-theta notation! Big-omega notation! Asymptotic algorithm analysis


3/37

Analysis of Algorithms

Algorithm Input Output

An algorithm is a step-by-step procedure forsolving a problem in a finite amount of time.


4/37

Running Time! Most algorithms transform inputobjects into output objects.

! Measure of goodness: Running time Space

! The running time of an algorithmtypically grows with the input sizeand other factors:

Hardware environments: processor,memory, disk. Software environments: OS, compiler.

! Focus: input size vs. running time.


5/37

Experimental Studies! Write a program

implementing thealgorithm

! Run the program withinputs of varying size andcomposition

! Use a method likeSystem.currentTimeMillis()toget an accurate measureof the actual running time

! Plot the results


6/37

Running time: worst case! Average case time is oftendifficult to determine.

! We focus on the worst case

running time. Easier to analyze Crucial to applications such as

games, finance and robotics


7/37

Limitations of Experiments! It is necessary to implement the

algorithm, which may be difficult

! Results may not be indicative of therunning time on other inputs not includedin the experiment.

! In order to compare two algorithms, thesame hardware and softwareenvironments must be used


8/37

Theoretical Analysis! Find alternative method.! Ideally: characterizes running time as a

function of the input size, n .! Uses a high-level description of the algorithminstead of an implementation

! Takes into account all possible inputs! Allows us to evaluate the speed of algorithms

independent of the hardware/softwareenvironment


9/37

Pseudocode! Mix of natural language and

programming constructs:human reader oriented.

! High-level description of analgorithm

! Less detailed than aprogram

! Preferred notation fordescribing algorithms

!Hides program designissues

Algorithm arrayMax ( A , n )Input array A of n integersOutput maximum element of A

currentMax A [0]for i 1 to n - 1 do

if A [i] > currentMax thencurrentMax A [i ]return currentMax

Example: find maxelement of an array


10/37

Pseudocode Details! Control flow

if then [else ] while do repeat until for do Indentation replaces braces

! Method declarationAlgorithm method (arg [, arg ])

Input Output

! Method callvar.method (arg [, arg ])

! Return valuereturn expression

! Expressions Assignment

(like = in C++/Java)= Equality testing

(like == in C++/Java)n 2 Superscripts and other

mathematicalformatting allowed


11/37

Primitive Operations! Basic computationsperformed by an algorithm

! Identifiable in pseudocode! Largely independent from the

programming language! Exact definition not important! Assumed to take a constant

amount of time in the RAMmodel

! Examples: Evaluating an

expression Assigning a value

to a variable Indexing into an

array Calling a method Returning from a

method


12/37

The Random Access Machine

(RAM) Model! A CPU

! An potentially unboundedbank of memory cells,each of which can hold anarbitrary number orcharacter

01

2

! Memory cells are numbered and accessingany cell in memory takes unit time.


13/37

Counting PrimitiveOperations! By inspecting the pseudocode, we can determine the

maximum number of primitive operations executed byan algorithm, as a function of the input size

Algorithm arrayMax ( A , n ) # operationscurrentMax A [0] 2for i 1 to n - 1 do n

if A [i] > currentMax then 2(n - 1) currentMax A [i ] [0, 2(n - 1)]

{ increment counter i } 2( n - 1)return currentMax 1

Total [5n - 1, 7 n - 3]


14/37

Worst case analysis! Average case analysis is typically challenging: Probability distribution of inputs.

! We focus on the worst case analysis: will perform well

on every case.


15/37

Estimating Running Time! Algorithm arrayMax executes 7n - 3 primitive

operations in the worst case. Define:a = Time taken by the fastest primitive operationb = Time taken by the slowest primitive operation

! Let T (n ) be worst-case time of arrayMax. Thena (7n - 3) T (n ) b(7n - 3)

! Hence, the running time T (n ) is bounded by twolinear functions


16/37

Asymptotic Notation! Is this level of details necessary?! How important is it to computer the

exact number of primitive operations?! How important are the set of primitive

operations?


17/37

Growth Rate of Running Time! Changing the hardware/ software

environment Affects T (n ) by a constant factor, but Does not alter the growth rate of T (n )

! The linear growth rate of the running

time T (n ) is an intrinsic property of algorithm arrayMax


18/37

Constant Factors! The growth rate is

not affected by constant factors or lower-order terms

! Examples 102n + 105 is a linear

function

105

n2

+

108

n is aquadratic function


19/37

Big-Oh Notation! Given functions f (n )and g (n ), we saythat f (n ) is O( g (n )) if there are positiveconstantsc and n 0 such that

f (n ) cg (n ) for n n 0


20/37

Big-Oh Example! Example: 2n + 10 is O(n )

2n + 10 cn (c - 2) n 10 n 10 / (c - 2) Pick c = 3 and n 0 = 10


21/37

Big-Oh Example! Example: the function

n 2 is not O(n ) n 2 cn n c The above inequality

cannot be satisfiedsince c must be aconstant


22/37

More Big-Oh Examples! 7n-2

7n-2 is O(n)need c > 0 and n 0 1 such that 7n-2 cn for n n0 this is true for c = 7 and n 0 = 1

3n 3 + 20n 2 + 53n 3 + 20n 2 + 5 is O(n 3)need c > 0 and n 0 1 such that 3n 3 + 20n 2 + 5 cn 3 for n n0 this is true for c = 4 and n 0 = 21

3 log n + 53 log n + 5 is O(log n)need c > 0 and n 0 1 such that 3 log n + 5 clog n for n n0 this is true for c = 8 and n 0 = 2


23/37

Big-Oh and Growth Rate! The big-Oh notation gives an upper bound on the

growth rate of a function! The statement f (n ) is O( g (n )) means that the growth

rate of f (n ) is no more than the growth rate of g (n )! We can use the big-Oh notation to rank functions

according to their growth rate

f (n ) is O( g (n )) g (n ) is O( f (n ))

g (n ) grows more Yes No

f (n ) grows more No YesSame growth Yes Yes


24/37

Big-Oh Rules! If is f (n ) a polynomial of degree d , then f (n ) is

O(n d ), i.e.,1. Drop lower-order terms2. Drop constant factors

! Use the smallest possible class of functions Say 2n is O(n ) instead of 2n is O(n

2

) ! Use the simplest expression of the class Say 3n + 5 is O(n ) instead of 3n + 5 is O(3n )


25/37

Asymptotic Algorithm Analysis! The asymptotic analysis of an algorithm determinesthe running time in big-Oh notation

! To perform the asymptotic analysis We find the worst-case number of primitive operationsexecuted as a function of the input size

We express this function with big-Oh notation! Example:

We determine that algorithm arrayMax executes at most7n - 3 primitive operations

We say that algorithm arrayMax runs in O(n ) time ! Since constant factors and lower-order terms are

eventually dropped anyhow, we can disregard themwhen counting primitive operations


26/37

Seven Important Functions! Seven functions thatoften appear inalgorithm analysis:

Constant 1

Logarithmic

log n Linear n N-Log-N n log n Quadratic n 2 Cubic n 3 Exponential 2n

! In a log-log chart, theslope of the linecorresponds to thegrowth rate of thefunction


27/37

Seven Important Functions


28/37

Asymptotic Analysis! Caution: 10 100 n vs. n 2

Running Maximum Problem Size (n)

Time 1 second 1 minute 1 hour

400n 2,500 150,500 9,000,000

20nlogn 4,096 166,666 7,826,087

2n 2 707 5,477 42,426n4 31 88 244

2n 19 25 31


29/37

Computing Prefix Averages! We further illustrateasymptotic analysis withtwo algorithms for prefixaverages

! The i -th prefix average of an array X is average of thefirst (i + 1) elements of X :

A [i ] = ( X [0] + X [1] + + X [i ])/( i+1)

! Computing the array A of prefix averages of anotherarray X has applications tofinancial analysis


30/37

Prefix Averages (Quadratic)! The following algorithm computes prefix averages inquadratic time by applying the definition

Algorithm prefixAverages1 ( X, n )

Input array X of n integersOutput array A of prefix averages of X #operations A new array of n integers n for i 0 to n - 1 do n

s X [0] n

for j 1 to i do 1 + 2 + + (n - 1) s s + X [ j ] 1 + 2 + + (n - 1)

A [i ] s / (i + 1) n return A 1


31/37

Arithmetic Progression! The running time of

prefixAverages1 isO(1 + 2 + + n )

! The sum of the first n integers is n (n + 1) / 2

There is a simple visualproof of this fact

! Thus, algorithm prefixAverages1 runs inO(n 2) time


32/37

Prefix Averages (Linear)! The following algorithm computes prefix averages inlinear time by keeping a running sum

Algorithm prefixAverages2 ( X, n )

Input array X of n integersOutput array A of prefix averages of X #operations A new array of n integers n s 0 1 for i 0 to n - 1 do n

s s + X [i ] n A [i ] s / (i + 1) n

return A 1! Algorithm prefixAverages2 runs in O(n ) time


33/37

! properties of logarithms: log

b(xy) = log

bx + log

by

logb (x/y) = log bx - log bylogbxa = alog bxlogba = log xa/log xb

! properties of exponentials : a (b+c) = a ba c abc = (a b)c ab /a c = a (b-c) b = a logab bc = a c*log ab

! Summations! Logarithms and Exponents

! Proof techniques:! Induction! Counterexample!

Contradiction! Basic probability

Math you need to Review


34/37

Relatives of Big-Oh! big-Omega

f(n) is (g(n)) if there is a constant c > 0and an integer constant n 0 1 such that

f(n) cg(n) for n n0

! big-Theta f(n) is (g(n)) if there are constants c > 0 and c

> 0 and an integer constant n 0 1 such that cg

(n) f(n) cg(n) for n n0


35/37

Intuition for Asymptotic

NotationBig-Oh

f(n) is O(g(n)) if f(n) is asymptoticallyless than or equal to g(n)

big-Omega f(n) is (g(n)) if f(n) is asymptotically

greater than or equal to g(n) big-Theta

f(n) is (g(n)) if f(n) is asymptoticallyequal to g(n)


36/37

Example Uses of the

Relatives of Big-Oh

f (n) is ( g (n)) if it is (n2) and O(n2). We have already seen the former,for the latter recall that f (n) is O( g (n)) if there is a constant c > 0 and aninteger constant n0 1 such that f( n) < c g (n) for n n0

Let c = 5 and n0 = 1

5n 2 is (n 2)

f (n) is ( g (n)) if there is a constant c > 0 and an integer constant n0 1such that f( n) c g (n) for n n0

let c = 1 and n0 = 1

5n 2 is (n )

f (n) is ( g (n)) if there is a constant c > 0 and an integer constant n0 1such that f (n) c g (n) for n n0

let c = 5 and n0

= 1

5n 2 is (n 2)


37/37

References! Chapter 4: Data Structures and

Algorithms by Goodrich and Tamassia

Documents

Lecture02 Analysis