Algorithms and Data Structures - BFHhnr1/algdata/slides/01Analysis.pdf · Algorithm Analysis Page 1 Berner Fachhochschule - Technik und Informatik Algorithms and Data Structures Algorithm

Algorithm Analysis Page 1

Berner Fachhochschule - Technik und Informatik

Algorithms and Data Structures

Algorithm Analysis

Dr. Rolf Haenni

Fall 2008

Berner Fachhochschule Rolf Haenni

Technik und Informatik Algorithms and Data Structures

Algorithm Analysis Page 2

Outline

Textbook Overview

Analysis of Algorithm

Pseudo-Code and Primitive Operations

Growth Rate and Big-Oh Notation

Big-Oh in Algorithm Analysis

Relatives of Big-Oh



Algorithm Analysis Page 3Textbook Overview

Outline

Textbook Overview





Relatives of Big-Oh







Algorithm Design I

Part I. Fundamental Tools

1. Algorithm Analysis

1.1 Methodologies for Analyzing Algorithms1.2 Asymptotic Notation1.3 A Quick Mathematical Review1.4 Case Studies in Algorithm Analysis1.5 Amortization1.6 Experimentation1.7 Exercises

2. Basic Data Structures

2.1 Stacks and Queues2.2 Vectors, Lists, and Sequences2.3 Trees2.4 Priority Queues and Heaps2.5 Dictionaries and Hash Tables




Algorithm Design II

2.6 Java Example: Heap2.7 Exercises

3. Search Trees and Skip Lists

3.1 Ordered Dictionaries and Binary Search Trees3.2 AVL Trees3.3 Bounded-Depth Search Trees3.4 Splay Trees3.5 Skip Lists3.6 Java Example: AVL and Red-Black Trees3.7 Exercises

4. Sorting, Sets, and Selection

4.1 Merge-Sort4.2 The Set Abstract Data Type4.3 Quick-Sort4.4 A Lower Bound on Comparison-Based Sorting4.5 Bucket-Sort and Radix-Sort




Algorithm Design III4.6 Comparison of Sorting Algorithms4.7 Selection4.8 Java Example: In-Place Quick-Sort4.9 Exercises

5. Fundamental Techniques

5.1 The Greedy Method5.2 Divide-and-Conquer5.3 Dynamic Programming5.4 Exercises




Algorithm Design IV

Part II. Graph Algorithms

1. Graphs

1.1 The Graph Abstract Data Type1.2 Data Structures for Graphs1.3 Graph Traversal1.4 Directed Graphs1.5 Java Example: Depth-First Search1.6 Exercises

2. Weighted Graphs

2.1 Single-Source Shortest Paths2.2 All-Pairs Shortest Paths2.3 Minimum Spanning Trees2.4 Java Example: Dijkstra’s Algorithm2.5 Exercises

3. Network Flow and Matching




Algorithm Design V

3.1 Flows and Cuts3.2 Maximum Flow3.3 Maximum Bipartite Matching3.4 Minimum-Cost Flow3.5 Java Example: Minimum-Cost Flow3.6 Exercises

Part III. Internet Algorithmics

1. Text Processing

1.1 Strings and Pattern Matching Algorithms1.2 Tries1.3 Text Compression1.4 Text Similarity Testing1.5 Exercises

2. Number Theory and Cryptography

2.1 Fundamental Algorithms Involving Numbers




Algorithm Design VI2.2 Cryptographic Computations2.3 Information Security Algorithms and Protocols2.4 The Fast Fourier Transform2.5 Java Example: FFT2.6 Exercises

3. Network Algorithms

3.1 Complexity Measures and Models3.2 Fundamental Distributed Algorithms3.3 Broadcast and Unicast Routing3.4 Multicast Routing3.5 Exercises




Algorithm Design VII

Part IV. Additional Topics

1. Computational Geometry

1.1 Range Trees1.2 Priority Search Trees1.3 Quadtrees and k-D Trees1.4 The Plane Sweep Technique1.5 Convex Hulls1.6 Java Example: Convex Hull1.7 Exercises

2. NP-Completeness

2.1 P and NP2.2 NP-Completeness2.3 Important NP-Complete Problems2.4 Approximation Algorithms2.5 Backtracking and Branch-and-Bound2.6 Exercises




Algorithm Design VIII

3. Algorithmic Frameworks

3.1 External-Memory Algorithms

3.2 Parallel Algorithms

3.3 Online Algorithms

3.4 Exercises



Algorithm Analysis Page 13Analysis of Algorithm

Outline

Textbook Overview





Relatives of Big-Oh




Running Time of Algorithms

I An algorithm is a step-by-step procedure for solving a problemin a finite amount of time

I Most algorithms transform input objects into output objects

I The running time of an algorithm typically grows with theinput size

I The average case running time is often difficult to determineI We focus on the worst case running time

Ý Easier to analyseÝ Crucial to applications such as games, finance, robotics

I Sometimes, it is also worth to study the best case runningtime




Running Time: Example

0

80

100

Input Size

20

40

60

1000 2000 3000 4000 5000

Runn

ing

Tim

e

Best case

Average case

Worst case




Problem Classes I

I There are feasible problems, which can be solved by analgorithm efficiently

Ý Examples: sorting a list, draw a line between two points,decide whether n is prime, etc.

I There are computable problems, which can be solved by analgorithm, but not efficiently

Ý Example: finding the shortest solution for large sliding puzzles

I There are undecidable problems, which can not be solved byan algorithm

Ý Example: for a given a description of a program and an input,decide whether the program finishes running or will run forever(halting problem)




Problem Classes II

All Problems

ComputableProblems

FeasibleProblems




Experimental Studies

I There are two ways to analyse the running time of analgorithm:

Ý Experimental studiesÝ Theoretical analysis

I Experiments usually involve the following major steps:

Ý Write a program implementing the algorithmÝ Run the program with inputs of varying size and compositionÝ Use a method like System.currentTimeMillis() to get an

accurate measure of the actual running timeÝ Plot the results




Experimental Studies: Example

0

1000

5000

8000

10000

2000

3000

4000

6000

7000

9000

Input Size1000 2000 3000 4000 5000

Runn

ing

Tim

e (m

s)




Experimental Setup I

To setup an experiment, perform the following steps with care

I Choosing the question

Ý Estimate the asymptotic running time in the average caseÝ Test which of two algorithm is fasterÝ If an algorithm depends on parameters, find the best valuesÝ For approximation algorithms, evaluate the quality of the

approximation

I Deciding what to measure

Ý Running time (system dependent)Ý Number of memory referencesÝ Number of primitive (e.g. arithmetic) operations/comparisons




Experimental Setup II

I Generating test data

Ý Enough samples for statistically significant resultsÝ Samples of varying sizesÝ Representative of the kind of data expected in practice

I Coding the solution and performing the experiment

Ý Reproducible resultsÝ Keep note of the details of the computational environment

I Evaluating the test results statistically




The Power TestThe power test is a methodology to approximate the growth rateof a polynomial (feasible) algorithm

I Actual running time: t(n) ≈ b·nc , b =???, c =???I Samples: (xi , yi )

Ý xi = problem sizeÝ yi = t(xi ) = actual running time

I Transform xi into x ′i = log xi and yi into y ′

i = log yi

I Plot (x ′i , y

′i ) → Log-Log-Scale

I If possible, draw a line through samples → polynomial

I b = the line’s y -axis intercept, c = slope of the line

I If the growth of the points increases → super-polynomial

I If the plotted points converge to a constant → sublinear




The Power Test

0 5 100

1

5

8

10

Polynomialb=2, c=5/6

Sublinear

Super-Polynomial

t(n) = 2n 5/6

1 2 3=1000

4 6=1Mio.

7 8 9=1Mia.

2

1000 = 3

4

1Mio. = 6

7

1Mia. = 9




Limits of Experiments

I It is necessary to implement the algorithm, which may bedifficult

I Results may not be indicative of the running time on otherinputs not included in the experiment

I The running times may depend strongly on the chosen setting(hardware, software, programming language)

I In order to compare two algorithms, the same hardware andsoftware environments must be used




Theoretical Analysis

I Uses a high-level description of the algorithm instead of animplementation

I Describes the running time as a function of the input size n:

Ý T (n) = number of primitive operations

I Takes into account all possible inputsI Allows to evaluate the speed of an algorithm independent of

Ý hardwareÝ softwareÝ programming languageÝ actual implementation



Algorithm Analysis Page 26Pseudo-Code and Primitive Operations

Outline

Textbook Overview





Relatives of Big-Oh




Pseudo-Code

I Pseudo-code allows a high-level description of an algorithm

I More structured than English prose, but less detailed than aprogram

I Preferred notation for describing algorithms

I Hides program design issues and details of a particularlanguage




Pseudo-Code: Example

Find the maximum element of an array:

Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of AcurrentMax ← A[0]for i ← 1 to n − 1 doif A[i ] > currentMax then

currentMax ← A[i ]return currentMax




Details of Pseudo-Code I

I The rules of using pseudo-code are very flexible, but followingsome guidelines is recommended

I Method/function declaration

Ý Algorithm method or function(arg [, arg . . .])Input ...Output ...

I Control flow

Ý if ... then ... [else ...]Ý while ... do ...Ý repeat ... until ...Ý for ... do ...Ý Indentation replaces braces resp. begin ... end




Details of Pseudo-Code II

I Expressions

Ý Assignments: ← (like = in Java)Ý Equality testing: = (like == in Java)Ý Superscript like n2 and other mathematical notation is allowedÝ Array access: A[i − 1]

I Method/function call

Ý object.method(arg [, arg . . .])Ý function(arg [, arg . . .])

I Return value

Ý return expression




The Random Access Machine Model

I To study algorithms in pseudo-code analytically, we need atheoretical model of a computing machine

I A random access machine consists of

Ý a CPUÝ a bank of memory cells, each of which can hold an arbitrary

number (of any size) or character

I The size of the memory is unlimited

I Memory cells are numbered and accessing any cell in memorytakes 1 unit time (= random access)

I All primitive operations of the CPU take 1 unit time




Primitive Operations

I The primitive operations of a random access machine are theonce we allow in a pseudo-code algorithm

Ý Evaluating an expression (arithmetic operations, comparisons)Ý Assigning a value to a variableÝ Indexing into an arrayÝ Calling a method/functionÝ Returning from a method/function

I Largely independent from the programming language

I By inspecting the pseudo-code, we can determine themaximum (or minimum) number of primitive operationsexecuted by an algorithm, as a function of the input size




Primitive Operations: Example

Algorithm arrayMax(A, n)currentMax ← A[0]for i ← 1 to n − 1 doif A[i ] > currentMax then

currentMax ← A[i ]increment counter ireturn currentMax

# Operations22 + n2(n − 1)2(n − 1)2(n − 1)1

Hence, the total number of primitive operations of arrayMax isT (n) = 7n − 1.



Algorithm Analysis Page 34Growth Rate and Big-Oh Notation

Outline

Textbook Overview





Relatives of Big-Oh




Estimating Actual Running Times

I In reality, the time to execute a primitive operation differs

Ý Let a be the time taken by the fastest primitive operationÝ Let b be the time taken by the slowest primitive operation

I If t(n) denotes the actual running time of the algorithm on aconcrete machine for a problem of size n, we get

a·T (n) ≤ t(n) ≤ b·T (n)

I For arrayMax , this means that the actual running time

a·(7n − 1) ≤ t(n) ≤ b·(7n − 1)

is bounded by two linear functions




Growth Rates of Running Times

I Changing the hardware/ software environment

Ý Affects the constants a and bÝ Therefore, affects t(n) only by a constant factorÝ Does not alter the growth rate of t(n)

I Therefore, we can consider the growth rate of T (n) as beingthe characteristic measure for evaluating an algorithm’srunning time

I We will use the terms “number of primitive operations” and“running time” interchangeably and denote them by T (n)

I The growth rate of T (n) is an intrinsic property of thealgorithm which is independent of its implementation




Polynomial Growth Rates

I Most interesting algorithms have a polynomial growth rate

Ý Linear: T (n) ≈ nÝ Quadratic: T (n) ≈ n2

Ý Cubic: T (n) ≈ n3

I In a log-log chart, a polynomial growth rate turns out to be aline (see power test)

I The slope of the line corresponds to the growth rate

I The growth rate is not affected by constant factors orlower-order terms

I Examples

Ý 102n + 10000 is linearÝ 105n2 + 108n is quadratic




Polynomial Growth Rate

1

Input Size n101

Runn

ing

Tim

e T

(n)

102 103 104 105 106 107

102

104

106

108

1010

1012

linear

cubic quadratic

1




Constant Factors

1

Input Size n101

Runn

ing

Tim

e T

(n)

102 103 104 105 106 107

102

104

106

108

1010

1012

n

1

102n +105




Big-Oh Notation

I The so-called big-Oh notation is useful to better compare andclassify the growth rates of different functions T (n)

I Given functions f (n) and g(n), we say that f (n) is O(g(n)), ifthere is a constant c > 0 and an integer n0 ≥ 1 such that

f (n) ≤ c ·g(n), for n ≥ n0

I Example 1: 2n + 10 is O(n)

Ý 2n + 10 ≤ c ·nÝ n ≥ 10/(c − 2), i.e. pick c = 3 and n0 = 10

I Example 2: n2 is not O(n)

Ý n2 ≤ c ·nÝ n ≤ c , which can not be satisfied since c is a constant




Big-Oh Example 1

1

Input Size n

Runn

ing

Tim

e T

(n) n

1

2n +10

10 100 1000

10

100

1000

3n




Big-Oh Example 2

1

Input Size n

Runn

ing

Tim

e T

(n)

n

1

100n

10 100 1000

10

100

10000

10n

1000n2




Big-Oh and Growth Rate

I The big-Oh notation gives an upper bound on the growth rateof a function

I The statement “f (n) is O(g(n))” means that the growth rateof f (n) is no more than the growth rate of g(n)

I We can use the big-Oh notation to order functions accordingto their growth rate

f (n) is O(g(n)) g(n) is O(f (n))

f (n) g(n) Yes No

f (n) g(n) No Yes

f (n) ≡ g(n) Yes Yes

I In other words, O(f (n)) can be seen as an equivalence class offunctions with equal growth rate




Big-Oh Rules

I If f (n) is a polynomial of degree d , i.e.

f (n) = a0 + a1n + a2n2 + . . . adnd

then f (n) is O(nd)

Ý drop lower-order termsÝ drop constant factorsÝ e.g. 3 + 2n2 + 5n4 is O(n4)

I Use the smallest possible equivalence class

Ý say ”2n is O(n)” instead of ”2n is O(n2)”Ý say ”3n + 5 is O(n)” instead of ”3n + 5 is O(3n)”



Algorithm Analysis Page 45Big-Oh in Algorithm Analysis

Outline

Textbook Overview





Relatives of Big-Oh




Asymptotic Algorithm Analysis

I The asymptotic analysis of an algorithm determines therunning time in big-Oh notation

I To perform the asymptotic analysis

Ý Find the worst-case number of primitive operations T (n)Ý Express this function with big-Oh notation

I Example: arrayMax

Ý We determine that algorithm arrayMax executes at mostT (n) = 7n − 1 primitive operations

Ý We say that algorithm arrayMax “runs in O(n) time”

I Since constant factors and lower-order terms are eventuallydropped anyhow, we can disregard them when countingprimitive operations




Typical Functions in Algorithms

Constant: O(1)

Logarithmic: O(log n)

Linear: O(n)

Quadratic: O(n2)

Polynomial: O(nd), d > 0

Exponential: O(bn), b > 1

Others: O(√

n), O(n log n)

Growth rate order:1 < log n <

√n < n < n log n < n2 < n3 < · · · < 2n < 3n < · · ·




Growth Rate Comparison

n log n√

n n n log n n2 n3 2n

2 1 1.4 2 2 4 8 4

4 2 2 4 8 16 64 16

8 3 2.8 8 24 64 512 256

16 4 4 16 64 256 4,096 65,536

32 5 5.7 32 160 1,024 32,768 4.29× 109

64 6 8 64 384 4,096 262,144 1.84× 1019

128 7 11 128 896 16,384 2,097,152 3.40× 1038

256 8 16 256 2,048 65,536 16,777,216 1.15× 1077

512 9 23 512 4,608 262,144 1.34× 108 1.34× 10154

1,024 10 32 1,024 10,240 1,048,576 1.07× 109 1.79× 10308

Atoms in the universe: ca. 1080




The Importance of Asymptotics

1 operation/µs 256 ops./µs

Running Time Maximum Problem Size n New Size

400n 2, 500 150, 000 9, 000, 000 256n

20n log n 4, 096 166, 666 7, 826, 087 256n log n7+log n

2n2 707 5, 477 42, 426 16n

n4 31 88 244 4n

2n 19 25 31 n + 8

1 second 1 minute 1 hour




Math You Need to Review

I Logarithms/exponentials: a = bc ⇔ logb a = c

Ý a = blogb a

I Properties of logarithms

Ý logb ac = logb a + logb cÝ logb

ac = logb a− logb c

Ý logb ac = c logb a

I Properties of exponentials

Ý ba+c = babc

Ý ba−c = (ba)/(bc)Ý bac = (ba)c = (bc)a

I Base change: logb a = logc alogc b and bc = ac loga b

I Computer science: base b = 2, i.e. log2 a = log a



Algorithm Analysis Page 51Relatives of Big-Oh

Outline

Textbook Overview





Relatives of Big-Oh




Relatives of Big-Oh I

I Big-Omega: we say that f (n) is Ω(g(n)), if there is aconstant c > 0 and an integer n0 ≥ 1 such that

f (n) ≥ c ·g(n), for n ≥ n0

I Little-Oh: we say that f (n) is o(g(n)), if there is a constantc > 0 and an integer n0 ≥ 1 such that

f (n) < c ·g(n), for n ≥ n0

I Little-Omega: we say that f (n) is ω(g(n)), if there is aconstant c > 0 and an integer n0 ≥ 1 such that

f (n) > c ·g(n), for n ≥ n0




Relatives of Big-Oh II

I Big-Theta: we say that f (n) is Θ(g(n)), if there are constantsc ′ > 0 and c ′′ > 0 and an integer n0 ≥ 1 such that

c ′·g(n) ≤ f (n) ≤ c ′′·g(n), for n ≥ n0

I The intuition behind the big-Oh relatives are

Ý Big-Oh: f (n) is asymptotically less or equal to g(n)Ý Little-Oh: f (n) is asymptotically strictly less than g(n)Ý Big-Omega: f (n) is asymptotically greater or equal to g(n)Ý Little-Omega: f (n) is asymptotically strictly greater than g(n)Ý Big-Theta: f (n) is asymptotically equal to g(n)

I The big-Theta notation is the most precise description of thegrowth rate, but the big-Oh notation is more commonly used



Documents

Algorithms and Data Structures - BFHhnr1/algdata/slides/01Analysis.pdf · Algorithm Analysis Page 1 Berner Fachhochschule - Technik und Informatik Algorithms and Data Structures Algorithm