Analysis of Algorithms

COME 355

Introduction

What is an algorithm?An algorithm is a sequence of unambiguous instructions

for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

“computer”

problem

algorithm

input output

Source: Levitin

Algorithm• An algorithm is a sequence of unambiguous

instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

1-4

What is an algorithm?• Recipe, process, method, technique, procedure, routine,… with

following requirements:1. Finiteness

• terminates after a finite number of steps2. Definiteness

• rigorously and unambiguously specified3. Input

• valid inputs are clearly specified4. Output

• can be proved to produce the correct output given a valid input

5. Effectiveness• steps are sufficiently simple and basic

Historical Perspective

• Muhammad ibn Musa al-Khwarizmi – 9th century mathematician

• http://en.wikipedia.org/wiki/Al-Khwarizmi

1-6

Euclid’s AlgorithmProblem: Find gcd(m,n), the greatest common divisor of two

nonnegative, not both zero integers m and n

Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ?

Euclid’s algorithm is based on repeated application of equality

gcd(m,n) = gcd(n, m mod n)

until the second number becomes 0, which makes the problem

trivial.

Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12

1-7

Two descriptions of Euclid’s algorithm

Step 1 If n = 0, return m and stop; otherwise go to Step 2

Step 2 Divide m by n and assign the value fo the remainder to r

Step 3 Assign the value of n to m and the value of r to n. Go to Step 1.

while n ≠ 0 do

r ← m mod n

m← n

n ← r

return m

Other methods for computing gcd(m,n)

Consecutive integer checking algorithmStep 1 Assign the value of min{m,n} to t

Step 2 Divide m by t. If the remainder is 0, go to Step 3; otherwise, go to Step 4

Step 3 Divide n by t. If the remainder is 0, return t and stop; otherwise, go to Step 4

Step 4 Decrease t by 1 and go to Step 2

1-9

Other methods for gcd(m,n) [cont.]

Middle-school procedure

Step 1 Find the prime factorization of m

Step 2 Find the prime factorization of n

Step 3 Find all the common prime factors

Step 4 Compute the product of all the common prime factors and return it as gcd(m,n)

Is this an algorithm?

Example:  • If  M = 24315273(11)4 and N = 22305473(11)1(17)2 • then the greatest common divisor is • gcd(M,N) = 2 min(4,2)3min(0,1)5min(2, 4)7min(3,3)(11)min(1,4)(17)min(0,2)

        = 22305273(11)1(17)0  = 225273(11)1

Sieve of Eratosthenes

• In mathematics, the Sieve of Eratosthenes is a simple, ancient algorithm for finding all prime numbers up to a specified integer. It works efficiently for the smaller primes (below 10 million). It was created by Eratosthenes, an ancient Greek mathematician.

• A prime number is a natural number which has exactly two distinct natural number divisors: 1 and itself.

• Algorithm• To find all the prime numbers less than or equal to a given integer n by

Eratosthenes' method:

1. Create a list of consecutive integers from two to n: (2, 3, 4, ..., n).

2. Initially, let p equal 2, the first prime number.

3. Strike from the list all multiples of p less than or equal to n. (2p, 3p, 4p, etc.)

4. Find the first number remaining on the list after p (this number is the next prime); replace p with this number.

5. Repeat steps 3 and 4 until p2 is greater than n.

6. All the remaining numbers in the list are prime.

1-11

Sieve of EratosthenesInput: Integer n ≥ 2

Output: List of primes less than or equal to n

for p ← 2 to n do A[p] ← p

for p ← 2 to n do

if A[p] 0 //p hasn’t been previously eliminated from the list j ← p* p

while j ≤ n do

A[j] ← 0 //mark element as eliminated

j ← j + p

Example• To find all the prime numbers less than or equal to 30, proceed as

follows:• First generate a list of integers from 2 to 30:• 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

27 28 29 30 Strike (sift out) the multiples of 2 resulting in: • 2 3 5 7 9 11 13 15 17 19 21 23 25 27 29 The first number in the list

after 2 is 3; strike the multiples of 3 from the list to get:• 2 3 5 7 11 13 17 19 23 25 29 The first number in the list after 3 is

5; strike the remaining multiples of 5 from the list:• 2 3 5 7 11 13 17 19 23 29 The first number in the list after 5 is 7,

but 7 squared is 49 which is greater than 30 so the process is finished. The final list consists of all the prime numbers less than or equal to 30.

1-13

Example of computational problem: sorting• Statement of problem:

• Input: A sequence of n numbers <a1, a2, …, an>

• Output: A reordering of the input sequence <a´1, a´

2, …, a´n> so that a´

i

≤ a´j whenever i < j

• Instance: The sequence <5, 3, 2, 8, 3>

• Algorithms:• Selection sort• Insertion sort• Merge sort• (many others)

1-14

Selection Sort

• Input: array a[1],…,a[n]

• Output: array a sorted in non-decreasing order

• Algorithm: for i=1 to n swap a[i] with smallest of a[i],…a[n]

1-15

Why study algorithms?

• Theoretical importance• the core of computer science

• Practical importance• A practitioner’s toolkit of known algorithms• Framework for designing and analyzing algorithms for

new problems

1-16

Basic Issues Related to Algorithms

• How to design algorithms

• How to express algorithms

• Proving correctness

• Efficiency (How to analyze algorithm efficiency?)

• Theoretical analysis

• Empirical analysis

• Optimality

1-17

Algorithm design techniques/strategies

• Brute force

• Divide and conquer

• Decrease and conquer

• Transform and conquer

• Space and time tradeoffs

• Greedy approach

• Dynamic programming

• Iterative improvement

• Backtracking

• Branch and bound

1-18

Analysis of Algorithms

• How good is the algorithm?• Correctness• Time efficiency• Space efficiency

• Does there exist a better algorithm?• Lower bounds• Optimality

1-19

Fundamental data structures

• list

• array

• linked list

• string

• stack

• queue

• priority queue

• graph

• tree

• set and dictionary

1-20

Some Typical Problems

• Sorting• Searching• Shortest paths in a graph• Minimum spanning tree• Primality testing• Traveling salesman problem• Knapsack problem• Chess• Towers of Hanoi• Program termination

Some Typical Problems

Fourier TransformInput: A sequence of n real or

complex values h_i, 0 <= i <= n-1, sampled at uniform

intervals from a function h. Problem: Compute the discrete

Fourier transform H of h

Nearest Neighbor Input: A set S of n points in d

dimensions; a query point q.

Problem:Which point in S is

closest to q?

SOURCE: Steve Skiena’s Algorithm Design Manual + Prof. K. Daniels (for problem descriptions, see graphics gallery at http://www.cs.sunysb.edu/~algorith)

Shortest PathInput: Edge-weighted graph G, with

start vertex s and end vertex t

Problem: Find the shortest path from s to t in G

Bin PackingInput: A set of n items with sizes

d_1,...,d_n. A set of m bins with capacity c_1,...,c_m.

Problem: How do you store the set of items using the fewest number of

bins?

Some Typical Problems

Transitive ClosureInput: A directed graph G=(V,E). Problem: Construct a graph

G'=(V,E') with edge (i,j) in E' iff there is a directed path from i to j in G. For transitive reduction, construct a small graph G'=(V,E') with a directed path from i to j in G' iff (i,j) in E.

Convex HullInput: A set S of n points in d-

dimensional space. Problem: Find the smallest convex

polygon containing all the points

of S.

Eulerian CycleInput: A graph G=(V,E).

Problem: Find the shortest tour of G visiting each edge at

least once.

Edge ColoringInput: A graph G=(V,E).

Problem: What is the smallest set of colors needed to color the edges of E such that no two edges with the same color share a vertex in

common?

Some Typical Problems

Hamiltonian CycleInput: A graph G=(V,E).

Problem: Find an ordering of the vertices such that each vertex is

visited exactly once.

CliqueInput: A graph G=(V,E).

Problem: What is the largest S that is a subset of V such that

for all x,y in S, (x,y) in E?

Tools of the Trade: Core Material

• Algorithm Design Patterns• dynamic programming, linear programming,

greedy algorithms, approximation algorithms, randomized algorithms, sweep

algorithms, (parallel algorithms)• Advanced Analysis Techniques• amortized analysis, probabilistic analysis

• Theoretical Computer Science principles• NP-completeness, NP-hardness

Asymptotic Growth of Functions

SummationsRecurrences

Sets

Probability

MATH

Proofs Calculus

Combinations Logarithms

Number Theory

Geometry TrigonometryComplex Numbers

Permutations Linear Algebra

Polynomials

Textbook

• Required: • Introduction to Algorithms

• by T.H. Corman, C.E. Leiserson, R.L. Rivest• McGraw-Hill• 2001• ISBN 0-07-013151-1

2nd Edition