Algorithm Theory

(01 Introduction)

Alois HeinzHeilbronn University,

Max-Planck-Str. 39, 74081 Heilbronnheinz@hs-heilbronn.de

Sept 30 2021

aph

Organisation

• Lecture: F236 and AlgorithmTheory Room, Thursday 14.00–15.30

• Practices: F236 and AlgorithmTheory Room, Thursday 15.45–17.15

• Course material:

https://mitarbeiter.hs-heilbronn.de/∼heinz/alth.html

• Course assessment: written examination, 90 min

aph 1

Further Reading

Sedgewick: Algorithms in Java, Parts 1-4 (Fundamental Algorithms, DataStructures, Sorting, Searching), Addison-Wesley Pearson Education; ISBN:0-201-36120-5

Cormen, Leiserson, Rivest, Stein: Introduction to Algorithms, The MITPress; ISBN: 978-0262032933

Baase, Van Gelder: Computer Algorithms – Introduction to Design & Ana-lysis, Addison-Wesley; ISBN: 0-201-61244-5

Goodrich, Tamassia Data Structures and Algorithms in Java, John Wiley &Sons; ISBN: 0-471-19308-9

Levitin: Introduction to The Design & Analysis of Algorithms, Addison-WesleyISBN: 0-201-74395-7

Grimaldi: Discrete and Combinatorial Mathematics: An Applied Introduction,Addison-Wesley, ISBN: 978-3-8348-0084-8;

aph 2

Topics of Lecture

Algorithm application areas

Sorting, searching, optimization, decision making . . .

Data structures

Arrays, strings, lists, trees, graphs, matrices, heaps, . . .

Algorithm design techniques

Divide-and-conquer, greedy, dynamic programming, randomization, algo-rithm transformation, heuristics, neural computing, . . .

Objective: Being able to find efficient algorithms for any kind of applicationarea

aph 3

Design and Analysis of Algorithms

Language to describe an algorithm:

Java, Maple, English, . . .

Mathematical Instruments for measuring the complexity(in time and space):

Big O notation

aph 4

Analysis of Running Time (1)

Given program P , that computes a solution to problem description x withlength |x| in running time TP (x)

best case: lower bound for running time, often easy to find,

often not very useful in practice:

TP,best(n) = inf TP (x) |n = |x|

worst case: gives (warranted) upper bound for running time,

often easy to find, but often too pessimistic in practice:

TP,worst(n) = sup TP (x) |n = |x|

aph 5

Analysis of Running Time (2)

average case: expected running time, often difficult to find, important forpractice:

let be Q(x) the probability for the occurrence of input x with respect to all

other inputs of equal length n = |x|:

TP,average(n) =∑

x, n=|x|

TP (x)Q(x)

questions:

• what is the universe of inputs, how can we find Q(x)?• do all inputs of same length n have equal probability?

aph 6

Different Measures for the Input Size

Time complexity of program P is expressed as a function of the input sizen = |x|: TP (n).

unit cost model: it is assumed that each data element needs approximatelyequal memory space and equal time for processing.

the size of the input can be expressed number of data elements.

example: sorting a set of integer numbers

logarithmic cost model: it can be assumed that each data element needsmemory space and time for processing that depends on its (numeric) length.

(for a number n > 0 we need dlog2(n+ 1)e bits to write it down)

the total size of the input can be expressed as the sum of the logarithmiclengths of all elements = the number of all bits needed.

example: factorization of a given number into prime factors.

aph 7

Finding the Order of a given Function

Big O, Ω, Θ Notation:Big O, Ω, and Θ will give upper, lower and exact bounds for the asymptoticgrowth of a given function

Idea:don’t care about constant addents or factors when evaluating the complexity.

Rationals:

• We are interested in the asymptotic behavior when inputs get large• it can be very laborious to find exact measures• it is always possible to get a linear speed-up

(replacement of hardware or software)

Goal: measure complexity using function classes.

e.g.: O(f) is the set of functions that can be upper bounded by a fixedconstant times f , they have the complexity order of at most f .

aph 8

Θ Notation

Given g : N→ R≥0, Θ(g) is this set of functions:

Θ(g) = f : N→ R≥0 | ∃ positive constants c1, c2, n0 with

f : N→ R≥0 | 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) ∀n ≥ n0

Θ gives a tight asymptotic bound.

f ∈ Θ(g)⇒ f is big Theta of g

we write f(n) ∈ Θ(g(n)) or even f(n) = Θ(g(n)).

aph 9

Visualization of Θ Notation

aph 10

O Notation

Given g : N→ R≥0, O(g) is this set of functions:

O(g) = f : N→ R≥0 | ∃ positive constants c, n0 with

f : N→ R≥0 | 0 ≤ f(n) ≤ cg(n) ∀n ≥ n0

O gives an asymptotic upper bound.

f ∈ O(g)⇒ f is big O of g

we write f(n) ∈ O(g(n)) or even f(n) = O(g(n)).

• f(n) = Θ(g(n)) =⇒ f(n) = O(g(n)) and Θ(g) ⊂ O(g)

• Here, nothing is said about the quality of the bound

aph 11

Visualization of O-Notation

aph 12

Ω Notation

Given g : N→ R≥0, Ω(g) is this set of functions:

Ω(g) = f : N→ R≥0 | ∃ positive constants c, n0 with

f : N→ R≥0 | 0 ≤ cg(n) ≤ f(n) ∀n ≥ n0

Ω gives an asymptotic lower bound.

f ∈ Ω(g)⇒ f is big Omega of g

we write f(n) ∈ Ω(g(n)) or even f(n) = Ω(g(n))

• f(n) = Θ(g(n)) =⇒ f(n) = Ω(g(n)) and Θ(g) ⊂ Ω(g)

• Again, nothing is said about the quality of the bound

• f(n) = Θ(g(n))⇐⇒ f(n) = O(g(n)) and f(n) = Ω(g(n))

aph 13

Visualization of Ω Notation

aph 14

Hierarchies of Orders

Order Name

O(1) constant bounded functions

O(log n) logarithmic functions

O(log2 n) quadratic logarithmic functions

O(n) linear functions

O(n log n) n log n growing functions

O(n2) quadratic functions

O(n3) cubic functions

O(nk) polynomial functions k const

f is polynomially bounded, if ∃ polynomial p with f ∈ O(p).

f grows exponentially, if ∃c > 1 with f ∈ Ω(n→ cn)

aph 15

How Problem Size scales with Time

Assume: We can switch to a 10 times faster computer. Instead of problem sizep now a larger problem can be handeled in the same time:

T(n) new problem size

n 10 p

n log n (about 10) p ≈ 10 ln(p)LambertW (10 p ln(p)) p

n2 3.16 p ≈√

10 p

n3 2.15 p ≈ 3√

10 p2n 3.32 + p ≈ log 10 + p

aph 16

Divide & Conquer Method

General definition of the principle:

D&C principle = method M to solve problem P of size n:

1. (Base case) if n < d, solve P directly; otherwise

2. (Divide) divide P into smaller parts P1, . . .Pk, k ≥ 1

3. (Conquer) solve each single Pi recursively with the same method M

4. (Merge) merge to partial solutions to get a solution of P .

characteristic features of D&C methods:

• broad applicability

• Analysis of running times can be done by solving a system of recursiveequations

• Efficiency of solution can be increased by changing the number and size ofthe smaller problems, and by increasing the efficiency of the divide and/ormerge steps

aph 17

The Master Method

Used to find closed solutions for the order of a function given by a system ofrecursive equations

T (n) =

c if n < daT (n/b) + f(n) if n ≥ d

with natural number constants c ≥ 1 and d ≥ 1 and real constants a ≥ 1 andb > 1 and a real function f(n) ≥ 0 for n ≥ d. Discriminate 3 cases:

1. (∃ε > 0) f(n) ∈ O(nlogb a−ε)⇒ T (n) ∈ Θ(nlogb a)

2. (∃k ≥ 0) f(n) ∈ Θ(nlogb a(log n)k)⇒ T (n) ∈ Θ(nlogb a(log n)k+1)

3. (∃ε > 0) f(n) ∈ Ω(nlogb a+ε) ∧(∃δ < 1) (∀n ≥ d) af(n/b) ≤ δf(n)⇒ T (n) ∈ Θ(f(n))

aph 18

Problem: Evaluating a Recursive Function

Find a way to evaluate for a given n ∈ N the number a(n) of binary words oflength n with at least two 0-digits between any two 1-digits.

• find a recursive relation for a(n)

• find the most efficient way to evaluate a(n)

aph 19

Solution: Evaluating a Recursive Function

Find a way to evaluate for a given n ∈ N the number a(n) of binary words oflength n with at least two 0-digits between any two 1-digits.

• find a recursive relation for a(n):

a(n) =

n+ 1 if n ≤ 2 ,a(n− 1) + a(n− 3) else

rationale: a binary word w of length n ≥ 3 can be 0 x . . . x︸ ︷︷ ︸n−1

or 100 x . . . x︸ ︷︷ ︸n−3

.

a(n)|n=0...10 = 1, 2, 3, 4, 6, 9, 13, 19, 28, 41, 60

• find the most efficient way to evaluate a(n):

a(n) =

Mn

1

2

3

[1,1]

with M =

0 1 0

0 0 1

1 0 1

aph 20

Solution: Evaluating a Recursive Function

Mn for n = 0, . . . , 4: 1 0 0

0 1 0

0 0 1

0 1 0

0 0 1

1 0 1

0 0 1

1 0 1

1 1 1

1 0 1

1 1 1

1 1 2

1 1 1

1 1 2

2 1 3

(left) multiplication with M

• shifts all rows one position upwards

• places the sum of 1st and 3rd row into row 3

How fast is the evaluation of a(n)?

aph 21

Fast Matrix Powering

Mn =

1 . . . 0... . . . ...

0 . . . 1

if n = 0, else

M if n = 1, else

(M2)n/2 if 2|n, else

(M2)n/2 M otherwise

number of multiplications to compute the n-th power of M :

T (n) = O(log n)

aph 22

Problem: Finding Partitions

let P1,2,3(n) = # of partitions of n into pieces of size 1, 2, 3.

P1,2,3(6) = 7, because 6 = 1 + 1 + 1 + 1 + 1 + 1︸ ︷︷ ︸1

= 1 + 1 + 1 + 1 + 2︸ ︷︷ ︸2

=

1 + 1 + 2 + 2︸ ︷︷ ︸3

= 2 + 2 + 2︸ ︷︷ ︸4

= 1 + 1 + 1 + 3︸ ︷︷ ︸5

= 1 + 2 + 3︸ ︷︷ ︸6

= 3 + 3︸ ︷︷ ︸7

.

P1,2,3(n)|n=0,...,10 = 1, 1, 2, 3, 4, 5, 7, 8, 10, 12, 14

How fast can we compute P1,2,3(n)?

aph 23

Solution: Finding Partitions

P1(n)|n=0,... = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, . . .

if we pack all numbers into a formal power series, we get the generating function:

gf(P1) = 1× x0 + 1× x1 + 1× x2 + 1× x3 + . . . =1

1− x

gf(P2) = 1× x0 + 0× x1 + 1× x2 + 0× x3 + . . . =1

1− x2

gf(P3) = 1× x0 + 0× x1 + 0× x2 + 1× x2 + . . . =1

1− x3

gf(P1,2,3) = gf(P1) gf(P2) gf(P3) =1

(1− x)(1− x2)(1− x3)

=1

1− x− x2 + x4 + x5 − x6

aph 24

Solution: Finding Partitions

P1,2,3(n) = the coefficient of xn in

gf(P1,2,3) = 1/(1− x− x2 + x4 + x5 − x6)

Performing the division, we see that a(n) = P1,2,3(n) can be evaluatedrecursively:

a(n) = a(n− 1) + a(n− 2)− a(n− 4)− a(n− 5) + a(n− 6)

How can this be computed?

aph 25

Exercises

1. Write a Java method that can compute the n-th power (n ∈ N) of a given square matrix

M with BigInteger items efficiently.

2. Write an efficient Java program that reads a natural number n from the input and then

computes the number a(n) of binary words of length n with at least two 0-digits between

two 1-digits.

3. Write an efficient Java program that reads a natural number n and a positive natural

number k from the input and then computes the number a(n) of binary words of length

n with at least k 0-digits between two 1-digits.

4. Write a Java program that reads a natural number n from the input and computes the

number of partitions of n into pieces of size 2, 3, 4.

5. Write a Java program that reads a natural number n and two positive natural numbers k

and m from the input and computes the number of partitions of n into pieces of size k

and m.

aph 26