Discrete Optimization 2010 Lecture 1 Introduction ...uetzm/do/DO_Lecture1.pdf · Lecture 1 Introduction / Algorithms & Spanning Trees ... 1 Introduction 2 Analysis of Algorithms

Introduction Algorithms Spanning Trees

Discrete Optimization 2010Lecture 1

Introduction / Algorithms & Spanning Trees

Marc UetzUniversity of Twente

[email protected]

Lecture 1: sheet 1 / 43 Marc Uetz Discrete Optimization

mailto:[email protected]


Outline

1 Introduction

2 Analysis of Algorithms

3 Minimum Spanning Trees



Organization

Lectures

lecture of 2 × 45 min. approx., maybe sometimes:

discussion of homework exercises

Your Assignments

Following the lectures

Reading the literature

Solving homework assignments (in teams of 2-3, please)

Hand-In (next Lecture) or email (by Monday 13:00)Homework is corrected by TA (Ruben Hoeksma)

Written exam



Organization

Grading

Homework exercises 40%

Written exam 60%

Norm

to pass, exam ≥ 5.5 and total grade ≥ 5.5

Online Material

http://www.math.utwente.nl/~uetzm/do/


http://www.math.utwente.nl/~uetzm/do/


Literature

Reader, with selcted chapters from

R. K. Ahuja, T. L. Magnanti and J. B. Orlin: Network Flows, Prentice Hall,1993.

Cook, W. J., W. H. Cunningham, W. R. Pulleyblank, and A. Schrijver,Combinatorial Optimization, Wiley, 1998.

Cormen, T. H., C. E. Leiserson, and R. L. Rivest, Introduction to Algorithms,MIT Press, 1990.

U. Faigle, W. Kern, and G. Still: Algorithmic Principles of MathematicalProgramming, Springer, 2002

V. V. Vazirani, Approximation Algorithms, Springer, 2001.

For the reader, pay e 5,- in cash or transfer to account

ABN-AMRO bank (BIC: ABNANL2A)Account nr. 405343663 (IBAN: NL33ABNA0405343663)

Reference: “OFI 500.30100, LNMB reader”



Overview & Schedule (Preliminary)

Date Session Topics

20.09. Introduction, Minimum Spanning Trees27.09. Matroids, Shortest Path Algorithms04.10. Maximum Flow Algorithms11.10. Minimum Cost Flow Algorithms18.10. Matchings & Total Unimodularity25.10. Glimpse of Integer Programming01.11. P, NP, NP-completeness08.11. NP-complete problems15.11. Approximation Algorithms22.12. Randomized Algorithms & Derandomization29.11. Primal Dual Approximation Algorithms06.12. Approximation Schemes & Inapproximability17.01. Exam (Location & Time TBA), Utrecht



Outline

1 Introduction





Optimization Problems

Definition

An instance I = (S , f ) of an optimization problem P is

S = set of solutions

f : S → < objective function

Any s ∈ S is a solution, and f (s) its value.

Definition

A solution s∗ is an optimal solution for instance I = (S , f ) if

f (s∗) ≤ (≥) f (s) ∀s ∈ S .



Optimization Problems

Definition

An optimization problem P = set of all instances I ∈ P(sharing the same description)

Examples:

Linear Programming instances: c ∈ Qn,A ∈ Qn×m, b ∈ Qm

MST Instances: edge-weighted graph G = (V ,E , c)

Definition

A combinatorial (discrete) optimization problem P is anoptimization problem, so that for any instance I = (S , f ) ∈ P thesolution set S is finite (or at most countable)



Example: Linear Programming a.k.a. LP

Each instance of LP (standard form) is given by

c ∈ Qn,A ∈ Qn×m, b ∈ Qm

minimize cx

subject to Ax ≤ b

x ≥ 0

solutions S = x ∈ <n | x ≥ 0,Ax ≤ b values f (x) = cx for all x ∈ S

LP problem

Find solution x ∈ S minimizing cx



LP is a Combinatorial Problem

!"#$%&'(&)*&%++"#*'%,'-)+."#%/)&"%0'(&).0$+

x ≥ 0

Suffices to consider finitely many solutions

S = • = basic feasible solutions

= vertices of the polyhedron x ∈ <n | x ≥ 0,Ax ≤ b

Solved combinatorially by Dantzig’s simplex algorithm (1949)



Integer Programming a.k.a. IP

Just as LP, only restrict to x integer

Clearly: solutions set S finite (or, at most countable) by definition,as vertices in general not integer, not solved via LP



Spanning Trees

Definition: Minimal connected subgraph of given G = (V ,E )



Minimum Spanning Tree a.k.a. MST

Each instance of MST is given by

An edge-weighted undirected graph G = (V ,E , c) with

V = set of vertices / nodes

E = set of edges ⊆ v ,w | v ,w ∈ V edge costs ce for all e = v ,w ∈ E

solutions S = T ⊆ E | T is a spanning tree values f (T ) =

∑e∈T ce

MST problem

Find a minimum cost spanning tree T



Algorithms

Definition

Informally: Set of rules specifying a computational procedure,where rules are arithmetic and logical instructions (such as :=, ±,· , / , ≤, AND,OR, IF, THEN, . . . ) which, given some initial state(input), terminate in some final state (output)

Etymology

Persian Al-Khwarizmi (780-850), father ofAlgebra, wrote “On Calculation with HinduNumerals”; this was translated into latin“Algoritmi on. . . ”, later evolving into“algorithm” = “calculation method”


http://en.wikipedia.org/w/index.php?title=Muhammad_ibn_M%C5%ABs%C4%81_al-Khw%C4%81rizm%C4%AB&oldid=237168986


Algorithm Design

Algorithm = think of a procedure that computes solutions to(combinatorial) problems

Main Issues

Efficiency: How long does it take to compute a solution?

Quality: Claims about the quality of the solution?

Possibilities for analyzing algorithms

empirical

average case

worst case



Why so Pessimistic (Worst Case)?

Empirical Analysis

Computer dependent (Moore’s Law: factor 2 per 18 months)

Language dependent (C++, C#, Java, C, FORTRAN,. . . )

Compiler & Programmer dependent (Code optimization,. . . )

Average case

Distribution of problem instances?

Difficult analytically

Worst Case

Simple & sound statements: Performance guarantees!

Downside: too pessimistic (“pathological instances”)



Computation Time of an Algorithm

Definition

The computation time of an algorithm is the number of basicinstructions that the algorithm performs until termination.

Want . . .

asses computation time for problems, rather than instances

focus only on behavior for large instances (size→∞)

Need to capture

1 how to asses the problem size?

2 if size→∞, what computation time? (asymptotic analysis)



Problem Size & Data Structures

# bytes to represent the instance, depends on data structure

!"#$%&'()*+&,(-./#0*.1(2&.134(#5(6(!"#$%&'

!"#$%&'(#)$*+$(,-.$)#/&,)#0%&'(#)$*+$(,-.$)#/&,)#0 -*$0#.1),(#$2%3$,%.-2%1#

452'67#8$9,:#%$2$;)26"$;)26"$!! <$=<$="">>##?@$?@$A"A<$>$A#A<%@

B0C21#%13$'2-),5$=&'(?8$

B0C21#%13$7,.-$)*8

1

34

2

n!"#"$%"&'()*

n+!m #"$!"&'()*



Why Data Structures Matter

Consider Problem “EDGE”

Given graph G = (V ,E ), two vertices i , j ∈ V

Question: Does edge i , j exist?

Adjacency matrix A: If (aij == 1) return “yes”; else return “no”;one basic instruction only O( 1 )

Adjacency lists L: for all k ∈ Liif (k == j) return “yes”;

return “no”;at most |Li |+ 2 basic instructions O( n )

Side remark: Why would I ever use adjacency lists then?



Asymptotic upper bounds (Big-O)

Definition

g(n) ∈ O(f (n)) ⇔ There is constant c > 0 and n0 ∈ N so that

g(n) ≤ c · f (n) ∀ n ≥ n0

!"#$%&'&()*+%%,-*.'+/0"*1.(2345

6,7(/(&('/8*21/5*! !171/55*(7 " constant c and " n0 such that:

for all n# n0 g(n) " c!f(n)

g(n)

c ! f(n)

n0



Asymptotic lower bounds (big-Ω)

Definition

g(n) ∈ Ω( f (n) ) ⇔ There is constant c > 0 and n0 ∈ N so that

g(n) ≥ c · f (n) ∀ n ≥ n0

!"#$%&'&()*+',-.*/'012"*3/(45!6

7-8(1(&('19*4316*!!383166*(8 " constant c and " n0 such that:

for all n# n0 g(n) # c!f(n)

g(n)

c ! f(n)

n0



Asymptotic equivalent (big-Θ)

Definition

g(n) ∈ Θ( f (n) ) ⇔ g(n) ∈ O( f (n) ) and g(n) ∈ Ω( f (n) )

!"#$%&'&()*++#,-./(0*+-1&,23(45!6

7-8(1(&('19,4216,! !282166,,,(8,,,,,4216,! :282166,*1;,4216,!"282166

c2!f(n)

c1!f(n)

g(n)

n0



Asymptotically Insignificant (little-o)

Definition

g(n) ∈ o( f (n) ) ⇔ ∀ constants c > 0 there is n0 ∈ N so that

g(n) < c · f (n) ∀ n ≥ n0

!"#$%&'&()*++#,(-"(.-(/()*-&,0+(&&+12'3

41/(-(&('-5,.0-3,! !0/0-33,(/ " constant c>0 # n0 such that:

for all n$ n0 g(n) 6 c!f(n)

g(n)

c ! f(n)

n0



Examples (you may try to prove them all)!"#$%&'(#)*$+



Notation for Functions f ∈ . . .

!"#$%&"'(')"&%*"+%,+"-'.%"*%*/&0')"&1

!"#$ %&'()*')

!"+&,n$ +&,*-.)/0.%

!"1+&,n2c$ 3&+4+&,*-.)/0.%

&"n$ (56+.'7*-

O"n$ +.'7*-

!"n+&,n$ 85*(.+.'7*-

!"n9$ 85*:-*).%

3&+4'&0.*+

!"cn$ 7;3&'7').*+

!"n<$ =*%)&-.*+ &->%&06.'*)&-.*+



Encoding Length of a Problem

Example: Given graph G = (V ,E ) with |V | = n and |E | = m

Adjacency matrix

Encoding length `(G ) ∈ Θ( n2 ) [12n2]

Adjacency lists

Encoding length `(G ) ∈ Θ( n + m ) [n + 2m]

Notice: both encoding length of problems and computation time ofalgorithms depend on data structures! Is that a problem?



Polynomial Equivalence

Given a combinatorial optimization problem P and two differentencodings L1 and L2 (with encoding lengths `i , i = 1, 2)

Definition

Encodings L1 and L2 are polynomially equivalent if there are twopolynomial functions p1 and p2 such that, for all instances I ,

`1(I ) ≤ p1(`2(I ))

`2(I ) ≤ p2(`1(I ))

Theorem

Adjacency lists and adjacency matrix are polynomially equivalentencodings for graphs. (Proof: Exercise)



Encoding Numbers

Theorem

Given any (natural) number n, its encoding in binary is Θ( log n )

Recall, 1=1, 2=10, 3=11, 4=100, 5 = 101, . . . , 2k = 1 0000︸︷︷︸k times

,. . .

Proof: Let k be such that

2k−1 ≤ n < 2k

then binary has exactly k = blog2 nc+ 1 ∈ Θ( log2 n ) digits

Remark

All k-ary encodings are polynomially equivalent with binary, as longas k ∈ O(1) and k > 1, because logk n = log2 n/ log2 k



Polynomial Time Algorithms

Definition

Given a combinatorial optimization problem P and encoding L.Algorithm A is a polynomial time algorithm for P if

for any instance I ∈ P, A terminates with a solution s of I

there is a polynomial function p such that, if nI = |L(I )|is the encoding length of I , then

tA(I ) ∈ O( p(nI ) ) ∀ I ∈ P

where tA(I ) is the number of basic instructions of A on I

computation time is then said to be in O( p(n) )

A solves problem P if solution s is optimal for all I ∈ P

encodings don’t matter, as long as polynomially equivalent



Remarks on Polytime Algorithms

The definition

addresses the worst case (true for all instances)√

is relative to problem size, as it should be√

is asymptotic (of interest is size→∞)√

is insensitive to equivalent problem encodings√

Question

Why care about polynomial time algorithms anyhow?



Answer 1

ask him: Jack Edmonds (Photo from 2009)



Answer 2: Computation Times on 2.2 GHz

Say, we can do 2.2 · 109 operations per second (2.2 GHz)

algorithm A’s computation timea, for tA(n) =

n log n n log n n2 n3 2n

16 ≈0 ≈0 ≈0 0.002 ms 0.03 ms64 ≈0 ≈0 0.002 ms 0.12 ms 266 y

256 ≈0 0.001 ms 0.06 ms 7.6 ms 1.6 · 1060 y4.096 ≈0 0.02 ms 7.6 ms 31 s ???

65.536 ≈0 0.47 ms 2 s 78 h ???16.7 Mio ≈0 0.2 s 35 h 68066 y ???

as=second, ms=1/1000 s, h=hour, y=year



Outline

1 Introduction





Spanning Trees

T ⊆ E is a spanning tree for graph G = (V ,E ) if graph (V ,T ) isconnected and acyclic.⇔ graph (V ,T ) is connected/acyclic and |T | = |V | − 1⇔ in (V ,T ) there is a unique path between any two v ,w ∈ V



Minimum Spanning Trees (MST)

MST problem

Given an edge-weighted, connected, undirected graphG = (V ,E , c), with |V | = n and |E | = m, find a minimum weightspanning tree (MST).(cheapest connected acyclic subgraph)

!"#"$%$&'()##"#* +,-- .,/01-$

!"#$%&'()*+& !",#$%-.&&&'! % /0(12&)&()*++,+-'./00()*++,+-'./00 "/&,#$&-&"2&)&3($$

,"4$4.&50%%$53$6&)%6&)5758"5-

9%*:3;&

! !()*+&!'"',#$% -

! $6'$&50232 10$'0" %

<:3*:3;

! & ! % 2:5+&3+)3&&,#$&'-&"2&) 3($$& ="3+&1"%"1)8&303)8&5023&

1,&- "'!$ " > 1,0-4

! "

#$

%$&!%

#%

"&

$%

!&

$&

! "

#$

%$&!%

#%

"&

$%

!&

$&



Spanning Trees and Cuts

Definitions

For subset of nodes W ⊆ V , δ(W ) denotes the cut inducedby W : all edges “leaving” W [δ(W ) = δ(V \W )]

Given a spanning tree T of G , let C (e) be the cut induced bydeleting edge e from T

!"#$$%$&'()**+'#$,'-./+

!"#$%%&'()'$'*+"" *,"+" -. $'/)-0/" 1$*, 2"*3"") $)4 *35 )56". ! $)6'"

!"#$%%&'()'$'*+"" #7'#$ % ,$. *35'#5815)")*.'9"$#,'%! & -.'$'2+-6:";

<"=-)-*-5).&

! >5+'$)4 ./2."* 5=')56".'" ()*!9';" & -.'#$%%"6'*," +,-+,- -)6/#"6 24

'

98"$)-): $%%'"6:". -)'?'*,$* @%"$A" BC7'-D"D7'#5))"#*' $)6'(!';

! ?-A") $'.1$))-): *+""'5='?7'3" /." *,"')5*$*-5) .9%;

=5+ *," #/* -)'/ *,$* -. -)6/#"6 24 6"%"*-): "6:"'%*-) #

!

"

#

"$

!%

#$

&

'

(

"#

#!

#)

"%

!'

#'

!

"

#

"$

!%

#$

&

'

(

"$

#!

#)

"%

!'

#'



The Cut Condition

Theorem

Given a graph G = (V ,E ) with edge costs ce , e ∈ E , then aspanning tree T ⊆ E is an MST if and only if

ce ≤ cf for all edges e ∈ T and all edges f ∈ C (e) .

(i.e., T has the cheapest connection for any cut)

!"#$%&'$#()*+,#*-./0$#$./

!"#$%#&'()*+#, -(.%-/" !"0#$%1 2*3" #4.#(5$636 &' $('! %$

3"#, -(6/-,,*,. 3%## !" 7(*6(-,(89! *: -,4($,;< *:

# #4.#6 #! !(-,4(# #4.#6 :! =0#1'((((50#1(! 50:1

!

"

#

"$

!%

#$

&

'

(

"#

#!

#)

"%

!'

#'



The Cut Condition

Proof



The Path Condition

In a tree T , denote by PT (v ,w) the (unique) path from v to w

Theorem (Exercise)

Given a graph G = (V ,E ) with edge costs ce , e ∈ E , then aspanning tree T ⊆ E is an MST if and only if

ce ≤ cf for all edges f = v ,w ∈ E\T and all edges e ∈ PT (v ,w)

!"#$%&'$#()*+&#,*-./0$#$./

!"#$%#&'()*+#, -(.%-/" !"0#$%1 2*3" #4.#(5$636 &'$('! %$

3"#, -(6/-,,*,. 3%## !" 7(*6(-(89! *: -,4($,;< *:

# #4.#6 :=>?@+A! 7!! -,4(# #4.#6 #! B!0?@+1'((((50#1(" 50:1

C#:*,# *,(-(3%## !@((B!0?@+1(-6(3"# 0?,*D?#1(/-3" :%$& ?(3$(+E

!

"

"#

!$

%#

&

'

"%

%!

%(

"$

%'

!'% )

This does not mean that PT (v ,w) is a shortest (v ,w)-path!



Kruskal’s Algorithm (1956)

Algorithm 1: Kruskal

input : G = (V ,E , c)output: T ⊆ E , minimum spanning tree of Gsort edges such that ce1 ≤ · · · ≤ cem ;T = ∅;for (i = 1, . . . ,m) do

if (T ∪ ei ) is acyclic thenT = T ∪ ei ;

Theorem

Kruskals algorithm solves MST problem in time O( m log m + n2 ).



Kruskal’s Algorithm: Correctness

Proof that T is a tree

resulting T is acyclic by definition

for any W ⊆ V , T contains the cheapest edge from δ(W )(by the sorting), therefore T is connected (T ∩ δ(W ) 6= ∅)

Proof that T is MST (using the path condition)

any edge f = v ,w not added to T creates cycle, namelyv ,w ∪ PT (v ,w),in particular, all edges e ∈ PT (v ,w) are already in T

by the sorting of the edges ⇒ cf ≥ ce ∀e ∈ PT (v ,w)



Kruskal’s Algorithm: Computation Time

To start with

O( m log m ) for sorting the ce values (MergeSort)

need to do m times: Is T ∪ e acyclic?, and if so, add e to T

We need a clever data structure!

Store & update to which component any node belongs:

Initialize x(v) = v ∀ v ∈ V [n components] O( n )

m times we do for an edge v ,w:Check (T ∪ v ,w acyclic) ⇔ x(v) 6= x(w) O( 1 )

If yes, add v ,w to T [merge 2 components, n − 1 times]:For all i ∈ V : If x(i) == x(w) let x(i) := x(v) O( n )

O( m log m ) + O( n ) + mO( 1 ) + (n− 1)O( n ) ∈ O( m log m + n2 )


Documents

Discrete Optimization 2010 Lecture 1 Introduction ...uetzm/do/DO_Lecture1.pdf · Lecture 1 Introduction / Algorithms & Spanning Trees ... 1 Introduction 2 Analysis of Algorithms