Greedy Algorithms Technique Greedy Algorithms Technique Dr. M. Sakalli, Dr. M. Sakalli, modified from Levitin and CLSRmodified from Levitin and CLSR

Greedy TechniqueGreedy Technique The The first key ingredientfirst key ingredient is the is the greedy-choice propertygreedy-choice property: a : a

globally optimal solution can be arrived at by making a globally optimal solution can be arrived at by making a locally optimal (greedy) choice. locally optimal (greedy) choice.

In a greedy algorithm, choice is determined on fly at each In a greedy algorithm, choice is determined on fly at each step, (while algorithm progresses), may seem to be best at step, (while algorithm progresses), may seem to be best at the moment and then solves the subproblems after the the moment and then solves the subproblems after the choice is made. The choice made by a greedy algorithm choice is made. The choice made by a greedy algorithm may be depend on choices so far, but it cannot depend on may be depend on choices so far, but it cannot depend on any future choices or on the solutions to subproblems. any future choices or on the solutions to subproblems. Thus, unlike dynamic programming, which solves the Thus, unlike dynamic programming, which solves the subproblems bottom up, subproblems bottom up, a greedy strategy usually a greedy strategy usually progresses in a progresses in a top-down top-down fashionfashion, making one greedy , making one greedy choice after another, iteratively reducing each given choice after another, iteratively reducing each given problem instance to a smaller one. problem instance to a smaller one.

Algorithms for optimization problems typically go through a Algorithms for optimization problems typically go through a sequence of steps, using dynamic programming to determine sequence of steps, using dynamic programming to determine thethe bestbest choices, choices, (bottom-up) for subproblem solutions. (bottom-up) for subproblem solutions.

But in many cases much simpler, more efficient algorithms are But in many cases much simpler, more efficient algorithms are possible. A possible. A greedy algorithmgreedy algorithm always always makes the choice makes the choice for a for a locally optimal solution which seems the locally optimal solution which seems the best at the current best at the current momentmoment with the hope of a globally optimal solution. In the with the hope of a globally optimal solution. In the most cases does not always yield optimal solutions, but for most cases does not always yield optimal solutions, but for many problems. many problems.

The activity-selection problem, for which a greedy algorithm The activity-selection problem, for which a greedy algorithm efficiently computes a solution, works well for a wide range of efficiently computes a solution, works well for a wide range of problems, i.e, minimum-spanning-tree algorithms, Dijkstra's problems, i.e, minimum-spanning-tree algorithms, Dijkstra's algorithm for shortest paths form a single source, and algorithm for shortest paths form a single source, and Chvátal's greedy set-covering heuristic. Chvátal's greedy set-covering heuristic.

Greedy algorithms constructs a solution to an Greedy algorithms constructs a solution to an optimization optimization problemproblem piece by piecepiece by piece through a sequence of choices that are: through a sequence of choices that are:

feasiblefeasible locally optimallocally optimal Irrevocable Irrevocable (binding and abiding)(binding and abiding)

Applications of the Greedy StrategyApplications of the Greedy Strategy

Optimal solutions:Optimal solutions:

• change making for “normal” coin denominationschange making for “normal” coin denominations

• minimum spanning tree (MST)minimum spanning tree (MST)

• single-source shortest paths single-source shortest paths

• simple scheduling problemssimple scheduling problems

• Huffman codesHuffman codes

Approximations:Approximations:

• traveling salesman problem (TSP)traveling salesman problem (TSP)

• knapsack problemknapsack problem

• other combinatorial optimization problemsother combinatorial optimization problems

Change-Making ProblemChange-Making Problem

Given unlimited amounts of coins of denominations Given unlimited amounts of coins of denominations dd1 1 > … > > … > ddm m , ,

give change for amount give change for amount n n with the least number of coinswith the least number of coins

Example: Example: dd1 1 = 25c, = 25c, dd2 2 =10c, =10c, dd3 3 = 5c, = 5c, dd4 4 = 1c and = 1c and n = n = 48c48c

Greedy solution: d1+ 2 d2+ 3 d1Greedy solution: d1+ 2 d2+ 3 d1

Greedy solution isGreedy solution is optimal for any amount and “normal’’ set of denominationsoptimal for any amount and “normal’’ set of denominations may not be optimal for arbitrary coin denominationsmay not be optimal for arbitrary coin denominations

Let a graph beLet a graph be G G = { = {V,V, EE}, we say that }, we say that HH is a is a subgraphsubgraph of of GG and we write and we write HHGG if the vertices and edges of if the vertices and edges of HH = { = {V’,V’, E’E’} is } is a subset of the graph a subset of the graph GG, , VV''VV, , EE''E.E.

HH need not to accommodate all the edges of need not to accommodate all the edges of G.G. IfIf the the verticesvertices of H are of H are connectedconnected and if its all the connecting and if its all the connecting

edges edges overlayoverlay withwith the the edgesedges in in GG connecting the same connecting the same adjacent vertices then, adjacent vertices then, HH is called is called induced subgraphinduced subgraph. . EdgeEdge induced, induced, vertexvertex induced, or induced, or neitherneither edge nor vertex induced. edge nor vertex induced.

Note: if at least one path does exist between every pair of Note: if at least one path does exist between every pair of vertices, then this is a connected (but not directed yet) graph, vertices, then this is a connected (but not directed yet) graph, directed one is the one whose every edge can only be followed directed one is the one whose every edge can only be followed from one vertex to another. from one vertex to another.

Induced subgraphInduced subgraph

Directed acyclic grph, DAGDirected acyclic grph, DAG

CycleCycle definition seems to be ambiguous, two or three vertices, (only one definition seems to be ambiguous, two or three vertices, (only one vertex), if there is a path returning back to the same initial vertex again vertex), if there is a path returning back to the same initial vertex again then there should be a cyclic case too. But!! then there should be a cyclic case too. But!!

A A cyclic graphcyclic graph is a graph that has at least one cycle (through another is a graph that has at least one cycle (through another vertex -at least one for bi-directional and at least two for uni-directional vertex -at least one for bi-directional and at least two for uni-directional edged graphs); an edged graphs); an acyclic graphacyclic graph is the one that contains no cycles. The is the one that contains no cycles. The girthgirth is the number of the edges involved in the shortest cycle, g>3 is triangle is the number of the edges involved in the shortest cycle, g>3 is triangle free graph. free graph.

A A sourcesource is a vertex is a vertex with having no incident edgeswith having no incident edges, while a , while a sinksink is a vertex is a vertex with with no diverging edgesno diverging edges from itself. from itself. A directed acyclic graphA directed acyclic graph ( (DAGDAG) is ) is defined from a source to a vertex or from a vertex to a sink, or from a defined from a source to a vertex or from a vertex to a sink, or from a source to a sink, with no directed cycles. source to a sink, with no directed cycles.

A finite DAG must have at least one source and at least one sink.A finite DAG must have at least one source and at least one sink. The The depthdepth of a vertexof a vertex in a finite DAG is the length of the longest path from in a finite DAG is the length of the longest path from

a source to that vertex, while a source to that vertex, while its its heightheight is the length of the longest path is the length of the longest path from that vertex to a sink. from that vertex to a sink.

The The lengthlength of a finite DAG of a finite DAG is the length (number of edges) of a longest is the length (number of edges) of a longest directed path. It is equal to the maximum height of all sources and equal to directed path. It is equal to the maximum height of all sources and equal to the maximum depth of all sinks. Wikipedia. the maximum depth of all sinks. Wikipedia.

Minimum Weight Spanning TreeMinimum Weight Spanning Tree

c

db

a6

2

4

3

1

OO((E E lg lg VV) using binary heaps. ) using binary heaps. OO((EE + + VV lg lg VV), using fib heap. ), using fib heap.

A A spanning subgraphspanning subgraph is a subgraph that is a subgraph that contains all the vertices of the original contains all the vertices of the original graph. A graph. A spanning treespanning tree is a is a spanningspanning subgraphsubgraph that is of a that is of a acyclicacyclic graph of graph of GG: (a : (a connectedconnected acyclicacyclic subgraphsubgraph) that ) that includes all of vertices of includes all of vertices of GG..

Minimum spanning treeMinimum spanning tree of a of a weightedweighted, , connected graph connected graph GG: a spanning tree of : a spanning tree of GG of of minimum total weightminimum total weight. Graph . Graph GG = = {{VV, , EE} and the weight function } and the weight function ww of a of a spanning tree spanning tree TT, that connects all , that connects all VV. .

w(T)=w(T)=((EE, , VV))ЄЄTT w(w(uu, , vv))

All weights are distinct, “injective”All weights are distinct, “injective”

Partitioning Partitioning VV of of GG into into AA and and VV--AA

GENERIC-MST(GENERIC-MST(G, wG, w))

1.1. AA Empty set Empty set //Initialize//Initialize

2.2. while while AA does not form a spanning tree does not form a spanning tree //Terminate//Terminate

3.3. do if edge (do if edge (u, vu, v) is safe edge for ) is safe edge for AA

4.4. AAAA {( {(u, vu, v)} )} //Add safe edges//Add safe edges

5.5. return return AA

How to decide to the light edgeHow to decide to the light edge

Definitions: a Definitions: a cutcut ( (S, V - SS, V - S) of an undirected graph ) of an undirected graph GG = = ((V, EV, E) is a partition of ) is a partition of VV, and a , and a light edgelight edge is the one, one is the one, one of its endpoints (vertices) resides in of its endpoints (vertices) resides in S,S, and the other in and the other in V - V - SS, with minimum possible weight, and a cut , with minimum possible weight, and a cut respectsrespects to A to A ifthere is no any edge of ifthere is no any edge of AA crossing the cut. crossing the cut.

…….… .… do if edge (do if edge (u, vu, v) is safe for ) is safe for AA In line 3 of the pseudo codeIn line 3 of the pseudo code given for generic mst(g, w), given for generic mst(g, w),

there must be a spanning tree there must be a spanning tree TT such that such that AA TT, and if , and if there is an edge of (there is an edge of (uu, , vv))TT such that ( such that (uu, , vv))AA, then (, then (uu, , vv) ) is the one said safe for is the one said safe for A.A. The rule: TheoremThe rule: Theorem

T’

Tu

v

x yHow to decide to the light edgeHow to decide to the light edge

Theorem: Suppose Theorem: Suppose GG = ( = (V, EV, E) is a connected, undirected graph with a real-) is a connected, undirected graph with a real-valued weight function valued weight function ww defined on defined on E,E, and and let let AA be a subset of be a subset of EE that is that is included included in some MSTin some MST in in GG, , ((S, V - SS, V - S) be a cut that respects ) be a cut that respects AA, and , and ((uu, , vv) be ) be a light edge crossing (a light edge crossing (S, V - SS, V - S).). Then, edge ( Then, edge (uu, , vv) is safe to be united to ) is safe to be united to AA..

Proof by contradiction: Proof by contradiction: Let Let TT be a mst of be a mst of GG, that , that AATT and suppose another and suppose another mst mst T’T’, with , with AAT’T’ and a light edge of { and a light edge of {uu, , vv}}T’T’ such that ( such that (uu, , vv))TT. . T'T' (of (of GG) having ) having AA{({(uu, , vv)} by using a cut-and-paste technique is possible, then )} by using a cut-and-paste technique is possible, then ((uu, , vv) must be a safe edge for ) must be a safe edge for AA. But we have another path of T, {x, y} . But we have another path of T, {x, y} crossing the cut {S, V-S}, (in which one side includes crossing the cut {S, V-S}, (in which one side includes AA), This generates a ), This generates a cycle (for which we are paying for two crossings and contradicts to the cycle (for which we are paying for two crossings and contradicts to the definition of mst). To minimize the cost we have to exclude the one with definition of mst). To minimize the cost we have to exclude the one with the heavier edge, and include the light one. the heavier edge, and include the light one. T’T’ = = T T - {- {xx, , vv} } {({(uu, , vv)} )} ww((u, vu, v)) ≤≤ ww((x, yx, y). Therefore, w). Therefore, w(T') = w(T) - w((T') = w(T) - w(x,x, yy) + w() + w(uu, v) , v) ≤≤w(T)w(T)

V-SS

S

Overlapping subproblems! DP.but MST leads to an efficient algorithm.

Both MST algorithms Kruskal, and Prim, determine a safe edge in line 3 Both MST algorithms Kruskal, and Prim, determine a safe edge in line 3 of generic-mst. of generic-mst.

In Kruskal's algorithm, In Kruskal's algorithm, the set the set AA is a forest is a forest. . - The safe edge added to - The safe edge added to AA is always a least-weight edge is always a least-weight edge connecting two connecting two distinct componentsdistinct components- many induced subtrees, and gradually merge into each other. - many induced subtrees, and gradually merge into each other.

In Prim's algorithm, the set In Prim's algorithm, the set AA forms forms a single tree growsa single tree grows like a snowball as like a snowball as one mass.one mass. The safe edge added to The safe edge added to AA is always a least-weighted edge is always a least-weighted edge connecting the tree to a vertex not in the tree.connecting the tree to a vertex not in the tree.

Corollary:Corollary:Let Let

GG = ( = (V, EV, E);); be a connected, undirected graph, be a connected, undirected graph, weight function weight function ww on on EE, , let let AA be a subset of be a subset of EE that is included in some MST for that is included in some MST for GG, and , and CC be a be a connected component (tree) in the forestconnected component (tree) in the forest GGAA = ( = (V, AV, A). If (). If (u, vu, v) is a light ) is a light edge connecting edge connecting CC to some other component in to some other component in GGAA, then edge (, then edge (u, vu, v) is ) is safe for safe for AA..

Kruskal AlgorithmKruskal Algorithm

It finds a safe edge to add to the growing forest: It finds a safe edge to add to the growing forest: • A safe edge (A safe edge (uu, , vv): ):

– connecting any two trees in the forest and connecting any two trees in the forest and

– having the least weight. having the least weight.

– Let Let CC1 and 1 and CC2 denote the two trees that are connected by (2 denote the two trees that are connected by (uu, , vv). Since (). Since (uu,,vv) must ) must be a light edge connecting be a light edge connecting CC1 to some other tree, from corollary it must be a safe 1 to some other tree, from corollary it must be a safe edge for edge for CC1.1.

– Then the next step is union of two disjoint trees C1 and C2. Then the next step is union of two disjoint trees C1 and C2.

Kruskal's algorithm is a Kruskal's algorithm is a greedygreedy algorithm, algorithm, because at each step it adds to because at each step it adds to the forest an edge of the least possible weightthe forest an edge of the least possible weight..

Implementation of Kruskal's algorithm employs Implementation of Kruskal's algorithm employs a disjoint-set data a disjoint-set data structurestructure to maintain several disjoint sets of elementsto maintain several disjoint sets of elements. Each set . Each set contains contains the vertices in a tree of the current forestthe vertices in a tree of the current forest. The operation . The operation FIND-SET(FIND-SET(uu) ) returns a representativereturns a representative element from the set that contains element from the set that contains uu. .

Thus, FIND-SET(Thus, FIND-SET(uu) == FIND-SET() == FIND-SET(vv), then vertices ), then vertices uu and and vv belong to the belong to the same tree otherwise combine two trees, if {u, v} is a light edge - UNION same tree otherwise combine two trees, if {u, v} is a light edge - UNION procedure.procedure.

MST-KRUSKAL(MST-KRUSKAL(GG, , ww)) 1 1 AA Empty set Empty set 2 for each 2 for each vv VV[[GG] ] // for each // for each vertexvertex 3 do MAKE-SET (3 do MAKE-SET (vv) ) //create |//create |VV| trees, | trees, 4 sort the edges of 4 sort the edges of EE by nondecreasing weight by nondecreasing weight ww 5 for each 5 for each ee((uu, , vv) ) EE, in order by nondecreasing weight, in order by nondecreasing weight 6 do if FIND-SET(6 do if FIND-SET(uu) ) FIND-SET( FIND-SET(vv) ) //Check if//Check if 7 then 7 then AA AA {( {(uu, , vv)} )} //if not in the same set//if not in the same set 8 UNION (8 UNION (uu, , vv) ) //merge two components//merge two components 9 return 9 return AA set set AA to the empty set and to the empty set and

The The running timerunning time for a for a GG = ( = (VV, , EE) depends on the ) depends on the implementationimplementation of the of the disjoint-set data structuredisjoint-set data structure. .

Assume the Assume the disjoint-set-forestdisjoint-set-forest implementation implementation with the union-by-rank and with the union-by-rank and path-compression heuristicspath-compression heuristics, since it is the , since it is the asymptotically fastestasymptotically fastest implementation known. implementation known.

Initialization: Initialization: OO((VV), time to sort the edges in line 4: ), time to sort the edges in line 4: OO((EE lg lg EE). ). There are There are OO((EE) operations on the disjoint-set forest, which in total take ) operations on the disjoint-set forest, which in total take

OO((EE αα((EE, , VV)) time, where )) time, where αα is the functional inverse of Ackermann's is the functional inverse of Ackermann's function defined. Since (function defined. Since (EE, , VV) = ) = OO(lg (lg EE), the total running time of ), the total running time of Kruskal's algorithm is Kruskal's algorithm is OO((EE lg lg EE).).

Prim’s MST algorithmPrim’s MST algorithm Operates much Operates much like Dijkstra's algorithmlike Dijkstra's algorithm for finding shortest paths for finding shortest paths

in a graphin a graph. Prim's algorithm has the property that the edges in the . Prim's algorithm has the property that the edges in the set set AA always form a single tree. always form a single tree.

Starting from an arbitrary root vertex Starting from an arbitrary root vertex rr (tree (tree AA) and expanding ) and expanding one vertex at a time, until the tree spans all the vertices in one vertex at a time, until the tree spans all the vertices in VV. At . At each turn, a light edge connecting a vertex in each turn, a light edge connecting a vertex in AA to a vertex in to a vertex in VV - - AA is added to the tree. is added to the tree.

On each iteration, On each iteration, construct Tconstruct Tii+1+1 from T from Ti i by adding vertex not in by adding vertex not in TTi i (A) that is that is closest to those already in closest to those already in TTii ( (this is a “greedy” stepthis is a “greedy” step!)!)

The same Corollary: The rule allows merging of the edges that are The same Corollary: The rule allows merging of the edges that are safe for safe for AA; therefore, Terminates when the all vertices are included ; therefore, Terminates when the all vertices are included in A, there the edges in in A, there the edges in AA form a minimum spanning tree. form a minimum spanning tree.

This strategy is "This strategy is "greedygreedy" since the tree is augmented at each step " since the tree is augmented at each step with an edge that contributes the minimum amount possible to the with an edge that contributes the minimum amount possible to the tree's weight. tree's weight.

Needs priority queue for locating closest fringe vertexNeeds priority queue for locating closest fringe vertex Next analysis rom the notes of CLRS, and listen from MITNext analysis rom the notes of CLRS, and listen from MIT

Prim Algorithm ExamplePrim Algorithm Example

Notes about Kruskal’s algorithmNotes about Kruskal’s algorithm

Algorithm looks easier than Prim’s but is harder to Algorithm looks easier than Prim’s but is harder to implement (checking for cycles!)implement (checking for cycles!)

Cycle checking: a cycle is created iff added edge connects Cycle checking: a cycle is created iff added edge connects vertices in the same connected componentvertices in the same connected component

Union-find Union-find algorithms algorithms

Shortest paths – Dijkstra’s algorithmShortest paths – Dijkstra’s algorithm

Single Source Shortest Paths ProblemSingle Source Shortest Paths Problem: Given a weighted : Given a weighted

connected graph G, find shortest paths from source vertex connected graph G, find shortest paths from source vertex ss

to each of the other verticesto each of the other vertices

Dijkstra’s algorithmDijkstra’s algorithm: Similar to Prim’s MST algorithm, with : Similar to Prim’s MST algorithm, with

a different way of computing numerical labels: Among verticesa different way of computing numerical labels: Among vertices

not already in the tree, it finds vertex not already in the tree, it finds vertex uu with the smallest with the smallest sumsum

ddvv + + ww((vv,,uu))

where where

vv is a vertex for which shortest path has been already found is a vertex for which shortest path has been already found on preceding iterations (such vertices form a tree) on preceding iterations (such vertices form a tree)

ddvv is the length of the shortest path form source to is the length of the shortest path form source to vv

w w((vv,,uu) is the length (weight) of edge from ) is the length (weight) of edge from vv to to uu

ExampleExample

d4

Tree vertices Remaining verticesTree vertices Remaining vertices

a(-,0) b(a,3) c(-,∞) d(a,7) e(-,∞)

a

b4

e

37

62 5

c

a

b

d

4c

e

3

7 4

62 5

a

b

d

4c

e

3

7 4

62 5

a

b

d

4c

e

3

7 4

62 5

b(a,3) c(b,3+4) d(b,3+2) e(-,∞)

d(b,5) c(b,7) e(d,5+4)

c(b,7) e(d,9)

e(d,9)

d

a

b

d

4c

e

3

7 4

62 5

Notes on Dijkstra’s algorithmNotes on Dijkstra’s algorithm

Doesn’t work for graphs with negative weightsDoesn’t work for graphs with negative weights

Applicable to both undirected and directed graphsApplicable to both undirected and directed graphs

EfficiencyEfficiency• O(|V|O(|V|22) for graphs represented by weight matrix and ) for graphs represented by weight matrix and

array implementation of priority queuearray implementation of priority queue• O(O(|E+V|log|V|) |E+V|log|V|) for graphs represented by adj. lists and for graphs represented by adj. lists and

min-heap implementation of priority queuemin-heap implementation of priority queue• Fib heap O(E+VlogV, amertized.)Fib heap O(E+VlogV, amertized.)

Don’t mix up Dijkstra’s algorithm with Prim’s algorithm!Don’t mix up Dijkstra’s algorithm with Prim’s algorithm!

Coding ProblemCoding Problem

CodingCoding: assignment of bit strings to alphabet characters: assignment of bit strings to alphabet characters

CodewordsCodewords: bit strings assigned for characters of alphabet: bit strings assigned for characters of alphabet

Two types of codes:Two types of codes: fixed-length encodingfixed-length encoding (e.g., ASCII) (e.g., ASCII) variable-length encodingvariable-length encoding (e,g., Morse code) (e,g., Morse code)

Prefix-free codesPrefix-free codes: no codeword is a prefix of another codeword: no codeword is a prefix of another codeword

Problem: If frequencies of the character occurrences areProblem: If frequencies of the character occurrences are

known, what is the best binary prefix-free code?known, what is the best binary prefix-free code?

Huffman codesHuffman codes

Any binary tree with edges labeled with 0’s and 1’s yields a prefix-free Any binary tree with edges labeled with 0’s and 1’s yields a prefix-free code of characters assigned to its leavescode of characters assigned to its leaves

Optimal binary tree minimizing the expected (weighted average) length of Optimal binary tree minimizing the expected (weighted average) length of a codeword can be constructed as followsa codeword can be constructed as follows

Huffman’s algorithmHuffman’s algorithm

Initialize Initialize nn one-node trees with alphabet characters and the tree weights one-node trees with alphabet characters and the tree weights

with their frequencies.with their frequencies.

Repeat the following step Repeat the following step nn-1 times: join two binary trees with smallest -1 times: join two binary trees with smallest

weights into one (as left and right subtrees) and make its weight equal the weights into one (as left and right subtrees) and make its weight equal the

sum of the weights of the two trees.sum of the weights of the two trees.

Mark edges leading to left and right subtrees with 0’s and 1’s, respectively.Mark edges leading to left and right subtrees with 0’s and 1’s, respectively.

ExampleExample

character Acharacter A B C D _B C D _

frequency 0.35 0.1 0.2 0.2 0.15frequency 0.35 0.1 0.2 0.2 0.15

codeword 11 100 00 01 101codeword 11 100 00 01 101

average bits per character: 2.25average bits per character: 2.25

for fixed-length encoding: 3for fixed-length encoding: 3

compression ratiocompression ratio: (3-2.25)/3*100% = 25%: (3-2.25)/3*100% = 25%

0.25

0.1

B

0.15

_

0.2

C

0.2

D

0.35

A

0.2

C

0.2

D

0.35

A

0.1

B

0.15

_

0.4

0.2

C

0.2

D

0.6

0.25

0.1

B

0.15

_

0.6

1.0

0 1

0.4

0.2

C

0.2

D0.25

0.1

B

0.15

_

0 1 0

0

1

1

0.25

0.1

B

0.15

_

0.35

A

0.4

0.2

C

0.2

D

0.35

A

0.35

A