1

09/15/2010

Recitation #2

Lab Programming Assignment – Disjoint Sets Equivalence Relations

o A relation R is defined on a set S if for every pair of elements ( ) is either

true or false. If is true, then we say that a is related to b

o An equivalence relation is a relation R that satisfies three properties:

(Reflexive) , for all

(Symmetric) if and only if

(Transitive) and implies that

o Give an example Friends on Facebook (research that everybody is related within 5 -

7 degrees of people you know to anybody else on the planet)

Given an equivalence relation we want to decide if a and b are related we want to be

divided in equivalence classes every member of appears in exactly one equivalence class

sets are disjoint

Input is usually a collection of N sets, each with one element

Operations:

o Union(x, y) O(1)

Find roots for these two values

Merge these two trees into one

Try to minimize height of resulting tree attach smaller height to bigger height

subtree (union-by-height; there is also union-by-size)

Height increases only by 1 if trees of equal height are joined trivial

implementation

o Find(x) O(N)

Return root

o Path Compression is used to improve speed of Find operation

On the way to the root, every node’s parent is set to the root

Hard to recomputed heights when doing that

Heights are only estimated heights union-by-rank

Worst-Case of union-by-rank with path compression using M ( ) unions and finds is

( ) where is the number of times that logarithm of N needs to be applied until

result is , e.g.

o It’s a pain to prove check the book

ADT is usually optimized for fast “find” or fast “union” solution in book makes “union” easy

and “find” hard

Use arrays to store tree representation of set example

2

o In class lecture slide store same number if root node, in book store -1 instead

o Implementation code is in the book also for path compression

o Sarah asks you to have 2 arrays (one for parent, one for rank), but you could just use the

solution of the book that just uses one array

Fun-application of generating mazes maybe do a little example

0 1

2 3

4 -1 -1 1 3 1

i= 0 1 2 3 4

3

Class Skeleton import java.io.File;

import java.io.IOException;

import java.util.Scanner;

public class DisjointSet{

private int[] parent;

private int[] rank;

// simple DisjointSet constructor

public DisjointSet(int size)

{

// instantiate arrays, and initialize the values of parent

// and rank arrays.

}

// returns the set that the given element is in

// performs path compression to make the algorithm more

// efficient

public int find(int i)

{

}

// unions the sets that elements i and j are in

public void union(int i, int j)

{

}

// unions the trees rooted at i and j

public void mergeTrees(int i, int j)

{

// unions the two sets and also tries to keep the height

// of the trees as small as possible

}

public void print()

{

}

public static void main(String[] args) throws Exception

{

}

}

4

Recitation #3

Lab Programming Assignment – Bucket Sort

Sarah explained in class that the lower bound complexity for comparison-based sorting algorithms is

( ) what about algorithms that are not based on comparisons?

What’s the lowest you can get with any sorting algorithms? ( ), because you have to look at

each element at least once

Explain non-comparison based sorting algorithms by covering (1) Counting Sort, (2) Radix Sort, and

(3) Bucket Sort

Counting Sort:

o Assume that all n numbers are positive integers [ ]

o Here is the algorithm for sorting an array [0]a , … , [ 1]a n :

1. Create an auxiliary C, indexed from 0 to m , and initialize each value to 0.

2. Run through the input array A, incrementing the number of occurrences of each

value 0 through m by adding +1 to the appropriate index in C. (Thus C is a frequency

array)

3. Run through the array C, a 2nd time so that the value stored in each array slot

represents the # of elements ≤ the index value in the original array A.

4. Now, run through the original input array A, and for each value in A, use the

auxiliary array C, to tell you the proper placement of that value in the sorted input,

which is a new array B. Decrement the value in C if it is bigger than -1 .

5. Copy the sorted array B into A.

o Example:

1. Sort the list (3,6,4,4,2,1,3) , where 0,6x create auxiliary array C:

0 1 2 3 4 5 6

0 0 0 0 0 0 0

2. Count occurrences in array C

0 1 2 3 4 5 6

0 1 1 2 2 0 1

3. Calculate indexes in array C

0 1 2 3 4 5 6

-1 0 1 3 5 0 6

4. Create array B and insert first element (3) index 3 from C

Array B:

0 1 2 3 4 5 6

3

Array C:

0 1 2 3 4 5 6

5

-1 0 1 2 5 0 6

4. Create array B and insert second element (6) index 6 from C

Array B:

0 1 2 3 4 5 6

3 6

Array C:

0 1 2 3 4 5 6

-1 0 1 2 5 0 5

…. (And so on with step 4)

Radix Sort:

o Based on lexicographic ordering (also called dictionary order)

E.g. “b, c, d, ba" would be lexicographically sorted as "b, ba, c, d "

E.g. “2, 3, 1, 10” would be sorted as “1, 10, 2, 3”

o We know numbers have a fixed length of digits (e.g. : 100 – 999)

o Algorithm:

1. Take the least significant digit (or group of bits, both being examples of radices) of

each key.

2. Group the keys based on that digit, but otherwise keep the original order of keys.

3. Repeat the grouping process with each more significant digit.

o Example:

170, 045, 075, 090, 002, 024, 802, 066

Start with least-significant digit (unit digit)

0 1 2 3 4 5 6 7 8 9

170 090

002 802

024 045 075

066

170, 090, 002, 802, 024, 045, 075, 066

Continue with next most-significant digit (ten digit)

0 1 2 3 4 5 6 7 8 9

002 802

024 045 066 170 075

090

022, 802, 024, 045, 066, 170, 075, 090

Continue with next most-significant digit (hundred digit)

0 1 2 3 4 5 6 7 8 9

022 024

170 802

6

045 066 075 090

022, 024, 045, 066, 075, 090, 170, 802

o How would we do it for the most-significant digit Radix Sort?

o Running time ( ), where n is the number of keys, and k is the average key length

o Works also great for strings

Bucket Sort:

o Works best if elements are distributed uniformly between min and max We need

additional information about the input numbers

o floating-point numbers between ( ) and ( ): use buckets, each having the

range ( ( ) ( ) )

o Bucket indexing by ⌊ [ ] ( )

( ) ( )⌋ (for numbers between [ ) it would be ⌊ [ ]⌋).

just do an extra check if index is

o Algorithm:

1. Set up an array of initially empty "buckets."

2. Scatter: Go over the original array, putting each object in its bucket.

3. Sort each non-empty bucket. You can use something inefficient like insertion sort here:

( )

4. Gather: Visit the buckets in order and put all elements back into the original array.

o In its most basic form have one bucket for each value from [min, max]. Then go through

input array once and count elements by occurrence. Degenerated to Counting Sort

problem with memory, what to do about floating-point? ( ), where n is length of

input array, k length of counting array

Implementation:

o Verify that sorting is correct by using this isSorted() function in skeleton

o Implement linked list that automatically keeps order either with intelligently inserting

item or use Collections.sort() at the end on unsorted lists

o Implement Node (which is a bucket, that uses this linked list)

Why don’t we just use bucket sort for all sorting?

o Space requirements and overhead high constant can be worse than for reasonable

values of .

7

09/29/2010

Recitation #4 – Exam Review

Administrative Recitation # 2

o Grading Criteria changed correct handling of underscores

o WebCourses button “View Previous Comments”

Due date for bucket sort lab programs 11:59 p.m. tonight

Exam is tonight Panic !

Exam Review Go over proof for lower bound of comparison-based sorting see below

Series formulas

Where:

1 ... first term

... n-th termn

a

a

... number of elements

... common ratio

n

r

Finite geometric series of n elements (e.g. 1

2n

i

i

): 1(1 )

1

na rS

r

Finite arithmetic series of n elements (e.g. 1

n

i

i

): 1( )

2

nn a aS

Infinite geometric series (e.g. 1

1

2

i

i

): 1 , 11

aS r

r

Mixed arithmetic-geometric series try to get rid of geometric component by using the “S-Trick”

(Subtraction Trick)

S

xS

S xS

choose x to be the common ratio of geometric component

Big-Oh notation and definition

Definitions (here ( )f f n and ( )g g n ):

1. Big-Oh: ( )f O g iff 0( 0, 0)c n such that f cg for all 0n n

2. Big-Omega: ( )f g iff 0( 0, 0)c n such that f cg for all 0n n

3. Big-Theta: ( )f g iff ( )f O g and ( )f g

4. Little-Oh: ( )f o g iff ( )f O g and ( ) ( )f g g

8

o Simple Hierarchy: 2 2 3log log log 2Nc N N N N N N N

o How do you prove relationships analytically?

1. Look at the limit that the quotient of the two functions has:

0 ( )

lim 0 ( )

( )N

f o gf

c f gg

f g

o Always remember: ( ) ( )f O g g f

Sorting Algorithms Review

Counting Sort Demonstrated last time.

Radix Sort Demonstrated last time.

Bucket Sort Demonstrated last time.

Shell Sort

Merge Sort Should know by now

Quick Sort Should know by now

Shell Sort

Best case ( )O N , Worst case 2( log )O N N

The basic idea:

Instead of sorting all the elements at once, sort a small set of them

Such as every 5th element (you can use insertion sort)

Then sort every 3rd element, etc.

Finally sort all the elements using insertion sort.

Rationale:

A small insertion sort is quite efficient.

A larger insertion sort can be efficient, if the elements are already “close” to sorted order.

By doing the smaller insertion sorts, the elements do get closer to in order before the larger

insertions are done.

Example (Elements with the same color are in the same sorting set):

Select every 5th element

Sort sets of elements and select every 3rd element

Sort sets of elements and select all elements

Final Insertion Sort Result

9

Works best when increment sequence is geometric series in which every term is approximately 2.2

times smaller than the previous one

AVL Trees

Binary Tree with balance condition

At any node, the difference of height between the left and right subtree is at most 1

Insertion Average and Worst Case (log )O N

- Insert in tree and trace up towards root and see if there is an imbalance at any node

- For cases 2 and 3, starting from , list nodes 1 2 3, ,k k k in-order Always make 2k the new

root

- 4 possibilities of what can happen:

1. Insertion into the left subtree of the left child of

2. Insertion into the right subtree of the left child of

3. Insertion into the left subtree of the right child of

4. Insertion into the right subtree of the right child of

Deletion Average and Worst Case (log )O N

10

1. If the node is a leaf, remove it. Retrace back up the tree starting with the parent of deleted

node to the root, adjusting the balance factor as needed single or double rotation might

be necessary.

2. If it’s not a leaf, replace with the largest in its left subtree (inorder predecessor) and remove

that node. (or replace with the smallest in its right subtree (inorder successor)).

3. After deletion, retrace the path back up the tree, starting with the parent of the

replacement, to the root, adjusting the balance factor as needed.

Search Average and Worst Case (log )O N

2-4 Trees

o Specific type of multitree.

o Every node must have in between 2 and 4 children. (Thus each internal node must store in

between 1 and 3 keys)

o All external nodes (the null children of leaf nodes) and leaf nodes have the same depth.

o Left subtree of a key always contains values that are than key, right subtree always has

values that are

o Insertion: (log )O N

1. Wiggle your way through to leaf node

2. If leaf node becomes full, send 3rd value to parents and split the other 3 values

3. If parent is not able to accept new value, repeat the procedure and make new root

o Deletion: (log )O N

11

See attached printout

o Search: (log )O N

Binary Heaps

Similar to binary search tree, but ALL values in the subtree rooted at a node are (min heap)

Used in priority queue and for Heap Sort

Store in array, children of node i are nodes 2i and (2 1)i , parent of node i is / 2i

Insertion (log )O n

o Insert in next open spot in the heap might be wrong spot

o Percolate up is necessary

o If the parent of the new node is greater than the inserted value, swap them.

(a single percolate up step)

o Continue this process until the inserted node’s parent stores a number lower than it.

o Since the height of the tree is (log )O n , this is an (log )O n operation.

Deletion (deleteMin) (log )O n

o Return value stored in the root

o Place the “last” node in the vacated root; Probably not the correct location.

o Percolate Down

1. Compare the element to its children.

2. If one of the children is less than the node, swap the lowest.

(This is a single percolate down step)

3. CONTINUE until this “last” node has children with larger values than it.

Heapify (MakeHeap) ( )O n after careful anlysis

o Construct heap out of unsorted elements

o Place all the unsorted elements in a complete binary tree.

o Going through the nodes in backwards order, and skipping the leaf nodes, run Percolate

Down on each of these nodes.

o Notice that each subtree below any node that has already run Percolate Down is already a

heap. So after we run Percolate Down on the root, the whole tree is a heap.

12

A Lower Bound for Comparison-Based Sorting Decision tree is used to prove a lower bound every algorithm based on comparisons can be

represented by decision tree

Decision tree is binary tree; each node represents a set of possible orderings, consistent with

comparisons that have been made already; results of the comparisons are tree edges

Example for sorting three elements a, b, c:

Number of comparisons is equal to the depth of the deepest leaf (necessary to resolve all

ambiguities)

A list of N elements can have N! different orderings, hence decision tree has N! leaves (in

example 3! = 6 orderings)

Lemmas:

1. A binary tree of depth has at most leaves (proof by induction, full tree).

2. A binary tree with leaves must have depth at least ⌈ ⌉ (follows from lemma 1,

because maximum , thus ⌈ ⌉)

3. Any sorting algorithm that uses comparisons requires at least ⌈ ⌉ comparisons

(because decision tree has N! leaves and lemma 2).

Final Lemma:

o Any sorting algorithm that uses comparisons requires at least ( ) comparisons

o Proof:

log( !) log( ( 1)( 2) (2)(1))

log( ) log( 1) log( / 2) log(2) log(1)

log( ) log( 1) log( / 2)

log log log 2 log 12 2 2 2

log2 2

( log )

N N N N

N N N

N N N

N N N NN N

N NN

N N

13

Sorting Algorithms Runtime

Name Average Worst Stable Notes Bubble Sort 2n

2n Y Easy to implement

Selection Sort 2n 2n N

Insertion Sort 2n 2n Y Average case ( )O n d , where d is # of inversions

Merge Sort logn n logn n Y

Heap Sort logn n logn n N

Quicksort logn n 2n Y

Shell Sort - 2logn n N

Counting Sort n k n k Y k is length of counting array (max – min + 1)

Radix Sort nk nk Y k is key length (in digits), usually implies ( )O n

Bucket Sort n 2n Y Case assume that insertion sort is used for buckets,

Performance degrades with clustering

14

10/13/2010

Recitation #5

Assignment #3 Due on 10/20/2010, 11:59 p.m. (only have one week’s time)

You basically have an undirected, weighted graph; survivors are nodes and edges are given as

weighted paths between survivors in the input file

It is guaranteed that the graph is connected, i.e. all nodes/survivors are reachable

Implement Kruskal’s Algorithm for calculating a Minimum Spanning Tree

o MST of weighted, undirected graph is tree formed by connecting all of the vertices with

minimal overall cost

o minimum cost Minimum; covers every vertex Spanning; Acyclic Tree

o There could be multiple MSTs (all with the same minimum weight)

o From Lecture Slides:

Kruskal’s Algorithm

Let V =

For i=1 to n-1, (where there are n vertices in a graph)

V = V e, where e is the edge with the minimum edge

weight not already in V, and that does NOT

form a cycle when added to V.

Return V

• Basically, you build the MST of the graph by continually adding in the smallest weighted edge into the MST that doesn't form a cycle.

– When you are done, you'll have an MST.

– You HAVE to make sure you never add an edge the forms a cycle and that you always add the minimum of ALL the edges left that don't.

Kruskal’s

• The reason this works: – is that each added edge is

connecting between two sets of vertices,

– and since we select the edges in order by weight,

– we are always selecting the minimum edge weight that connects the two sets of vertices.

• Cycle detection:– Keep track of disjoint

sets.

– Initially, each vertex is in its own disjoint set.

– When you add an edge you are unioning two sets.

– A union cannot happen if the two vertices are already in the same set.• This would create a

cycle.

15

Use Disjoint Sets for cycle

detection (from recitation #3) or

you have to write it again

Use Binary Heap to always

extract edge with minimum

weight (insert all edges into heap

(possibly percolate up) then

deleteMin() and percolate down)

Pseudocode is in the book!

Output format: Be careful,

because Sarah wants you to sort

the output, e.g. SURVIVOR1

SURVIVOR2 where S1 comes

alphabetically before S2.

Test separate data structures

first, before you try to make them work together as a whole

Graph Data Structures Adjacency Matrix

o Just a two-dimensional array

o By convention vertex is connected to itself [ ][ ] 0A u u

o Undirected graph: For each edge ( , )u v , set [ ][ ] [ ][ ] 1A u v A v u , else

[ ][ ] [ ][ ]A u v A v u

o Directed graph: For each edge ( , )u v , set [ ][ ] 1A u v , else [ ][ ]A u v

o If edge has a weight, put that in adjacency matrix; use or for non-existent

edges

o Space requirement of 2(| | )V

o Is appropriate if graph is dense: 2| | (| | )E V waste of a lot of space if graph is only

sparse

o In digraph, it’s really easy to find all incoming edges

Adjacency List

o Linked list of connected vertices

o Better solution if graph is sparse

o Weights can be saved as extra attribute

o Space requirement of (| | | |)O E V

Kruskal’sv1

v7v6

v4v3 v5

v2

1

4

2

4

2

8

31 10

7

65

v1

v4

1

v1

v4

1

v7

v6 1

v7

v6 1

v1

v2

2

v4

v3

2

v4

v2

3

v7

v4

4

v6

v3

5

v7

v5

6

v1

v3

4

v4

v5

7

v6

v4

8

v5

v2

10

v2

2

v3

2

3

CYCLE!

4

4

v5

6

All Vertices, we’re

done!!

Determine the MST:

Edges in sorted order:

Given Graph G:

16

1

4 5

3 2

1 4

5

3

2

2

3 2

1

5

-3

Adjacency List 1

2

3

4

5

2 55

11 33 55

22 44

33 55

11 22 44

1

2

3

4

5

3w = 2

3w = 2

2w = 3

4w = 1

2w = 5

5w = -3

Adjacency Matrix 1 2 3 4 5

1 0 1 1

2 1 0 1 1

3 1 0 1 4 1 0 1

5 1 1 1 0

1 2 3 4 5

1 0 2 2 0 2 3 3 0 1 4 5 0 -3

5 0

Graph Coloring We deal with unweighted, undirected graphs

To color a graph, assign a color to each vertex such that no two vertices

connected by an edge are the same color in complete graph every

vertex has different color

Chromatic number = minimum number of colors necessary to color a

graph

All bipartite graphs can be colored in 2 colors ↔ All graphs that can be

colored in 2 colors are bipartite

Problem can be solved in linear time for 2 colors

Problem becomes NP-hard for colors

Practical Applications

o Coloring of map with minimum amount of colors; No two neighboring countries can

have the same color; Countries are vertices, shared borders are edges

14

5

3

2

17

o Aquarium Mix; For a number of fish species decide how many tanks are needed to

house all of them, so they don’t eat each other; Fish Species are vertices; Predator-Prey

relationships are edges in graph; Chromatic Number equals minimum amount of tanks

that you need so fish don’t eat each other

Graph Search Methods Depth-First Search

o Search down a path from a particular source vertex as far as possible

o When you can go no farther, backtrack to last vertex from which a different path could

have been taken

o Continue that until all vertices covered

o Mark edges and vertices that have already been visited

Breadth-First Search

o Opposite of Depth-First Search

o Search all paths at a uniform depth from the source before moving onto deeper paths.

o Mark edges and vertices that have already been visited

Lab Programming Assignment – 2-Coloring of a Graph Due date: 10/20/2010, 11:59 p.m.

Use one of the data structures (adjacency matrix, adjacency list) to save the graph

Run Breadth-First Search

o Implementation suggestion in lecture notes and in the book

Depth First Search

• Start with 1• 2,4,3

– Stop at 3

• Backtrack to 4, can we search?• 7,6

– Stop at 2

• Backtrack to 6, can we search?• 10,13

– Stop at 13

• Backtrack to 10,6,7• 9, 11, 8, 5

– Stop at 5

• Backtrack to 8,11• 15,14,12

– STOP – All nodes are marked!!

1 2 5

106

11 15

3 4 7 129

13

14

8

Output

List:

1 2 4 3 7 6 10 13 9 11 8 5 15 14 12

1 2

3

5

6

11

4

10

15

7 9

8

12

13

14

Breadth First Search

• L0: 1• L1: 2, 3• L2: 4,5,6,11• L3: 7,8,10,9,15• L4: 13,12,14

1 2 5

106

11 15

3 4 7 129

13

14

1 2

3

5

6

11

4

8

10

15

7 9

8

12

13

14

Final output: 1, 2, 3, 4, 5, 6, 11, 7, 8, 10, 9, 15, 13, 12, 14

18

10/20/2010

Recitation #6

Most common sources of errors for students

1. Make up your own test cases. Many students just rely on the test cases on the assignment sheet or the additional test cases you give them, rather than making up their own and testing the code extensively.

2. Modularity. Students should learn to develop and test the code in stages/modules. One thing I noticed a lot in office hours is that students think of their code as this monolithic black box that should take your input file and print the desired output. If that doesn't work, they tinker in all kinds of different places without actually knowing what components of the code are actually working and which are the culprit for the faulty output. Assignment #3 was a perfect example for that. Students should first develop a working heap and test it before they go on. Then implement the Disjoint Sets and run through the same tests. Only once this is all done, put everything together and connect the dots. If there are any errors in this last stage you can exclude the Disjoint Sets and the binary heap from your search, because you already verified these modules.

3. Debugging. This relates to (2), but I think any programmer should know how to use a graphical debugger. No matter what IDE you use to develop your code, it will include a graphical debugger. You can set breakpoints, watch variable values change, and step through the code line-by-line if necessary. Many students still rely on printouts in their code, which makes finding bugs a lot harder. I will try to encourage students in my labs tomorrow to familiarize themselves with the Eclipse debugger for their next assignment

Using Eclipse Debugger Please NEVER use printouts for debugging your code

A Graphical debugger is the way to go every IDE has a good debugger built-in

Great way to see variable values and trace through code

Click on the little “bug” icon instead of the run icon to start debugging mode it will ask you to

change to the debugging perspective

Right-click on the vertical bar at the left of the source file and select “Toggle Breakpoint”

o A breakpoint will stop code execution at that line

o You can set conditions on when this breakpoint should trigger

o Once the program stops you can see the value of all variables

o You can step through the code line-by-line and make sure that everything works the way

you want it

Just search for “Eclipse Debugger” and you should find a wealth of websites and tutorials

19

Greedy Bottom-up approach

Greedy takes the step that seems the best at the time of execution; very short-sighted and

looking for immediate gain, i.e. “greedy”

Greedy algorithms try to make locally optimal decisions hoping that this will contribute to a

globally optimal solution

E.g. Dijkstra’s, Kruskal’s, Prim’s

Divide-And-Conquer Top-Down approach

Traditionally, algorithms with at least 2 recursive calls are divide-and-conquer, while algorithms

with one (e.g. find() in Disjoint Sets) are not

Works as follows:

o If the problem is small enough, solve it (base case). Else:

o Divide: Divide big problem into smaller disjoint subproblems.

o Conquer: Solve subproblems recursively.

o Combine: The solution to the original problem is then formed from the solutions of the

subproblems

Because of recursion, timing analysis often involves recurrences, e.g. ( ) ( / 2) ( )T n T n O n

User Master Theorem for those

E.g. Quicksort, Mergesort, Integer Multiplication, Strassen’s

Recurrence Relations If an algorithm contains a recursive call to itself its running time can often be described by a

recurrence relation, e.g.

(1) if 1( )

2 ( / 2) ( ) if 1

nT n

T n n n

this is MergeSort’s runtime

Different ways to solve that:

1. Telescoping Method:

o Write the recurrence equation repeatedly “following” the recurrence

20

o Adding up all equations usually leads to cancellation of a lot of terms

o E.g. ( ) ( 1)T N T N cN

Write out equation repeatedly:

( 1) ( 2) ( 1)

( 2) ( 3) ( 2)

(2) (1) (2)

T N T N c N

T N T N c N

T T c

Adding up all the equations yields:

( ) ( 1) (2) ( 1) (1) 2T N T N T T N T cN c

Subtracting all duplicate terms gives the solution:

2

2

( ) (1)

( )

N

i

T N T c i

O N

o E.g. ( ) 2 ( / 2)T N T N N

Same steps as above, but witch little twist that I divide by N first:

( ) ( / 2)1

/ 2

T N T N

N N

Then do the telescoping:

( / 2) ( / 4)1

/ 2 / 4

(2) (1)1

2 1

T N T N

N N

T T

Adding the equations and subtracting duplicate terms:

( ) (1)log

1

( ) log ( log )

T N TN

N

T N N N N O N N

2. Substitution Method:

o Substitute the recurrence repeatedly on the right side

o Look for a pattern and solve

o E.g. ( ) ( 1)T N T N cN

Substitute repeatedly:

21

2

2

( ) ( 2) ( 1)

( 3) ( 2) ( 1)

(1) (2) ( 1)

(1) ( )N

i

T N T N c N

T N c N c N

T c c N

T c i O N

3. Master Theorem:

o Closed-form solution to most common recurrence relationships

( ) ( / ) ( )T N aT N b f N , where 1, 1a b

o Have to distinguish between three separate cases:

1. If log( ) b a

f N O N

, for some constant 0

Then log( ) b a

T N N

2. If log( ) logb a kf N N N , for some constant 0k

Then log 1( ) logb a kT N N N

3. If log( ) b a

f N N

, for some constant 0

And ( / ) ( )af N b cf N , for some constant 1c and sufficiently large N

Then ( ) ( ( ))T N f N

Dynamic Programming Recursion can sometimes be inefficient, because you might solve the same subproblems over

and over again

Dynamic Programming tries to avoid making redundant method calls

Example in the book: Fibonacci numbers:

o ( ) ( 1) ( 2)

with [ (1) 1; (0) 0]

Fib n Fib n Fib n

Fib Fib

o Recursive solution has lots of calls to (0)Fib and (1)Fib that are unnecessary, because

why do I have to calculate that again and again?

o Solution: Save all intermediate results of function calls and re-use them

22

Why did recursion work so well for divide-and-conquer?

o Because the subproblems were substantially smaller than the original problem , e.g. N/2

full recursion tree only has logarithmic depth

o In typical DP solution, subproblems are only slightly smaller from original problem (see

Fibonacci numbers).

o In this case it is easier to enumerate all distinct subproblems and solve them in the right

order

23

Subset Sum Divide-And-Conquer solution This method to solve a variant of the subset sum problem was discussed by Sarah in class

Lab Programming Assignment Solve the subset sum problem with Dynamic Programming

From Sarah’s E-Mail: “If you could review this and also describe how it makes redundant

recursive calls, so a dynamic programming solution would be faster that would be great. Then if

they need a hint on how to implement the Dynamic Programming version, suggest a boolean

array of size target+1 and describe what those boolean values correspond to. They will probably

also need an example to show this. I suggest a target of 8 and values array [4, 3, 4, 2] since

that's what we used in class. “

From Wikipedia:

24

10/27/2010

Recitation – Exam Review for Exam 2

Exam Review Make sure you understand how most of the presented algorithms work in principle

Mostly driven by students’ questions

If they are not forthcoming, definitely cover the following topics: Master Theorem, Topological Sort

Graph ↔ Adjacency Matrix

It is essential for most problems on graphs that you understand how to read an adjacency matrix and

convert between different representations.

Note: Notation is sometimes different for simple unweighted graphs

1

4 5

3 2

1 2 3 4 5

1 0 1 1

2 1 0 1 1

3 1 0 1 4 1 0 1

5 1 1 1 0

Both solutions are valid and equivalent.

Master Theorem

Given the following recurrence scheme:

( ) ( / ) ( )T n aT n b f n , where ( ) ( ), 0df n n d

Have to distinguish between 3 different cases:

1. da b ( ) ( )dT n n

2. da b ( ) ( log )dT n n n

3. da b log( ) ( )b a

T n n

1 2 3 4 5

1 0 1 0 0 1

2 1 0 1 0 1

3 0 1 0 1 0

4 0 0 1 0 1

5 1 1 0 1 0

25

Be careful, because you cannot use it for non-divisive recurrences e.g. ( ) ( 1) ( )T n T n O n Use

telescoping or substitution instead. The only thing you will have to do in the exam is being able to apply

the Master Theorem.

Topological Sort

It’s basically a Depth-First Search, but more interesting, because it occurs on a directed graph.

Sometimes your DFS will complete, but not all vertices have been visited. In that case continue in

numerical order to look for the next node with which to restart the DFS.

When you write down the solution, start in the back of the sequence, so you can easily check the validity

of the constraints at the end of your sort.

Alternatively, write down all depth-first searches.

E.g. start at vertex 3

Valid DFS:

3, 8, 9, 10

2

5, 11

7

Now write these down in reverse order to get the topological sort:

7, 5, 11, 2, 3, 8, 9, 10

Dijkstra’s Algorithm

Finds all shortest paths from one start vertex to all other vertices in the graph.

Start out with all vertices connected to start vertex, e.g. A.

Then choose vertex, e.g. B, that has smallest cost and look at its incident edges. Add the cost to

the already known cost for B and see if it’s smaller than what I already have in my table

Example:

Use Dijkstra's Algorithm to determine the shortest distance from vertex A to all other vertices in the

graph with the weighted adjacency matrix given below:

A B C D E F G

A 0 12 ∞ 3 ∞ 6 ∞ B 2 0 7 ∞ 10 ∞ 4 C ∞ ∞ 0 ∞ ∞ ∞ 1

26

D 7 8 13 0 ∞ 2 ∞ E ∞ 2 3 ∞ 0 ∞ 6 F ∞ 5 12 ∞ 4 0 12 G ∞ ∞ 1 2 3 4 0

Add to S B C D E F G A 12 ∞ 3 ∞ 6 ∞

D 11 16 3 ∞ 5 5 ∞

F 10 16 3 9 9 5 17

E 10 12 3 9 5 15

B 10 12 3 9 5 14

C 10 12 3 9 5 13 13

G 10 12 3 9 5 13 Red shows the next node that’s chosen. Green shows all the nodes that change their shortest path.

1. Initially, just take over whatever edges are incident to A.

2. Take next vertex to which we have shortest path already (and that we didn’t visit) D

3. D has already cost of 3. There are 4 incident edges to D: DA 7, DB 8, DC 13, DF 2

4. Ignore edge DA (because we know the path from A to A has always weight 0.

5. Edge DB adds cost 8 to D’s cost of 3 8 + 3 = 11 < 12 replace 12 with 11

6. Look at all other edges incident to D, following pattern in (5)

7. Go to (2) and repeat until done

Path Reconstruction from A to G:

Only look at column of G at first

Last time it changed when C was added, thus remember C, G

Look at column C, changed value when E was added E, C, G

Column of E changed value when F was added F, E, C, G

Column of F changed when D was added D, F, E, C, G

Then the shortest path is A, D, F, E, C, G with weight 13

Prim’s Algorithm

Algorithm is very similar to Kruskal’s algorithm see notes for Recitation #5. Instead of globally sorting

all edges though and building the MST by connecting disjoint sets, we build the MST up incrementally

(from some starting vertex) and sort all edges that are incident to this growing set.

Example:

A B C D E F G

A 0 12 2 3 ∞ 6 1 B 12 0 7 ∞ 10 ∞ 4

G ∞ ∞ 17

15

14

13 13

13

27

C 2 7 0 ∞ 4 7 1 D 3 ∞ ∞ 0 ∞ 2 ∞ E ∞ 10 4 ∞ 0 ∞ 6 F 6 ∞ 7 2 ∞ 0 12 G 1 4 1 ∞ 6 12 0

Write down all currently considered edges into a list and keep track of that whenever you add a

vertex during Prim’s search.

Always look at upper triangular matrix for getting edges it’s symmetrical

Always ignore edges that are internal to MST, that way you avoid cycles

Start at vertex B edges BA 12, BC 7, BE 10, BG 4

Choose edge with smallest cost: BG 4. Add G to MST edges BA 12, BC 7, BE 10, GA 1, GB 4,

GC 1, GE 6, GF 12

Choose edge with smallest cost: GA 1. Add A to MST edges AB 12, AC 2, AD 3, AF 6, BA 12, BC

7, BE 10, GB 4, GC 1, GE 6, GF 12

Choose edge with smallest cost: GC 1. Add C to MST edges AB 12, AC 2, AD 3, AF 6, BA 12, BC

7, BE 10, CA 2, CB 7, CE 4, CF 7, GB 4, GE 6, GF 12

Choose edge with smallest cost: AC 2. Creates cycle, so ignore

Choose edge with smallest cost: AD 3. Add D to MST

Choose edge with smallest cost: DF 2. Add F to MST

Choose edge with smallest cost: CE 4. Add E to MST

Minimum Spanning Tree has cost 15.

Huffman Encoding

See Arup’s notes for a great explanation

28

11/03/2010

Recitation #7

Dynamic Programming Review:

o Recursion can sometimes be inefficient, because you might solve the same subproblems

over and over again

o Dynamic Programming tries to avoid making redundant method calls

A few more notes on the subject:

o Optimal substructure:

optimal solution can be constructed efficiently from optimal solutions to its

subproblems

typically use Greedy algorithms for that, but if the problem also exhibits

overlapping subproblems, we use DP

Nice examples (from Wikipedia):

Shortest path between two cities is likely to exhibit optimal

substructure. That is, if the shortest route from Seattle to Los Angeles

passes through Portland and then Sacramento, then the shortest route

from Portland to Los Angeles must pass through Sacramento too.

Problem of finding the cheapest airline ticket from Buenos Aires to

Moscow. Even if that ticket involves stops in Miami and then London, we

can't conclude that the cheapest ticket from Miami to Moscow stops in

London, because the price at which an airline sells a multi-flight trip is

usually not the sum of the prices at which it would sell the individual

flights in the trip.

o Overlapping subproblems:

Problem can be broken down into subproblems which are reused several times

Think of the Fibonacci sequence and how terms were used over and over again

29

30

Longest Common Subsequence DP Solution Solved by Sarah in class

See attached sheet

Example of recursive solution:

LCS(“BEER”, “BEAR”)

LCS(“BEE”, “BEA”)1 +

LCS(“BE”, “BEA”) LCS(“BEE”, “BE”)MAX( )

LCS(“BE”, “B”)1 +LCS(“B”, “BEA”) LCS(“BE”, “BE”)MAX( )

LCS(“B”, “B”)1 +

1

2

1

1

2

2

2

3

Example of DP solution:

= LCS(“BE”, “BE”) = 1 + LCS(“B”, ”B”) [1 + value from up and left cell in table]

= LCS(“BEER”, “BEA”) = MAX( LCS(“BEE”, “BEA”), LCS(“BEER, “BE”) )

[max of values from top and left cells in table]

B E A R

B 1 1 1 1

E 1 2 2 2

E 1 2 2 2

R 1 2 2 3

o Reconstructing the LCS sequence in string form

Start on bottom right cell construct string backwards

If characters match, take it and go diagonally to the top left

If characters don’t match, choose the maximum of the cell to the left or the cell

to the top and continue there

LCS(“BEER”, “BEAR”) = “BER”

2

2

B E A R

B 1 1 1 1

E 1 2 2 2

E 1 2 2 2

R 1 2 2 3

31

Lab Programming Assignment – Longest Palindrome DP Solve the longest palindrome problem with Dynamic Programming

Find the longest palindrome, i.e. word that reads the same from the front and the back (e.g.

RACECAR, ROTATOR, HANNAH in English; LAGERTONNENNOTREGAL in German), that is a

subsequence of a given string.

Just for fun the “Dammit I’m mad” poem by comedian Demetri Martin (222 word palindrome)

Hint: Solution finds the longest common subsequence of string and its reverse

Code for LCS is in Sarah’s slides, just have to implement reconstruction of subsequence string

Assignment #4 Problem (B)

o Use Dijkstra’s on each vertex OR use Floyd-Warshall OR even simpler use Transitive

Closure (because you only try to determine reachability) See notes for recitation #8

Problem (A)

o Need to implement Edit Distance

Edit Distance

o The edit distance computes the cost of modifying a given source string (with operations insertion,

deletion, and replacement) to yield a target string.

o Source string will always be in the first row, target string in first column

o The way Sarah showed it in class, each operation (insertion, deletion, replacement) had exactly the

same weight If you fill out the table like that, you will know the number of edit operations, but

because we have a heterogeneous cost for replacement operations, there is no easy way to do a

unique path reconstruction

o Better Approach: Incorporate weight in building your matrix, i.e. insertions and replacements are 2,

costs are based on the cost matrix C given by Sarah

A T G C

A 0 2 3 4

T 2 0 4 5

G 3 4 0 1

C 4 5 1 0

o In the matrix, each move to the right implies a deletion has to happen, e.g. to get from “AC” to the

empty string “”, we would first delete “C” and then delete “A”’; deletions are cost 2

A C T C G T

0 2 4

o Each move to the bottom implies an insertion has to happen, e.g. if we know how to get from “A”

to “”, then we can get from “” to “C” by inserting “C”; insertions are also cost 2

32

A

0 2

C 2 4

o Each move diagonally signifies a replacement, e.g. to get from “A” to “C” you can simply replace “A”

with “C”; in our example the cost for replacements vary

A

0 2

C 2 4

o Now, each table cell can be reached either through insertion (from the top), through deletion (from

the left), or through replacement (diagonally from the upper left). The new value of your cell will

then be the minimum of the values in these respective cells plus the cost of the associated

operations. If X Y , then just take the value diagonally from the upper left, as no edit operation

needs to be performed.

,

min( 2, 2, cost( , )) ,

b if X Yn

a c b X Y otherwise

o Then the filled-out table for her first example looks like this:

A C T C G T

0 2 4 6 8 10 12

C 2 4 2 4 6 8 10

A 4 2 4 4 6 8 10

G 6 4 3 5 5 6 8

T 8 6 5 3 5 7 6

G 10 8 7 5 4 5 7

o Reconstruction of edit operations:

Start in the bottom-right cell

If letters are the same, no operation necessary, go diagonally to upper left

If letters are different see which operation yielded the minimum cost and then go to this cell

Example on the right (we start with string “ACTCGT”):

1. Delete “T”, look at “G” now: ACTCG

2. Keep “G”, look at “C”: ACTCG

3. Delete “C”, look at “T”: ACTG

4. Keep “T”, look at “C”: ACTG

5. Replace “C” with “G”, look at “A”: AGTG

6. Keep “A”, look at “”: AGTG

7. Insert “C”. Done: CAGTG

A C T C G T

0 2 4 6 8 10 12

C 2 4 2 4 6 8 10

A 4 2 4 4 6 8 10

G 6 4 3 5 5 6 8

T 8 6 5 3 5 7 6

G 10 8 7 5 4 5 7

b a

c nX

Y

repl ins

del

33

11/10/2010

Recitation #8

Dynamic Programming Go through review of DP from Recitation #7

Floyd-Warshall Algorithm to find the shortest paths between ALL vertices in a weighted, directed graph

Important: Even though, with standard definitions of graphs, there is never an edge from a

vertex to itself, there is a path, of length 0, from a vertex to itself

Essentially the same algorithm got published in 1962 by Robert Floyd and Stephen Warshall

Warshall’s Algorithm (Transitive Closure)

Works on unweighted, directed graph, D(i,j) = 1 if edge exists, =0 otherwise

Only decides if other vertices are reachable from a source vertex

Main idea: A path exists between two vertices i and j iff

o There is an edge from i to j; or

o There is a path from i to j going through vertex 1; or

o There is a path from i to j going through vertex 1 and/or 2; or

o ……

In the k-th stage, determine if a path exists between i and j using just vertices among 1, …, k ( 1)

( )

( 1) ( 1)

( , ) (path using just 1, ... , k-1)

( , )

( , ) ( , ) (path from i to k and from

k to i using just 1, ..., k-1)

k

k

k k

D i j

D i j

D i k D k j

When building the matrices, you can always ignore the k-th row and the k-th column

o Why? Absorption Rule of Boolean Logic. In k-th row ( ) ( ) ( 1) ( 1) ( 1) ( 1)( , ) ( , ) ( , ) ( ( , ) ( , )) ( , )k k k k k kD i j D k j D k j D k k D k j D k j

o In k-th column ( ) ( ) ( 1) ( 1) ( 1) ( 1)( , ) ( , ) ( , ) ( ( , ) ( , )) ( , )k k k k k kD i j D i k D i k D i k D k k D i k

Example:

Transitive Closure(W[1..n][1..n])

01 D W // D(0)

02 for k 1 to n do // compute D(k)

03 for i 1 to n do

04 for j 1 to n do

05 D[i][j] D[i][j] (D[i][k] D[k][j])

06 return D

i

j

k

kth stage

34

For your airline problem in assignment #4, you probably want to initialize the diagonal to 1, as

you assume that the city can reach itself

Floyd’s Algorithm

Essentially the same as Warshall’s, but now I have weights on the graph

Replace AND with + and OR with “min”

When building the matrices, you can always ignore the k-th row and the k-th column

o Why? Because ( , )D k k is always 0 and always will be

o In k-th row ( ) ( ) ( 1) ( 1) ( 1) ( 1)( , ) ( , ) min( ( , ), ( , ) ( , )) ( , )k k k k k kD i j D k j D k j D k k D k j D k j

o In k-th column ( ) ( ) ( 1) ( 1) ( 1) ( 1)( , ) ( , ) min( ( , ), ( , ) ( , )) ( , )k k k k k kD i j D i k D i k D i k D k k D i k

Example:

(0)

0 3 8

0 1

4 0

2 5 0

D

5

0 3 8

0 1

4 0

2 5 0

0 3 8

0 1

4 0

2 5

4

0

5

5

3

42

1

3

42

13

42

1

3

42

1

R0

0 0 1 0

1 0 0 1

0 0 0 0

0 1 0 0

R1

0 0 1 0

1 0 1 1

0 0 0 0

0 1 0 0

R2

0 0 1 0

1 0 1 1

0 0 0 0

1 1 1 1

R3

0 0 1 0

1 0 1 1

0 0 0 0

1 1 1 1

R4

0 0 1 0

1 1 1 1

0 0 0 0

1 1 1 1

3

42

1

Floyd’s Algorithm(W[1..n][1..n])

01 D W // D(0)

02 for k 1 to n do // compute D(k)

03 for i 1 to n do

04 for j 1 to n do

05 D[i][j] MIND[i][j], D[i][k] + D[k][j])

06 return D

1 4

3

23 1

-58

4

2

35

0 3 8 4

0 1

4 0 5

2 5 01

0 3 4

0 1

4 0 5

2 1 5 0

1

3 4

7

Floyd-Warshall – Path Reconstruction

Build path matrix P that always stores the last vertex visited on shortest path from i to j

Initially and ( , )

( , )otherwise

i i j D i jP i j

NIL

If ( , ) ( , )D i k D k j improves the estimate (the minimum changes), record the last vertex on

the path through k: ( ) ( 1)( , ) ( , )k kP i j P k j

For example above:

(0)

1 1

2

3

4 4

NIL NIL

NIL NIL NILP

NIL NIL NIL

NIL NIL

(1)

1 1

2

3

14 4

NIL NIL

NIL NIL NILP

NIL NIL NIL

NIL

(2)

1 21

2

3

4 1 4

2

NIL

NIL NIL NILP

NIL NIL

NIL

(3)

1 1 2

2

3 2

4 43

NIL

NIL NIL NILP

NIL NIL

NIL

(4)

4

4

1 2

2

3 2

4 3 4

4

4

NIL

NILP

NIL

NIL

Then you can reconstruct path from 1 to 3 backwards:

o First look at (1,3) 4P . This means that on the path from 1 to 3, the last vertex visited

on the shortest path before 3 is 4; path is now 1 … 4, 3

o Now look at (1,4) 2P . Path is now 1 … 2, 4, 3

o Now look at (1,2) 1P . We are done, because we found start vertex

o Final Path is 1, 2, 4, 3

Warshall’s Algorithm takes a lot less memory than Floyd’s

Only need to store bit matrix vs. 32-bit integers in Floyd’s

Caution: C++ and Java still use 1 byte for internal representation of “bool” 1 byte is the

smallest addressable unit

Use std::bitset or java.util.BitSet for vectors or matrices of bits that are stored efficiently

Recitation Problem – World Series You want to know what the probability is that Team A will win against Team B a best-of-n game

series

Assumptions

36

o Each game is independent of the rest, i.e. ( ) ( ) ( | ) ( ) ( )P A B P A P B A P A P B

o No tied games

o Probability of Team A winning is p , of Team B winning is 1q p

o n is an odd number

The series is over if / 2k n games are won by any team

Build table with ( 1) ( 1)k k cells, e.g. for 3n

0 1 2

0 X 0 0

1 1 0.6 0.6*0.6

2 1 0.6 + 0.4*0.6 0.4*0.6*0.6 + 0.6*0.6 + 0.6*0.4*0.6 = 0.648

( , )P i j denotes the probability that Team A wins series if A still needs i more victories and

Team B still needs j more victories

(0,0)P doesn’t make sense fill with “X”

(0, ) 1, 0P j j k because if Team A needs no more victories it will win

( ,0) 0, 0P i i k because if Team B needs no more victories, Team A has no chance to win

( , ) ( 1, ) (1 ) ( , 1)P i j pP i j p P i j

o If Team A wins, it needs one less victory, so new probability is ( 1, )P i j multiplied by

the probability of winning p

o If Team A loses, Team B needs one less victory, so new probability is ( , 1)P i j

multiplied by probability of losing (1 )p

( , )P k k is then the sought-for probability at the beginning of the series (Team A and B both still

need 2 victories).

The instruction sheet gives you a different solution hint, but the way that Arup describes it, you

would have to calculate the table from the bottom-right to the top-left

Extra Notes

Proof of Absorption law

( ) ( 1) ( ) (1 ) 1A A B A A B A B A A

37

Knapsack 0-1 Problem

N items to consider

No fractional parts of items either you take it or you don’t; iv is item’s value, iw is its weight

Maximize value of Knapsack that can hold a maximum weight of W

Initially call [ , ]B N W

[ 1, ] , if (because total weight would be )[ , ]

max( [ 1, ], [ 1, ] ) (item could be part of solution, or not)

k

k k

B k w w w wB k w

B k w B k w w v k

In English, this means that the best subset of kS that has total weight w is:

o The best subset of 1kS that has total weight w (we don’t take item), or

o The best subset of 1kS that has total weight kw w plus the item k (we take item

and add its value kv

Dynamic Programming Solution:

o Build table with 1N rows and 1W columns

o Initialize first row and first column with 0

o Fill out the table:

item above , if [ , ]

max(item above, item above and to the left ) ,otherwise

k

k k

w wB k w

w v

Reconstruction:

o Finds actual elements that are used to obtain maximum value

if B[i, k] ≠ B[I - 1, k] then

mark the ith item as in the knapsack

i = i-1, k = k - wi

else

i = i-1

Example:

o 1 {4 ,$4}I kg , 2 {2 ,$4}I kg , 3 {1 ,$3}I kg , 4 {6 ,$9}I kg , 5 {3 ,$5}I kg

o Maximum weight 10W kg

item/weight 0 1 2 3 4 5 6 7 8 9 10

0 0 0 0 0 0 0 0 0 0 0 0

1 0 0 0 0 6 6 6 6 6 6 6

2 0 0 4 4 6 6 10 10 10 10 10

3 0 3 4 7 7 9 9 13 13 13 13

4 0 3 3 7 7 9 9 13 13 16 16

5 0 3 3 7 8 9 12 13 14 16 18

o Reconstruction tells us that items 1 2 3 5{ , , , }I I I I with value of $18 are chosen

38

11/17/2010

Recitation #9

Backtracking Clever implementation of exhaustive search

Incrementally builds candidates to the solutions, and abandons each partial candidate c

(“backtracks”) as soon as it determines that c cannot possibly be completed to a valid solution

Analogy of depth-first building a decision tree, each time you backtrack, you are basically

saying that a whole branch of a tree can be disregarded, because all decisions that happen in

there cannot possibly lead to a correct solution

Can be only applied for problems that have the concept of “partial candidate solution” and a

relatively quick test of whether it can possibly be completed to a valid solution

Example: The Queens Problem

Consider an n n chess board, and the problem of placing n queens on

the board without the queens threatening one another.

The solution space is {1,2,3,..., }nn , because all queens can be positioned

in all combinations of columns

Trying all vectors 1( ,..., )np p implies nn cases. Noticing that all the

queens must reside in different columns reduces this number to !n

Root of the traversal tree has degree n, the children have degree (n-1), and so forth

Checking for threatening positions along the way will further reduce the number of visited

configurations

39

Assignment #5 – Tentaizu Backtracking Japanese game that is basically Minesweeper, but you don’t get any redundant queues about

where mines are

The game is played on a 7x7 board. Exactly 10 of the 49 squares are each hiding a star.

Your task is to find the squares containing the 10 hidden stars.

You are guaranteed that each given Tentaizu board will have a unique solution.

Board B is two-dimensional array using some convention, e.g.:

1 for empty spot

( , ) [1,8] for star hint

9 for cells containing star

B x y

Build secondary array C that keeps track of which surrounding cells put a possible dependency

on a given cell

Table 1: Left: Board B (-1 cells are omitted for clarity); Right: Auxiliary Array C

1 3

1 0

2

3 3

1

1 1

Explanation of steps for function FIND-STAR(x,y)

1. For all cells in the table starting at position (x,y)

2. If all stars are placed and all cells in B have a value less or equal to 0, return true.

3. If we reach beyond the last cell of the table, we couldn’t place all stars, return

false.

4. Don’t try to put a star in any of the hint locations. Continue.

5. If this cell has no dependencies, we don’t know anything, so continue.

6. Make sure that we don’t violate any constraints of number of stars in B if we

would place a star here. If we do, continue.

7. Assume we can place a star. Decrement all hint cells that the current cell

depends on.

8. Run FIND-STAR again on the current position. If it returns true, return true.

9. Well, that didn’t work out. Set current cell back to BLANK and increment all

dependencies in B.

1 (0,0) (5,0) 3 (5,0)

(0,0) (1,2)

(0,0) (1,2)

(1,2) (4,2) (5,0) (4,2)

(5,0) (4,2)

(5,0)

(1,2) 1 (1,2) (3,3)

(4,2) (3,3) 0 (4,2)

(1,2) (1,4)

(1,2) (1,4)

(1,2) (3,3) (1,4)

2 (4,2) (3,3)

(4,2) (6,4)

(6,4)

(1,4) 3 (3,3) (1,4)

(3,3) (4,5)

(3,3) (4,5)

(6,4) (4,5) 3

(1,4) (1,4) (1,4) (3,6)

(4,5) (3,6) 1

(6,4) (4,5) (6,6)

(6,4) (6,6)

(3,6) 1 (4,5) (3,6)

(4,5) (6,6) 1

40

Now cycle through board with function FIND-STAR(x,y): 1. FOR ALL cells (a,b) starting at (x,y)

2. IF all stars placed AND all cells in B <= 0, RETURN true.

3. IF (a,b) outside of table, RETURN false.

4. IF (B(a,b) >= 0) CONTINUE.

5. IF (C(a,b) == ) CONTINUE.

6. IF any (B(C(a,b)) == 0) CONTINUE.

7. SET B(a,b) = 9. Decrement all cells B(C(a,b))

8. IF (FIND-STAR(a,b)) RETURN true.

9. SET B(a,b) = -1. Increment all cells B(C(a,b))

Initially, call FIND-STAR(0,0)

This might seem inefficient, but the algorithm really only considers all the star positions around

the hint cells (which you would have to examine anyway

There are other ways to solve this problem

You could start with the cell that has the biggest number n and your initial decision would try all

different combinations of how to arrange the n numbers into the eight surrounding cells, e.g.

8

5

combinations of 5 stars into eight cells

On the next level of the decision tree, choose a different hint and try all star arrangements

around it; if any arrangement conflicts with the other stars we already have, back track and try

the other possibilities

41

Lab Problem – Robot Path Finding A robot is placed in an n n maze at starting position (1,0) in the maze and is asked to try to

reach the right side (if there is a path to ( 1, )n y where y is any number between 0 and 1n

At any given moment, the robot can only move 1 step

in one of 4 directions:

o Go North: ( , ) ( , 1)x y x y

o Go East : ( , ) ( 1, )x y x y

o Go South : ( , ) ( , 1)x y x y

o Go West : ( , ) ( 1, )x y x y

Solve the path using recursion with FIND-PATH

function:

FIND-PATH(x, y)

1. if (x,y outside maze) return false 2. if (x,y already visited) return

false

3. if (x,y not open) return false 4. if (x,y is goal) return true 5. if (FIND-PATH(North of x,y) ==

true) return true

6. if (FIND-PATH(East of x,y) == true) return true

7. if (FIND-PATH(South of x,y) == true) return true

8. if (FIND-PATH(West of x,y) == true) return true

9. return false

Initially, call FIND-PATH(1, 0)

Note, that if a path to the goal was found, it is

important that the algorithm stops, i.e., if going North

of x, y finds a path (i.e., returns true), then from the

current position (i.e., current call of FIND-PATH) there

is no need to check East, South or West. Instead, FIND-

PATH just need return true to the previous call.

Backtracking happens automatically through nature of

recursion, because it keeps track of the tried paths

It will find the path, if one exists (although not

necessarily the shortest one) It’s almost like a depth-

first search of the maze from your starting position

42

12/01/2010

Exam Review - Final

Network Flow Given a connected, directed graph ( ) with non-negative edge weights (capacities

)

Source “s” has only outgoing edges, Sink “t” has only incoming edges

Find maximum amount that can flow from “s” to “t”

Residual capacity of a path: the minimum of the residual capacities of the edges on that path

(equals max excess flow we can push down that path)

Augmenting path: defined as one where you have a path from the source to the sink where

every edge has a non-zero residual capacity

Flow must fulfill two conditions:

1. Conservation: What goes in, must come out. ( , ) ( , )

wu uz

w u E u z E

f f

2. Capacities: No edge capacities are violated. 0 e ef c for all e E

Side Note: These are just a bunch of linear constraints. Can be solved very efficiently with Linear

Programming (but we will see a different algorithm here).

Ford-Fulkerson Algorithm

While there exists an augmenting path

Add the appropriate flow to that augmenting path

That’s a pretty simple algorithm!

Need to take into account all possible paths from source to sink and check them for residual

capacity

Construct 2 extra graphs: flow graph fG and residual graph rG (initially rG G )

Example:

o Choose augmenting path s,b,d,t add 2 units to flow

43

o Choose augmenting path s,a,c,t add 2 units of flow

o Choose augmenting path s,a,d,t add 1 unit of flow

o Algorithm terminates, because t is unreachable from s.

o Resulting maximum flow is 5

Problem:

o What happens if we choose a different first path?

o Choose augmenting path s,a,d,t add 3 units to flow

44

o Algorithm would terminate with suboptimal solution 3

Solution:

o Allow algorithm to “change its mind”

o For every edge ( , )v w with flow ,v wf in the flow graph, add edge ( , )w v of capacity

,v wf to residual graph rG

o Back to problem example add 3 units of flow along s,a,d,t

o Add 2 units of flow along s,b,d,a,c,t

o Terminates with maximum flow of 5 correct solution

How do we choose the next path?

45

o Always choose augmenting path that

allows the largest increase in flow

o Another way is to choose path with the

least number of edges Edmonds-

Karp Algorithm

o Classic example for possible bad

augmentation:

Cut of a graph

o ( , )C S T of graph ( , )G V E Is a partition of vertices of a graph into two disjoint

sets such that s S and t T ; {( , ) | , }u v E u S v T are forward edges;

{( , ) }v u E are backward edges

o The flow across any cut equals the flow of a network.(flow is sum of the flows of forward

edges minus sum of flows of backward edges) from flow graph fG

o The cut-set is the set {( , ) | , }u v E u S v T (forward edges between disjoint sets)

o Value of the cut is the sum of the weights in cut-set

o If edges of cut are removed, nothing can flow from source to sink

Max-Flow Min-Cut Theorem

o If f is a flow in a flow network ( , )G V E , with source s and sink t , then the

following conditions are equivalent:

1. f is a maximum flow in G

2. The residual network rG contains no augmented paths.

3. Value of the cut is | | ( , )f f S T for all cuts ( , )C S T in fG .

o Has nice applications in Computer Vision for problems that involve energy minimization,

e.g. foreground-background segmentation, stereo correspondence and image

smoothing

For above flow graph there are multiple minimum cuts possible (all have value of 5):

Hand out Recitation #10 problems

o Solutions will be posted a t a later time on the course website

46

o Discuss possible solution for problem 1

o Problem 2 is the optional homework problem

o Problem 3 is verbatim from slides

47

More Final Exam Review You can have 3 sheets of letter-sized paper filled on both sides, so don’t waste your time

remembering definitions or lengthy proofs

The following sections list the topics that weren’t covered in Exam 1 & 2 and that Sarah

mentioned in her topics list

Dynamic Programming Change Problem see following pages

Floyd-Warshall's Algorithm and path reconstruction Recitation #8; additional example on

review sheet

Transitive Closure Recitation #8; additional example there

Longest Common Subsequence Recitation #7; additional example on review sheet

Edit Distance Recitation #7; additional example on review sheet

Knapsack 0-1 Problem Recitation #8; additional example on review sheet

Matrix Chain Multiplication see following pages; additional example on review sheet

World Series Problem Recitation #8 problem; additional example below

Backtracking Eight Queens Recitation #9; additional example on review sheet

Magic Square additional example on review sheet

Sudoku See following pages

Tentaizu Assignment #5, see explanation in recitation #9

Network Flow Maximum Flow Today’s material; additional example above

Minimum Cut Today’s material; additional example above

Use of BFS

Bipartite Matching Sarah hasn’t covered that yet

Matching Instructors to classes Today’s material

Filling an orchestra, etc. Today’s material

48

Additional Explanations and Examples

Change Problem

Problem Description:

o Given a certain amount of money, how many different ways are there to make change

for that amount of money?

DP solution:

o We note that the number of ways to make change for n cents using denomination d can

be split up into counting two groups:

1. The number of ways to make change for n cents using denominations LESS than

d (value in the table directly above)

2. The number of ways to make change for n cents using at least ONE coin of

denomination d (value in the table that is in the same row, by d spots to the

left)

o Essentially, each row label stands for the number of cents we are making change for

o Each column label stands for the largest coin value allowed to make change

Example:

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

2 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8

5 1 1 2 2 3 4 5 6 7 8 10 11 13 14 16 18

10 1 1 2 2 3 4 5 6 7 8 11 12 15 16 19 22

Matrix Chain Multiplication Problem Description:

o Given some matrices to multiply, determine the best order to multiply them so you

minimize the number of single element multiplications, i.e. determine the most

efficient way to parenthesize the matrices

o First off, it should be noted that matrix multiplication is associative, but not

commutative. But since it is associative, we always have: ((AB)(CD)) = (A(B(CD))), or any

other grouping as long as the matrices are in the same consecutive order.

o Goal is to determine the fewest number of multiplications necessary to compute the

product.

o Let ,i jN denote the minimal number of multiplications necessary to find the product:

1...i i jA A A If 0 1 1... nA A A A , we must determine 0, 1nN

o and let 1i id d denote the dimensions of matrix iA

DP solution:

o Our final multiplication will ALWAYS be of the form 0 1 1 1( ... )( ... )k k nA A A A A

49

o In essence, there is exactly one value of k for which we should "split" our work into two

separate cases so that we get an optimal result

o Count the number of multiplications in each of these choices and pick the minimum

o Recursive formula:

, , 1, 1 1valid values of (min( ))i j i k k j i k jN k N N d d d Look at all possible ways

to split multiplication

o Build table Initialize diagonal to 0 and all other entries to

o Order of filling N matrix:

A B C D

A 1 5 8 10

B 2 6 9

C 3 7

D 4

Example:

A is 5x4, B is 4x6, C is 6x2, D is 2x7 5, 4, 6, 2, 7A B C D Ed d d d d

Determine multiplications for 2 matrices:

AB uses 5x4x6 = 120 element (A,B); BC uses 4x6x2 = 48 element (B,C) ; CD uses 6x2x7 = 84

element (C,D)

A B C D

A 0 120

B 0 48

C 0 84

D 0

Determine multiplications for 3 matrices:

(AB)C uses , , , 120 0 5 6 2 180A C A B C C A C DN N N d d d

A(BC) uses , , ,min(180, ) min(180,0 48 5 4 2) 88A C A A B C A B DN N N d d d

(BC)D uses , , , 48 0 4 2 7 104B D B C D D B D EN N N d d d

B(CD) uses , , ,min(104, ) min(104,0 84 4 6 7) 104B D B B C D B C EN N N d d d

A B C D

A 0 120 88

B 0 48 104

C 0 84

D 0

Determine multiplications for 4 matrices:

50

A(BCD) uses , , , 0 104 5 4 7 244A D A A B D A B EN N N d d d

(AB)(CD) uses

, , ,min(244, )

min(244,120 84 5 6 7) min(244,414) 244

A D A B C D A C EN N N d d d

(ABC)D uses

, , ,min(244, )

min(244,88 0 5 2 7) min(244,158) 158

A D A C D D A D EN N N d d d

A B C D

A 0 120 88 158

B 0 48 104

C 0 84

D 0

World Series Problem Example:

o Best-of-5 series; 0.3p 5 / 2 3k

0 1 2 3

0 X 0 0 0

1 1 0.3 0.3*0.3 = 0.09 0.09*0.3 = 0.027

2 1 0.3 + 0.7*0.3 = 0.51

0.3*0.51 + 0.7*0.09 = 0.216

0.3*0.216 + 0.7*0.027 = 0.0837

3 1 0.3 + 0.7*0.51 = 0.657

0.3*0.657 + 0.7*0.216 = 0.3483

0.3*0.3483 + 0.7*0.0837 = 0.16308

Sudoku Problem Description:

o Each row contains numbers 1-9 with no repeats

o Each column contains numbers 1-9 with no repeats

o The boxes must contain numbers 1-9 with no repeats

Backtracking solution:

o Keep track of current board int[][] and available

options for each square boolean[][][]

o At each step, need to check if rows, columns, and

boxes are ok return false if not

o Reduce choice by filling in empty squares based on

previous choices

8

4

2

6

3 4

9 1

9 6

8 4

2 1 6

3 5 7

8 4

7 5

2 6

1 3

9

7

1

4