Algorithm Design and Analysis (ADA)

242-535 ADA: 7. Dynamic Prog. 1

• Objectiveo introduce DP, its two hallmarks, and two major

programming techniqueso look at two examples: the fibonacci series and

LCS

Algorithm Design and Analysis (ADA)

242-535, Semester 1 2014-2015

7. Dynamic Programming

Overview7. Is the c[] Algorithm

Optimal?8. Repeated

Subproblems?9. LCS() as a DP

Problem10.Bottom-up Examples11.Finding the LCS12.Space and Bottom-up

1. Dynamic Programming (DP)

2. Fibonacci Series3. Features of DP Code4. Longest Common

Subsequence (LCS)5. Towards a Better LCS

Algorithm6. Recursive Definition

of c[]


• The "programming" in DP predates computing, and means "using tables to store information".

• DP is a recursive programming technique:o the problem is recursively described in terms of

smilar but smaller subproblems

1. Dynamic Programming (DP)


• Optimal solution from optimal partso An optimal (best) solution to the problem is a

composition of optimal (best) subproblem solutions

• Repeated (overlapping) sub-problemso the problem contains the same subproblems,

repeated many times

What makes DP Different?Hallmark #1

Hallmark #2


• Unix diff for comparing two files

• Smith-Waterman for genetic sequence alignmento determines similar regions between two protein

sequences (strings)

• Bellman-Ford for shortest path routing in networks

• Cocke-Kasami-Younger for parsing context free grammars

Some Famous DP Examples


• Series defined by • a0 = 1 • a1 = 1• an = an-1 + an-2

• Recursive algorithm:

• Running Time? O(2n)

2. Fibonacci Series1, 1, 2, 3, 5, 8, 13, 21, 34, …

fib(n)1. if n == 0 or n == 1, then 2. return 13. else4. a = fib(n-1)5. b = fib(n-2)6. return a+b


• Fibonacci can be viewed as a DP problem:o the optimal solution (fib(n)) is a combination of

optimal sub-solutions (fib(n-1) and fib(n-2))

o There are a lot of repeated subproblems • look at the execution graph (next slide)• 2n subproblems, but only n are different

Computing Fibonacci Faster

7


• Execution of fib(5):lots of repeated work,that only gets worsefor bigger n


• Memoization: values returned by recursive calls are stored in a table (array) so they do not need to be recalculatedo this can save enormous amount of running time

• Bottom-up algorithms: simple solutions are computed first (e.g. fib(0), fib(1), ...), leading to larger solutions (e.g. fib(23), fib(24), ...) o this means that the solutions are added to the table in a

fixed order which may allow them to be calculated quicker and the table to be less large

3. Features of DP Code


• Running time is linear = O(n)• Requires extra space for the m[] table =

O(n)

Memoization fib() Algorithm

fib(n)1.if (m[n] == 0) then2. m[n] = fib(n − 1) +

fib(n − 2) 3.return m[n] :

:

m[]0123

n-1

1100

0

::

Bottom-up fib() Algorithmint fib(int n){ if (n == 0) return 1; else { int prev = 1; int curr = 1; int temp; for (int i=1; i < n; i++) { temp = prev + curr; prev = curr; curr = temp; } return curr; }}

• Running time = O(n)

• Space requirement is 3 variables = O(1) !o this has nothing to do with

changing from recursion to a loop, but with changing from top-down to bottom-up execution, which in this case is easier to write as a loop

4. Longest Common Subsequence (LCS)

• Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both.

x: A B C B D A B

y: B D C A B A

BCBA = LCS(x, y)

andBDABBCAB


Check every subsequence of x[1 . . m] to see if it is also a subsequence of y[1 . . n].

Analysis• Checking time for each subsequence is O(n).• 2m subsequences of x[] (can use or not use each

element in x).Worst-case running time = O(n*2m), exponential time.

Brute-force LCS Algorithm

SLOW == BAD


Simplify the problem:1. Find the length of a LCS

2. We'll extend the algorithm later to find the LCS.

5. Towards a Better LCS Algorithm


• If X = < A, B, C, B, D, A, B > then

• A prefix is x[1 .. 4] == < A, B, C, B >o we abbreviate this as x4

• Also x0 is the empty sequence

Prefixes


• c[] is a table (2D array) for storing LCS lengths:c[i, j] = | LCS(x[1. . i], y[1. . j]) |

• | s | is the length of a sequence s

• Since x is of length m, and y is of length n, theno c[m, n] = | LCS(x, y) |

Creating a Table of Lengths


• Since X0 and Y0 are empty strings, their LCS is always empty (i.e. c[0, 0] == 0)

• The LCS of an empty string and any other string is empty, so for every i and j: c[0, j] == c[i, 0] == 0

Calculating LCS Lengths


Initial c[] 0 1 2 3 4 5

0

1

2

3

4

0

0

00000

0

0

0


• The first line of this definition fills the top row and first column of c[] with 0's, as in the non-recursive approach.

6. Recursive Definition of c[]


• When we calculate c[i, j], there are two cases:• First case: x[i] == y[j]: one more symbol in

strings X and Y matches, so the length of LCS Xi and Yj equals the length of LCS of smaller strings Xi-1 and Yi-1 , plus 1

otherwise]),1[],1,[max(

],[][ if1]1,1[],[

jicjicjyixjic

jic


• Second case: x[i] != y[j]• As symbols don’t match, our solution is not

improved, and the length of LCS(Xi , Yj) is the same as the biggest from before (i.e. max of LCS(Xi, Yj-1) and LCS(Xi-1,Yj)


],[][ if1]1,1[],[

jicjicjyixjic

jic


• One advantage of a recursive definition for c[] is that it makes it easy to show that c is optimal by inductiono c[i, j] increases the size of a sub-solution (line

2) or uses the bigger of two sub-solutions (line 3)

o assuming that a smaller c[] entry is optimal, then a larger c[] entry is optimal.

o when combined with the base cases, which are optimal, then c[] is an optimal solution for all entries

7. Is c[] Algorithm Optimal?


• c[] is an optimal way to calculate the length of a LCS using smaller optimal solutions (Hallmark #1)

• The c[] algorithm can be used to return the LCS (see later), so LCS also has Hallmark #1

Is LCS() Optimal?


LCS(x, y, i, j) if i == 0 or j == 0 then c[i, j] ← 0 else if x[i] == y[ j]

then c[i, j] ← LCS(x, y, i–1, j–1) + 1else c[i, j] ← max( LCS(x, y, i–1, j),

LCS(x, y, i, j–1) ) return c[i, j]

• The recursive definition of c[] has been changed into a LCS() function which returns c[]

LCS Length as Recursive Code


• Does the LCS() algorithm have many repeating (overlapping) subproblems?o i.e. does it have DP Hallmark #2?

• Consider the worst case executiono x[i] ≠ y[ j], in which case the algorithm evaluates two

subproblems, each with only one parameter decremented

8. Repeated Subproblems?


Height = m + n. The total work is exponential, butwe’re repeating lots of subproblems.

Recursion Tree (in worst cases)


• The number of distinct LCS subproblems for two strings of lengths m and n is only m*n.o a lot less than 2m+n total no. of problems

Dynamic Programming Hallmark #2


• LCS has both DP hallmarks, and so will benefit from the DP programming techniques:o recursiono memoizationo bottom-up execution

9. LCS() as a DP Problem


LCS(x, y, i, j) if c[i, j] is empty then // calculate if not already in c[i, j] if i == 0 or j == 0 then c[i, j] ← 0 else if x[i] == y[ j]

then c[i, j] ← LCS(x, y, i–1, j–1) + 1else c[i, j] ← max( LCS(x, y, i–1, j),

LCS(x, y, i, j–1) ) return c[i, j]

9.1. Memoization

Time = Θ(m*n) == constant work per table entrySpace = Θ(m*n)


• This algorithm works top-downo start with large subsequences, and calculate the smaller

subsequences

• Let's switch to bottom-up executiono calculate the small subsequences first, then move to

larger ones

9.2. Bottom-up Execution


LCS-Length(X, Y)1. m = length(X) // get the # of symbols in X2. n = length(Y) // get the # of symbols in Y

3. for i = 1 to m c[i,0] = 0 // special case: Y0

4. for j = 1 to n c[0,j] = 0 // special case: X0

5. for i = 1 to m // for all Xi 6. for j = 1 to n // for all Yj

7. if ( Xi == Yj )8. c[i,j] = c[i-1,j-1] + 19. else

c[i,j] = max( c[i-1,j], c[i,j-1] )10. return c

LCS Length Bottom-up

the same recursivedefinition of c[] asbefore


We’ll see how a bottom-up LCS works on:• X = ABCB• Y = BDCAB

10. Bottom-up Examples

LCS(X, Y) = BCBX = A B C BY = B D C A B

LCS-length(X, Y) = 3


LCS Example 1j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

X = ABCB; m = |X| = 4Y = BDCAB; n = |Y| = 5Allocate array c[5,4]

ABCBBDCAB


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

ABCBBDCAB

for i = 1 to m c[i,0] = 0 for j = 1 to n c[0,j] = 0


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

0

ABCBBDCAB

if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

0 0 0

ABCBBDCAB



j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

0 0 0 1

ABCBBDCAB



j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

000 1 1

ABCBBDCAB



j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

0 0 10 1

1

ABCBBDCAB



j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0


1000 1

1 1 11

ABCBBDCAB


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0


1000 1

1 1 1 1 2

ABCBBDCAB


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

if ( Xi == Yj )c[i,j] = c[i-1,j-1] + 1

else c[i,j] = max( c[i-1,j], c[i,j-1] )

1000 1

21 1 11

1 1

ABCBBDCAB


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0


1000 1

1 21 11

1 1 2

ABCBBDCAB


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

if ( Xi == Yj )c[i,j] = c[i-1,j-1] + 1


1000 1

1 21 1

1 1 2

1

22

ABCBBDCAB


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0


1000 1

1 21 1

1 1 2

1

22

1

ABCBBDCAB


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0

if ( Xi == Yj )c[i,j] = c[i-1,j-1] + 1


1000 1

1 21 1

1 1 2

1

22

1 1 2 2

ABCBBDCAB


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

B

Yj BB ACD

0

0

00000

0

0

0


1000 1

1 21 1

1 1 2

1

22

1 1 2 2 3

ABCBBDCAB


• The bottom-up LCS algorithm calculates the values of each entry of the array c[m, n]

• So what is the running time? O(m*n)

• Since each c[i, j] is calculated in constant time, and there are m*n elements in the array

Running Time


Example 2

m elementsin x[1..m]

n elements in y[1..n]

B == B, somax + 1


• So far, we have found the length of LCS.• We want to modify this algorithm to have it

calculate LCS of X and Y

Each c[i, j] depends on c[i-1, j] and c[i, j-1] or c[i-1, j-1]For each c[i, j] we can trace back how it was

calculated:

11. Finding the LCS

2

2 3

2 For example, here c[i, j] = c[i-1, j-1] +1 = 2+1=3


• So we can start from c[m,n] (bottom right of c[]) and move backwards

• Whenever c[i,j] = c[i-1, j-1]+1, record x[i] as part of the LCS

• When i=0 or j=0, we have reached the beginning, so can stop.

Remember that:


],[][ if1]1,1[],[

jicjicjyixjic

jic


Finding LCS Example 1j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

Yj BB ACD

0

0

00000

0

0

0

1000 1

1 21 1

1 1 2

1

22

1 1 2 2 3B


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

Yj BB ACD

0

0

00000

0

0

0

1000 1

1 21 1

1 1 2

1

22

1 1 2 2 3B


j 0 1 2 3 4 5

0

1

2

3

4

i

Xi

A

B

C

Yj BB ACD

0

0

00000

0

0

0

1000 1

1 21 1

1 1 2

1

22

1 1 2 2 3B

B C BLCS:


Reconstruct LCS by tracing backwards (start at bottom-right, follow numbers and the lines back to top-left).

LCS() = BCBA

Finding LCS for Example 2


• There may be several paths through the table, which represent different answers for LCS()

• LCS() = BDAB

• All of them have the LCS-length of 4

LCS() Offers Choices

0 0 0 0 0 0 0 00 0 1 1 1 1 1 10 0 1 1 1 2 2D 20 0 1 2 2 2 2C 20 1 1 2 2 2 3A 30 1 2 2 3 3 3B 40 1 2 2 3 3

AA B C B D B

B

A 4 4

0

4

0

1

23

1

2

34


• With this bottom-up approach, the space requirements can be reduced to O(min(m, n)) from O(m*n). Why?

• The calculation of each element in the current row only depends on 3 elements (2 from the previous row).

• Alternatively, a column can be calculated using the previous column.

• So the calculation space only requires 1 row or 1 column, which ever is smaller.

12. Space and Bottom-up

Documents

Algorithm Design and Analysis (ADA)