Design of Algorithms by Induction

Part 2

Bibliography: [Manber]- Chap 5

Design of algorithms by induction

• Represents a fundamental design principle that underlies techniques such as divide & conquer, dynamic programming and even greedy

• Question: how to reduce the problem to a smaller problem or a set of smaller problems ? (n -> n-1, n/2, n/4, …?)

• Examples:– The successful party problem [Manber 5.3]– The celebrity problem [Manber 5.5]– The skyline problem [Manber 5.6]– One knapsack problem [Manber 5.10]– The Max Consecutive Subsequence [Manber 5.8]

Part 1

Part 1

Conclusions (1)

• What is Design by induction ?• An algorithm design method that uses the idea

behind induction to solve problems– Instead of thinking about our algorithm as a sequence of steps

to be executed, think of proving a theorem that the algorithm exists

• We need to prove that this “theorem” holds for a base case, and that if it holds for “n-1” this implies that it holds for “n”

Conclusions (2)

• Why/when should we use Design by induction ?– It is a well defined method for approaching a large variety of

problems (“where do I start ?”)• Just take the statement of the problem and consider it is a theorem

to be proven by induction

– This method is always safe: designing by induction gets us to a correct algorithm, because we designed it proving its correctness

– Encourages abstract thinking vs coding => you can handle the reasoning in case of complex algorithms

• A[1..n]; A[n/4+1 .. 3n/4]

• L, R; L=(R+3L+1)/4, R=(L+3R+1)/4

– We can make it also efficient (see next slide)

Conclusions (3)

• The inductive step is always based on a reduction from problem size n to problems of size <n. How to efficiently make the reduction to smaller problems:– Sometimes one has to spend some effort to find the suitable

element to remove. (see Celebrity Problem)– If the amount of work needed to combine the subproblems is not

trivial, reduce by dividing in subproblems of equal size – divide and conquer (see Skyline Problem)

The Knapsack Problem

• This is one of the multiple variants of the knapsack problem, that applies to packaging goods in standard containers of a shipping company that have to be exactly filled up

• The problem: Given an integer K and n items of different sizes such that the i’th item has an integer size ki, find a subset of the items whose sizes sum to exactly K, or determine that no such subset exist

• P(n,K)• P(i,k) – the first i items and a knapsack of size k

Knapsack - Try 1

• Induction hypothesis: We know how to solve P(n-1, K)• Base case: n=1: there is a solution if the single element

is of size K• Inductive step: • Case 1: P(n-1,K) has a solution: we simply do not use the n’th item,

we already have the solutiom• Case 2: P(n-1,K) has no solution: this means that we must use the

n’th item of size kn. This implies that the rest of the items must fit into a smaller knapsack of size K- kn.We have reduced the problem to two smaller subproblems: P(n-1, K) and P(n-1, K-kn).

• In order to solve this, we need to strenghten the hypothesis

Knapsack - Try 2 – Solution 1

• Induction hypothesis: We know how to solve P(n-1, k) for all 0<=k<=K

• Base case: n=1: there is a solution if the single element is of size k

• Inductive step: • P(n-1, k) and P(n-1, k-kn).

• We have reduced the problem of size n to 2 problems of size n-1

• T(n)=2 T(n-1) + O(1)• T(n) is O(2n)

Knapsack – Solution 2

• Solution 1 solved 2n problems P(i,k)

• Actually, the total number of distinct problems P(i,k) is n*K

• 2n – n*K is redundant work !

• Solution: we store all the known results of all problems P(i,k) in an n*K matrix

Knapsack - Conclusions

• Divide-and-conquer algorithms partition the problem into disjoint subproblems, solve the subproblems, and then combine their solutions to solve the original problem.

• When the subproblems overlap, a divide-and-conquer algorithm does more work than necessary, repeatedly solving the common subsubproblems.

• In order to avoid repeated work, the solutions of subproblems are saved and when the same subproblem appears again the saved solution is used

• This technique is Dynamic programming

Finding the Maximum Consecutive Subsequence

• Problem: Given a sequence X = (x1, x2, …, xn) of (not necessarily positive) real numbers, find a subsequence xi; xi+1; … ; xj of consecutive elements such that the sum of the numbers in it is maximum over all subsequences of consecutive elements

• Example: The profit history (in billion $) of the company ProdIncCorp for the last 10 years is given below. Find the maximum amount that ProdIncCorp earned in any contiguous span of years.

y1 y2 y3 y4 y5 y6 y7 y8 y9 y10

2 -3 1.5 -1 3 -7 0.5 0.5 1 -2

Max consec subsequence – Try 1

• Base case: n=1: we have a single element and it is the max subsequence

• Induction hypothesis: We know how to find the maximum subsequence in a sequence of length < n.

• We know(we assume) that the maximum consecutive subsequence of x1;…; xn-1 is xi… xj , j<=n-1

• We have to find the maximum consecutive subsequence of x1;…; xn-1; xn – We distinguish 2 cases:

• Case 1: j=n-1 (the max subseq is a suffix of the given one)• Case 2: j<n-1 (the max subseq is no suffix of the given one)

Max consec subsequence – Try 1

• Case 1: j=n-1 (the max subseq is a suffix of the given one)

y1 y2 y3 y4 y5

2 -3 1.5 -1 3

y1 y2 y3 y4 y5 y6

2 -3 1.5 -1 3 -7

n-1 n

n-1 n

If x[n] >0 then the max seq will be xi;… xj; xn

If x[n] <0 then the max seq remains xi;… xj

Max consec subsequence – Try 1

• Case 2: j<n-1 (the max subseq is not a suffix)

y1 y2 y3 y4 y5 y6 y7

2 -3 1.5 -1 3 -7 0.5

n-1 n

y1 y2 y3 y4 y5 y6 y7 y8 y9

2 -3 1.5 -1 3 -7 0.5 0.5 8

n-1 n

We need to know also the maximum subsequence that is a suffix !

Max consec subsequence – Try 2

• Stronger Induction hypothesis: We know how to find the maximum subsequence in a sequence of length < n and the maximum subsequence that is a suffix

• Base case: when n=0, both subsequences are empty (their sums are 0)

• Inductive step: We add xn to the max suffix. If this sum is bigger than the global max subsequence, then we update it (as well as the new suffix). Otherwise we retain the previous max subsequence. We also need to find the new max suffix, and add xn to its old value. If this sum results negative, we take the empty set as max suffix.

Max consec subseq - SolutionAlgorithm Max_Subsequence(IN: X[1..n], OUT: GlobalMax, SuffixMax)

beginif (n=0) then

GlobalMax:=0; SuffixMax:=0;

else Max_Subsequence(X[1..n-1], GlobalMax1, SuffixMax1); GlobalMax:=GlobalMax1; if x[n] + SuffixMax1 > GlobalMax1 then

SuffixMax := SuffixMax1 + x[n];GlobalMax := SuffixMax;

else if x[n] + SuffixMax1 > 0 thenSuffixMax := SuffixMax1 + x[n];

elseSuffixMax := 0;

end T(n)=T(n-1)+cAlgorithm is O(n)

Max consec subseq - Solution

Algorithm Max_Subsequence(X,n)

Input: X (array of length n)

Output: Global_Max (The sum of the maximum subsequence)

begin

Global_Max:= 0;

Suffix_Max := 0;

for i=1 to n do

if x[i] + Suffix_Max > Global_Max then

Suffix_Max := SuffixMax + x[i];

Global_Max := Suffix_Max;

else if x[i] + Suffix_Max > 0 then

Suffix_Max := Suffix_Max + x[i];

else

Suffix_Max := 0;

end

It was straightforward to rewrite the solution in a non-recursive way !

Max consec subseq - Conclusion

• Sometimes we cannot prove that P(<n) => P(n)

• We can add an additional statement Q and prove easier • (P(< n) and Q(< n) ) => P(n)• Q is a property of the solution, property that we must

discover

• Attention: Q(n) must become part of the induction hypothesis, we must in fact prove:

• (P(< n) and Q(< n) ) => ( P(n) and Q(n))

This is Strengthen the Induction Hypothesis

Conclusions (4)

• The inductive step is always based on a reduction from problem size n to problems of size <n. How to efficiently make the reduction to smaller problems:– A reduction to several subproblems might eventually compute

the same subproblems again and again. In this case, when the subproblems are not disjoint, try to store these intermediate results – dynamic programming. (see Knapsack Problem)

Conclusions (5)

• What if induction does not succeed for the given problem statement?:– A straightforward reduction might not give sufficient information

to extend the result, in this case one can try to strengthen the induction hypothesis. (see Maximum Subsequence Problem)

• Try to find relevant properties of the solution and add these to the induction hypothesis