1

Design and Analysis of Algorithms

Yoram MosesLecture 7

April 22, 2010

http://www.ee.technion.ac.il/courses/046002

2

Linear Programming

3

Standard Form

Maximization All constraints are ≤ inequalities All variables are non-negative

Objective function

Constraints

Non-negativity constraints

4

Slack Form Maximization n+m variables and m constraints n “regular variables” and m “slack variables” All variables are non-negative All constraints, except for non-negativity, are equalities Slack variables appear only on LHS of equality constraints Each equality constraint has a single unique slack variable

5

Conversion to Slack Form: Example

Standard LP

Slack LP

6

Basic vs. Non-basic Variables

Slack variables basic variables Regular variables non-basic variables B = set of basic variables, N = set of non-basic variables

Ex: B = { 4,5,6 }, N = { 1,2,3 } |B| = m, |N| = n For a solution x, xB = basic part of x, xN = non-basic part of x

7

Short Matrix Form

|B| = m, |N| = n Constraints can be indexed by members of B For a solution x,

xB = basic part of x (i.e., xi for all i B) xN = non-basic part of x (i.e., xj for all j N)

8

Basic Solutions

Every slack form is associated with a basic solution: All non-basics are set to 0 (i.e., xN = 0)

All basics are set to corresponding free coefficients (i.e., xB = b)

Ex: x1 = 0, x2 = 0, x3 = 0, x4 = 30, x5 = 24, x6 = 36

9

Tight Constraints & Constraint Violation

Tight constraint: One in which the basic variable is forced to 0 in the basic solutionThe corresponding free coefficient is 0.

Violated constraint: One in which the basic variable is forced to be negative in the basic solution. the corresponding free coefficient is negative.

10

Basic Feasible Solutions

Basic Feasible Solution (BFS): A basic solution, which is feasible

Easy fact: A basic solution is feasible if and only if all free coefficients are non-negative.

Lemma: Every BFS corresponds to a vertex of the feasible region polytope.

11

Simplex Algorithm: Overview

Works in iterations At each iteration: transform one slack form

P into an equivalent slack form P’ Objective value of basic solution of P’ is

always at least as good as that of P Stop when reaching a local optimum

12

Moving to an Equivalent Slack Form How to increase the objective value of the BFS?

Increase from 0 the value of some non-basic variable, whose coefficient in the objective function is positive.

By how much? As much as possible without violating any of the constraints.

objective: 0

Can increase x1 by at most 9. Objective increases to 27.

13

Switching Basic with Non-basic Suppose that we increase a non-basic variable xi until

some constraint j becomes tight xj, the basic variable of the constraint j, becomes 0

We can thus switch between xi and xj

xi will become the basic variable of constraint j

xj will become a non-basic

When increasing x1 to 9, x6 becomes 0.

We switch between x1 and x6.

14

Switching Basic with Non-basic

We write x1 as a function of other non-basics and x6:

15

Switching Basic with Non-basic

We rewrite the objective function:

16

Switching Basic with Non-basic

We rewrite all the constraints as well, and obtain the following equivalent linear program:

17

Switching Basic with Non-basic

The new LP is equivalent to the previous LP We just rewrote x1 in terms of other variables

Basic solution: x1 = 9, x2 = 0, x3 = 0, x4 = 21, x5 = 6, x6 = 0 New objective value: 27

18

Example continued

Choose x3

First constraint to become tight is constraint 3.

objective: 27

objective: 27.75

19

Example continued

Choose x2

First constraint to become tight is constraint 2.

objective: 27.75

objective: 28

20

Example continued

No more non-basics whose coefficient in the objective function is positive We stop and output basic solution as the optimal solution Solution: x1 = 8, x2 = 4, x3 = 0, x4 = 18, x5 = 0, x6 = 0. Value: 28.

objective: 28

21

Back to the Standard Form

Solution: x1 = 8, x2 = 4, x3 = 0. Value: 28.

objective: 28

22

Pivoting

Pivot: a single iteration of the simplex algorithm1. Choose a non-basic variable xi whose coefficient in the

objective function is > 0 xi is called the “entering variable” If more than one exists, choose one according to some pivoting

rule

2. Find the first constraint j that will be violated when we increase the value of xi from 0 If more than one exists, choose one according to some pivoting

rule

3. Make xi the basic variable of constraint j, and make xj a non-basic variable xj is called the “leaving variable”

4. Write xi as a function of xj and the other non-basics5. Rewrite the objective function and the constraints

23

Pivoting: Geometric Intuition

Lemma: Pivoting corresponds to moving from one vertex of the feasible region to a neighbor vertex, whose objective value is at least as good.

24

Unbounded Programs

Sometimes it is possible to increase the value of the entering value unboundedly, without violating any constraint

In this case the optimal solution of the LP is unbounded Pivot will return “unbounded”

25

The Simplex Algorithm

1. find an initial BFS2. while there is a non-basic variable whose coefficient in

the objective function is > 03. run pivot4. if pivot returns “unbounded” return “unbounded”5. return BFS of current slack form as the optimal solution

Geometric view: Repeatedly move from a vertex of the feasible region to a better neighbor vertex, until a local maximum is reached.

Initial BFS is found by solving an auxiliary linear program (read section in book)

26

Simplex Analysis: Correctness

If LP is infeasible, S’x will fail to find an initial BFS If LP is unbounded, Pivot will return “unbounded” If LP has a bounded optimal solution, it has one at

a vertex Simplex will reach a local maximum vertex Local maximum vertex must be a global maximum Hence, Simplex will output an optimal solution

27

Simplex Analysis: Running Time

We have not specified the two “pivoting rules” For choosing the entering variable For choosing the leaving variable

Degeneracy: objective value of BFS does not improve in an invocation of Pivot

Unwise pivoting rules may lead to infinite loops (i.e., everlasting degeneracy)

28

Degeneracy

objective: 0 entering: x1

leaving: x4

objective: 8 entering: x3

leaving: x5

objective: 8 entering: x2

leaving: x1

29

Pivoting Rules

Bland’s rule: choose entering/leaving variable with smallest index.

Lemma: If Simplex uses Bland’s rule, it never cycles.

Conclusion: Simplex has at most iterations. Theorem [Kalai]: There is a randomized pivoting

rule, with which Simplex runs for a sub-exponential number of iterations in expectation.

Open problem: Is there a pivoting rule with which Simplex runs in polynomial time?

30

End of Lecture 8