1

IS703:Decision Support and Optimization

Week 5: Mathematical Programming

Lau Hoong ChuinSchool of Information Systems

2

Mathematical Programming - Scope

• Linear Programming

• Integer Programming

• Network flows (an introduction in previous class)

• Not Covered:– Convex Programming– Non-Linear Programming– etc

3

Linear Programming(Adapted from http://www.cs.princeton.edu/~wayne/cs423/lectures/lp.ppt)

Powerful model generalizes many classic problems:– shortest path, max flow, multicommodity flow, MST, matching, 2-

person zero sum gamesEssential tool for optimal allocation of scarce resources, among a number of competing activities.Ranked among most important scientific advances of 20th century.

– accounts for a major proportion of all scientific computationHelps find "good" solutions to NP-hard optimization problems.

– optimal solutions (branch-and-cut)– provably good solutions (randomized rounding)

4

Linear ProgrammingLP "standard" form.

Input data: rational numbers cj, bi, aij .Maximize linear objective function.Subject to linear inequalities.

njx

mibxa

xc

j

in

jjij

n

jjj

≤≤≥

≤≤≤∑

∑

=

=

10

1t. s.

max(P)

1

1

0t. s.

max(P)

≥≤

•

xbAx

xc

5

LP: GeometryGeometry.

Forms an n-dimensionalpolyhedron.

Convex: if y and z are feasible solutions, then so is αy + (1-α)z, for all 0<α<1.

njx

mibxa

xc

j

in

jjij

n

jjj

≤≤≥

≤≤≤∑

∑

=

=

10

1t. s.

max(P)

1

1

z

y

zy

Convex Not convex

Extreme points

6

LP: GeometryExtreme Point Theorem. If there exists an optimal solution to standard form LP (P), then there exists one that is an extreme point.

Only need to consider finitely many possible solutions.

Greed. Local optima areglobal optima.

7

LP: AlgorithmsSimplex. (Dantzig 1947)

Developed shortly after WWII in response to logistical problems:used for 1948 Berlin airlift.Practical solution method that moves from one extreme point to a neighboring extreme point.Finite (exponential) complexity, but no polynomial implementation known.

8

LP: Polynomial AlgorithmsEllipsoid. (Khachian 1979, 1980)

Solvable in polynomial time: O(n4 L) bit operations.– n = # variables – L = # bits in input

Theoretical tour de force.Not remotely practical.

Karmarkar's algorithm. (Karmarkar 1984)O(n3.5 L).Polynomial and reasonably efficientimplementations possible.

Interior point algorithms. O(n3 L).Competitive with simplex!

– will likely dominate on large problems soonExtends to even more general problems.

9

LP Application: Weighted Bipartite MatchingAssignment problem. Given a complete bipartite network Kn,n and edge weights cij, find a perfect matching of minimum weight.

Birkhoff-von Neumann theorem (1946, 1953). All extreme points of the above polyhedron are {0-1}-valued.

Corollary. Can solve assignment problem using LP techniques since LP algorithms return optimal solution that is an extreme point.

Remarks. (1) The above theorem is directly proven by applying the concept of totally unimodular matrix.

(2) Polynomial combinatorial algorithms also exist.

njix

njx

nix

xc

ij

niij

njij

njijij

ni

≤≤≥

≤≤=∑

≤≤=∑

∑∑

≤≤

≤≤

≤≤≤≤

,10

11

11t. s.

min

1

1

11

10

LP Application: Multicommodity FlowMulticommodity flow. Given a network G = (V, E) with edge capacities u(e) ≥ 0, edge costs c(e) ≥ 0, set of commodities K, and supply / demand dk(v) for commodity k at node v, find a minimum cost flow that satisfies all of the demand.

Applications.Transportation networks.Communication networks.VLSI design.

KkEeex

Eeeuex

KkVvvdexex

exec

kKk

k

k

ve

k

ve

kEe

kk

Kk

∈∈≥

∈≤

∈∈=−

∑

∑∑

∑∑

∈

∈∈

,0)(

)()(

,)()()( t.s.

)()(min

ofout in to

Remark. The problem is NP-hard if flows are integral!

Integer ProgrammingAn Integer Programming model is a linear programming problem where some or all of the variables are required to be non-negative integers.

These models are in general substantially harder than solving linear programming models.

Network models are special cases of integer programming models and are very efficiently solvable.

Several applications of integer programming models.

Branch and bound technique, one of the most popular algorithm to solve integer programming models.

12

Integer ProgrammingProblem: Given rational numbers aij, bi, cj, find integers xj that solve:

Claim. INTEGER-PROGRAMMING is NP-hard.Proof. VERTEX-COVER ≤ P INTEGER-PROGRAMMING.

njxnjx

mibxa

xc

j

j

in

jjij

n

jjj

≤≤≤≤≥

≤≤≥∑

∑

=

=

1integral10

1t. s.

min

1

1

13

Weighted Vertex CoverWeighted vertex cover. Given an undirected graph G = (V, E) with vertex weights wv ≥ 0, find a minimum weight subset of nodes S such that every edge is incident to at least one vertex in S.

NP-hard even if all weights are 1.

Integer programming formulation.

If x* is optimal solution to (ILP),then S = {v ∈ V : x*v = 1} is a minweight vertex cover.

3

6

10

7

A

E

H

B

D I

C

F

J

G

6

16

10

7

23

9

10

9

33

Total weight = 55.

32

VvxEwvxx

xwILP

v

wv

Vvvv

∈∈∈≥+

∑∈

}1,0{),(1t. s.

min)(

Classifications of IP ModelsPure IP Model: Where all variables must take integer values.

Maximize z = 3x1 + 2x2subject to x1 + x2 ≤ 6

x1, x2 ≥ 0, x1 and x2 integer

Mixed IP Model: Where some variables must be integer while others can take real values.

Maximize z = 3x1 + 2x2subject to x1 + x2 ≤ 6

x1, x2 ≥ 0, x1 integer

0-1 IP Model: Where all variables must take values 0 or 1.

Maximize z = x1 - x2subject to x1 + 2x2 ≤ 2

2x1 - x2 ≤ 1, x1, x2 = 0 or 1

Classifications of IP Models (contd.)LP Relaxation: The LP obtained by omitting all integer or 0-1 constraints on variables is called the LP relaxation of IP.

IP:Maximize z = 21x1 + 11x2subject to

7x1 + 4x2 ≤ 13x1, x2 ≥ 0, x1 and x2 integer

LP Relaxation:

Maximize z = 21x1 + 11x2subject to

7x1 + 4x2 ≤ 13x1, x2 ≥ 0

Fact:Optimal objective value of IP ≤

Optimal objective value of LP relaxation

Solving IP via LP relaxation

Solve the LP relaxationRound-off the solution to the nearest feasible integer solution

Any potential difficulty with this approach?

x x xx

xx

xx

x1

x2

1 2 3

1

3

2

7x1 + 4x2= 13

Capital Budgeting ProblemStockco Co. is considering four investments

It has $14,000 available for investment

Formulate an IP model to maximize the NPV obtained from the investments

IP:

Maximize z = 16x1 + 22x2 + 12x3 + 8x4subject to

5x1 + 7x2 + 4x3 + 3x4 ≤ 14x1, x2,,x3, x4 = 0, 1

Investmentchoice

1 2 3 4

Cashoutflow

$5000 $7000 $4000 $3000

NPV $16000 $22000 $12000 $8000

Fixed Charge ProblemGandhi cloth company manufactures three types of clothing: shirts, shorts, and pants

Machinery must be rented on a weekly basis to make each type of clothing. Rental Cost:

$200 per week for shirt machinery$150 per week for shorts machinery$100 per week for pants machinery

There are 150 hours of labor available per week and 160 square yards of cloth

Find a solution to maximize the weekly profitLabor hr Cloth yd Var. cost Price

Shirts 3 4 $12 $6Shorts 2 3 $8 $4Pants 6 4 $15 $8

Fixed Charge Problem (contd.)Decision Variables:

x1 = number of shirts produced each weekx2 = number of shorts produced each week x3 = number of pants produced each week

y1 = 1 if shirts are produced and 0 otherwisey2 = 1 if shorts are produced and 0 otherwisey3 = 1 if pants are produced and 0 otherwise

Formulation:

Max. z = 6x1 + 4x2 + 7x3 - 200y1 - 150 y2 - 100y3subject to

3x1 + 2x2 + 6x3 ≤ 1504x1 + 3x2 + 4x3 ≤ 160x1 ≤ M y1, x2 ≤ M y2, x3 ≤ M y3

x1, x2,,x3 ≥ 0, and integer; y1, y2,,y3 ≥ 0, and integer

Either-Or ConstraintsDorian Auto is considering manufacturing three types of auto: compact, midsize, large.

Resources required and profits obtained from these cars are given below.

We have 6,000 tons of steel and 60,000 hours of labor available.

If any car is produced, we must produce at least 1,000 units of that car.

Find a production plan to maximize the profit.Compact Midsize Large

Steel Req. 1.5 tons 3 tons 5 tonsLabor Req. 30 hours 25 hours 40 hoursProfit $2000 $3000 $4000

Either-Or Constraints (contd.)Decision Variables:

x1, x2, x3 = number of compact, midsize and large cars producedy1, y2, y3 = 1 if compact , midsize and large cars are produced or

not

Formulation:

Maximize z = 2x1 + 3x2 + 4x3subject tox1 ≤ My1; x2 ≤ My2; x3 ≤ My31000 - x1 ≤ M(1-y1)

1000 - x2 ≤ M(1-y2)1000 - x3 ≤ M(1-y3)

1.5 x1 + 3x2 + 5x3 ≤ 600030 x1 + 25x2 + 40 x3 ≤ 60000

x1, x2, x3 ≥ 0 and integer; y1, y2, y3 = 0 or 1

Set Covering ProblemsWestern Airlines has decided to have hubs in USA.

Western runs flights between the following cities: Atlanta, Boston, Chicago, Denver, Houston, Los Angeles, New Orleans, New York, Pittsburgh, Salt Lake City, San Francisco, and Seattle.

Western needs to have a hub within 1000 miles of each of these cities.Determine the minimum number of hubs

Cities within 1000 milesAtlanta (AT) AT, CH, HO, NO, NY, PIBoston (BO) BO, NY, PIChicago (CH) AT, CH, NY, NO, PIDenver (DE) DE, SLHouston (HO) AT, HO, NOLos Angeles (LA) LA, SL, SFNew Orleans (NO) AT, CH, HO, NONew York (NY) AT, BO, CH, NY, PIPittsburgh (PI) AT, BO, CH, NY, PISalt Lake City (SL) DE, LA, SL, SF, SESan Francisco (SF) LA, SL, SF, SESeattle (SE) SL, SF, SE

Formulation of Set Covering ProblemsDecision Variables:

xi = 1 if a hub is located in city ixi = 0 if a hub is not located in city i

Minimize xAT + xBO + xCH + xDE + xHO + xLA + xNO + xNY + xPI + xSL+ xSF + xSE

subject toAT BO CH DE HO LA NO NY PI SL SF SE Required

AT 1 0 1 0 1 0 1 1 1 0 0 0 xAT >= 1BO 0 1 0 0 0 0 0 1 1 0 0 0 xBO >= 1CH 1 0 1 0 0 0 1 1 1 0 0 0 xCH >= 1DE 0 0 0 1 0 0 0 0 0 1 0 0 xDE >= 1HO 1 0 0 0 1 0 1 0 0 0 0 0 xHO >= 1LA 0 0 0 0 0 1 0 0 0 1 1 0 xLA >= 1NO 1 0 1 0 1 0 1 0 0 0 0 0 xNO >= 1NY 1 1 1 0 0 0 0 1 1 0 0 0 xNY >= 1PI 1 1 1 0 0 0 0 1 1 0 0 0 xPI >= 1SL 0 0 0 1 0 1 0 0 0 1 1 1 xSL >= 1SF 0 0 0 0 0 1 0 0 0 1 1 1 xSF >= 1SE 0 0 0 0 0 0 0 0 0 1 1 1 xSE >= 1

IP: Logistics/Supply Chain Applications

Set covering and partitioningScheduling airline crewsManpower rostering

Facility locationLocating warehouses or franchises

Multicommodity distributionDistributing auto parts

Traveling salesman / Vehicle routingRouting pickups and deliveries

Exercise: Model TSP as an IP

Branch and BoundThe most popular approach for solving integer programming problems

Enumerate the entire solution space but only implicity; hence they are called implicit enumeration algorithms.

A general-purpose solution technique which must be specialized for individual IP's.

Running time grows exponentially with the problem size, but small to moderate size problems can be solved in reasonable time.

A divide and conquer algorithm, where a problem is divided into smaller and smaller subproblems. Each subproblem is solved separately, and the best solution is taken.

26

Branch-and-Bound Search

• Represent search space as trees• Search tree

– Each leaf node represents a feasible solution– Non-leaf node represents a subset of feasible solutions – Parent-child relationship represents partition of search

space into sub-spaces• Idea: if we can drastically reduce solution space

from one level to next, total number of nodes explored will be reduced

27

Branch-and-Bound Search

S

S1

S2S3

S4

S1 S2 S3 S4

S24

(a)

(b)

(c)

S1

S4

S21

S22

S31

S32

*

*

* = does not contain optimal solution

S1 S2 S3 S4

S21 S22 S23

Figure 1: Illustration of the search space of B&B.

28

Elements of Branch-and-Bound1. Branching rule

• Rule for dividing the solution space into sub-spaces (sub-problem)2. Node-selection (search/exploration) strategy

• Select next node to be explored next• DFS, BFS, Best-First Search

3. Bounding function • Function that computes the lower bound for the best solution

obtainable in that search space4. Upper bounding

• Process of obtaining an upper bound for each subproblem5. Pruning strategy

• If for a subproblem, UB ≤ LB, then the subproblem need not be explored further

29

Bounding Function

g

S

P

f

Figure 2: The relation between the bounding function g and the objective func-tion f on the sets S and P of feasible and potential solutions of a problem.

30

Branch-and-Bound Algorithm (sketch)• Start with complete feasible solution space and a naïve

initial solution• upper bound = objective value of best solution found so far• Among unexplored nodes do

– Select a non-leaf node based on selection strategy– Partition the feasible space into sub-spaces based on

branching rule– For each sub-space do

• If leaf node reached and beats best solution, update best solution, else discard and exit

• Compute lower-bound• If lower bound >= upper bound, prune the sub-space • Else create a sub-space node and add to list of

unexplored nodes

31

Example: TSP• Find optimal tour by systematically considering all

tours• Strategy 1 (exhaustive search):

– Consider all permutations of the cities (vertices)– Evaluate each tour (given by particular vertex order)– Keep track of shortest tour– (n-1)! permutations, each takes O(n) time to evaluate

• Don’t look at all n permutations, since we don’t care about starting point of tour: A,B,C,(A) is same tour as C,A,B,(C)

– Unacceptable for large n• Strategy 2 (dynamic programming):

– O(n22n)

32

Example: TSP• Strategy 3: Branch-and-Bound• Construct a tree with a root that represents all tours

(tours are feasible solutions)• Each node has two children

– One child represents tours with a particular edge– Other child represents tours without that edge– Choose edges in lexicographic order, i.e. ab, ac, ad, etc

33

TSP Search Tree

all tours

tours with(a,b)

tours without(a,b)

tours with(a,b)

and (a,c)

tours with(a,b)

not (a,c)

tours with(a,c)

not (a,b)

tours without(a,b)

or (a,c)

tours with(a,b)

and (a,c)not (a,d)

tours with(a,b) and(a,d) not

(a,c)

tours with(a,b) not(a,c) or(a,d)

tours with(a,d) not(a,b) or

(a,c)

tours without(a,d),

(a,b) or(a,c)

not shown:pruned

(infeasible)

34

Lower bound• Compute a lower bound on the cost of feasible

solutions “below” node n– If current solution is better (lower) than lower bound for

node n, need not explore any nodes below n• For TSP, lower bound on cost of tour

– What’s an obvious loose lower bound?– Tighter bound: half the sum over all nodes n of the 2

least cost edges incident upon n– More precisely, let e1(n) and e2(n) be the two least cost

edges incident to n, then LB = (∑n∈V e1(n)+e2(n)) / 2– Proof. Left as an exercise. (Hint: consider situation at

each node, and sum up over all nodes.)

35

Examplea

c

b

e

d

33

44

8

7

2

6

6

5

2 least cost edges adjacent to a are: (a,d) and (a,b) for total of 2+3=52 least cost edges adjacent to b are: (a,b) and (b,e) for total of 3+3=6c, d, e: total of 8, 7, 9 respectively⇒ lower bound on cost of tour is (5+6+8+7+9)/2 = 17.5

36

Lower Bound for Subset of Tours• Lower bound on the cost of node in search tree

– Each node represents tours defined by a set of edges that must be in tour and set of edges that may not be included

• Assume we must include (a,e) and exclude (b,c)a: (a,d) and (a,e), total 9 // must include (a,e) instead of (a,b)b: (a,b) and (b,e), total 6c: (a,c) and (c,d), total 9 // cannot use (b,c)d: (a,d) and (c,d), total 7e: (a,e) and (b,e), total 10 // must include (a,e) instead of (e,d)

⇒ lower bound = (9+6+9+7+10)/2 = 20.5

37

Reconstruct Search Tree• Each time we branch, decide on which edge to

include/exclude based on:– Include (x,y) if it becomes impossible for x or y to have two adjacent

edges in tour otherwise– Exclude (x,y) if it raises degree of x or y above 2– Exclude (x,y) if it would complete a non-tour cycle with selected

edges

• After branch, compute lower bounds for both children

• Prune child whose lower bound is as high or higher than best tour found so far

• If neither child can be pruned, explore child with smaller lower bound first

38

New Search Tree

B 17.5ab

C 18.5ab

D 20.5ac ad ae

E 18ac

G 23ad ae

F 18ad ae

H

bc bd bece de cd

tour=abcedacost=23

I

bc cd cebd de be

tour=abecdacost=21

pruned after discovery of I

pruned after discovery

of I

J 18.5ac

K 21ac ad ae

pruned

M 23.5ad ae

L 18.6ad ae

pruned

N

bc cd cebd de be

tour=acbedacost=19

P

bc bd bece de cd

tour=acebdacost=23

A 17.5no constraints

LB

39

Analysis

• Search space: still (n-1)! in worst-case• But much better on average

– Depends on the 3 elements of B&B

40

A simple way to get the initial solution:

Set xi = 1 for i = 1,2,… until the constraint is violated.

e.g. x1=1, x2=1, x3=0, x4=0.

max z = 16x1+22x2+12x3+8x4

subject to the following constraints:

5x1+7x2+4x3+3x4 <= 14

xj= 0 or 1, for all j

Lower bound = 16x1 + 22x2 = 38

Any sub-space that has an upper bound <=38 can be pruned. What is a good upper bound?

0-1 Knapsack Problem

Integer programming (IP)formulation

41

Upper Bounding Function:Fractional Knapsack Problem

0,5.0

,1

4

3

21

==

==

xx

xx :solution LPj p j w j ratio ranking

1 16 5 3.2 12 22 7 3.1 23 12 4 3.0 34 8 3 2.7 4

value ingnonincreas to according items order :it solve Toj

j

wp

Greedy! i.e. possible, as large as order this in s' the make and jx

Linear programming(LP) Relaxation

max z = 16x1+22x2+12x3+8x4

subject to the following constraints:

5x1+7x2+4x3+3x4 <= 14

0<= xj<=1, for all j

42

Branch and Bound TreeSubproblem 1z = 44, LB=38

x1 = x2 = 1x3 =.5

1

x3 = 1

2

Subproblem 3z = 43.4, LB=38

x1 =x3= 1, x2 = .7, x4=0

Subproblem 6z = 42, LB=38x1 =0, x2=x3=1

x4 = 15

Subproblem 7LB = 42

Infeasible6

x1 = 0 x1 = 1

x3 = 0

7

Subproblem 2z = 43.3, LB=38

x1 = x2=1x3 = 0, x4 =.67

x4 = 1

8 9

Subproblem 8z = 38, LB=42

x1 = x2=1x3 = x4 = 0

Subproblem 9z= 42.85, LB=42

x1 = x4 =1x3 = 0, x2 = .85

x4 = 0

pruned

x2 = 0

Subproblem 5z = 43.6, LB=38x1 =.6, x2=x3=1

x4 = 0

Subproblem 4z = 36, LB=38

x1 = x3=1x2 = 0, x4 =1

x2 = 1

3 4

pruned

LB=42

pruned

43

Practical Developments• Good formulations, heuristics and theory

Goal: to get LP solution as close as possible to IP solutionDisaggregation, adding constraints (cuts)

• PreprocessingAutomatic methods for reformulationSome interesting graph theory is involved

• Cut generation (branch-and-cut)Add cuts during the branch-and-bound

• Column generationImprove formulation by introducing an exponential number of variables.

44

Exercise: Generalized Assignment Problem

In this problem, we assign m jobs to n machines– Machine: The total available time of machine j is Tj

– Jobs: If job i is performed on machine j, it costs aij, and requires tij time units. A job must be assigned to one machine in its entirety

The problem is to find a minimum cost assignment of all jobs.

Design a B&B approach to find the optimal solution.