Structured Problems and Algorithms - Integer and quadratic

OutlineIntroduction

Integer ProgrammingQuadratic Programming

Structured Problems and AlgorithmsInteger and quadratic optimization problems

Jim Thrasher

Dept. of Engg. and Comp. Sci.,Univ. of Cal., Davis

Aug. 13, 2010

Jim Thrasher Structured Problems and Algorithms

OutlineIntroduction


Table of contents

1 Introduction

2 Integer Programming0-1 Integer Problems and MethodsBranch-and-Bound Algorithm - exampleBranch and bound - general

3 Quadratic ProgrammingQP ProblemQP ApplicationsQP Solution MethodsQP Solution by Complementary Pivot MethodExample of Complementary Pivot Algorithm


OutlineIntroduction


Benefits of Structured Problems

Optimization problems may become more tractable if we take advantageof the structured aspects of the problem rather than using generalizedapproaches. Methods for solving a variety of structured problems aredescribed in this survey.

1 Integer Programming: Some or all design variables must beinteger-valued. Both pure and mixed integer categories.

2 Quadratic Programming: Quadratic objective functions and linearconstraints.


OutlineIntroduction


Integer programming overview

Why integer-valued optimization?

1 Many practical problems require integer-valued solutions.

2 Can’t use LP solution when problem is constrained by small integerbounds.

3 Binary variables useful in selecting optimal mix of several candidateproduct inputs, task selection, or investment candidates.


OutlineIntroduction


0-1 Integer Problems and MethodsBranch-and-Bound Algorithm - exampleBranch and bound - general

Integer-constrained optimization

Integer programming problems are combinatorial as opposed tocontinuous so solutions can involve exhaustive search of finite but verylarge candidate sets. A very important precondition for solving integerprograms is reducing the size of the search set.

1 Solve LP, then truncate or round fractional values of LP optimum.

2 Maintain feasibility.

3 Enforce integer constraints only if xi < 20.

4 Search reduced set of candidate solutions.

5 Select one that is feasible and closest to LP optimum.

In practice we often accept as optimal integer solutions whose objectivefunction is within a few percent of the optimal solution. This may makethe difference between an acceptable solution or NO solution!


OutlineIntroduction



Structured Problems and Algorithms

1 Introduction




OutlineIntroduction



0-1 Integer ProblemsExample 11.1

Fixed-Charge Problem. Given a fixed number of products, made from afixed number of restricted-availability resources, determine the optimalprofitability of production.

N products

Kj fixed setup cost for j th product

Cj variable cost per unit of j th product

M resources, aij units of resource i for prod. j .

demand Demand for prod. j is dj , sells for pj per unit.

res. limits Max. of bi units of resource i available (i = 1, 2, . . . ,M)


OutlineIntroduction



Example 11.1

Formulation

δj Cost of production is nonlinear. Linearize w/binaryvariables.

Z Objective function: aggregate net profit.

δj = 0 Not producing some of the N products may result inoptimum obj. fcn.

decision δj ={

1 if product j is produced0 otherwise

}profit Maximize Z =

N∑j=1

pjxj −N∑j=1

(Kjδj + cjxj)

supplyN∑j=1

aijxj ≤ bj (i = 1, 2, . . . ,M)

demand1 xj ≤ djδj (j = 1, 2, . . . ,N)

demand2 xj ≥ 0 and δj ∈ {0, 1} ∀j ∈ {j = 1, 2, . . . ,N}


OutlineIntroduction



Example 11.2

A nonlinear 0-1 (binary) integer programming problem.

Maximize Z = x21 + x2x3 − x3

3

Subject to −2x1 + 3x2 + x3 ≤ 3

x1, x2, x3 ∈ {0, 1}Note that for any binary variable, xj , xk

j = xj . The objective functionreduces to:

Z = x1 + x2x3 − x3

x2x3 remains nonlinear. Introduce y1 = x2x3 as a new binary variable.Want y1 = 1 only when x2 = 1 ∧ x3 = 1. Use following constraints:

x2 + x3 − y1 ≤ 1

−x2−x3 +2y1 ≤ 0

x2 = x3 = 1 y1 ≥ 1∧ y1 ≤ 1⇒ y1 = 1

x2 = 0 ∨ x3 = 0 y1 ≤ x2+x3

2 ⇒ y1 = 0


OutlineIntroduction



Example 11.2

(Original) Z = x21 + x2x3 − x3

3

Fully linearized 0-1 problem.

Maximize Z = x1 + y1 − x3

Subject to −2x1 + 3x2 + x3 ≤ 3,

x2 + x3 − y1 ≤ 1,

−x2 − x3 + 2y1 ≤ 0,

x1, x2, x3, y1 ∈ {0, 1}

1 Use continuous variables rather than integer.

2 Generalize method for transforming products of binary vars.


OutlineIntroduction




1 Introduction




OutlineIntroduction



Branch-and-Bound Algorithm

use Most widely used for solving pure and mixed IPs.

use Most commercial packages are based on B-and-B.

approach A method for sharply reducing solution search set.

approach LPs with continuous variables are methodicallytransformed into MIPs.


OutlineIntroduction



Branch-and-Bound Algorithm

If LP optimal solution contains fractional values for some integervars., then these variables must be integers for the optimal integersolution.

A reasonable search will begin with integers near to the LP result,e.g. floor or ceiling for the variable under consideration.

But it may be that the optimal integer solution for a given variablewill not even be the nearest integer greater or less than the LP result.

Branch-and-bound systematically explores integer-valued candidatesnear the LP optimum values.

Even though B-and-B eliminates a very large percentage of ’possible’integer-valued inputs, the total computation can remain veryexpensive.


OutlineIntroduction



Branch-and-Bound AlgorithmExample Mixed-integer program

Solve the following MIP using the branch-and-bound method:

Maximize Z = 3x1 + 2x2

C1 x1 ≤ 2

C2 x2 ≤ 2

C3 x1 + x2 ≤ 3.5

C4 x1, x2 ∈ NFirst, solve this as an LP, allowing:

CLP4 x1, x2 ∈ R

The LP optimal solution is: x1 = 2, x2 = 1.5.The maximum of the objective function, Z , is: Z0 = 9.The following figure illustrates this result.


OutlineIntroduction



Branch-and-Bound AlgorithmLP-1

x2

x10

1

2

1 2

x2 = 2 (1.5, 2)

(2, 1.5)

x1 = 2

x1 + x2 = 3.5

z=

3=

3x1 +

2x2

z=

7=

3x1 +

2x2

z0 =

9(max)

LP-1


C1 x1 ≤ 2

C2 x2 ≤ 2

C3 x1 + x2 ≤ 3.5

Solution:

x1 = 2

x2 = 1.5

Obj Z0 = 9

NOT optimal for MIP.

Z0 = 9 is an upperbound.

Now try int. vals. for x2


OutlineIntroduction




x2

x10

1

2

1 2

x2 = 1(2, 1)

x1 = 2

z=

3 z0 =

8(max)

LP-2


C1 x1 ≤ 2

C2 x2 ≤ 1

C3 x1 + x2 ≤ 3.5

C4 x1, x2 ∈ N

Solution:

x1 = 2

x2 = 1

Obj Z0 = 8

Z0 = 8 is a lower bound.Now try x2 ≥ 2


OutlineIntroduction




x2

x10

1

2

1 2

A(1.5, 2)

B

z=

3

z0 =

8.5

LP-3


C1 x1 ≤ 2

C2, C2a x2 ≤ 2∧ x2 ≥ 2

C3 x1 + x2 ≤ 3.5

C4 x2 ∈ N

Solution:

x1 = 1.5

x2 = 2

Obj Z0 = 8.5

Z0 = 8.5 not feasiblex1 = 1.5 /∈ N


OutlineIntroduction




x2

x10

1

2

1 2

D(1, 2)

E

z=

3

z0 =

7

LP-4


C1 x1 ≤ 1

C2 x2 ≤ 2∧ x2 ≥ 2

C3 x1 + x2 ≤ 3.5

C4 x1, x2 ∈ N

Solution:

x1 = 1

x2 = 2

Obj Z0 = 7

Z0 = 7 < lower boundx1 ≥ 2 violates C3.


OutlineIntroduction




1 Introduction




OutlineIntroduction



Structure of branch and bound solutionTree-structured solution process

The process of solving a MIP by branch and bound method results in atree structure whose nodes represent tentative choices for the set ofinteger constraints to the original LP problem at the root node.

Tree Diagram of branch and bound procedure

Node 1: initial LP problem (LP-1) → Frac

Node 2: constrain x2 ≤ 1 (LP-2) → Int

Node 3: constrain x2 ≥ 2 (LP-3) → Frac

Node 4: constrain x1 ≥ 2 (LP-4) → Int

Node 5: constrain x1 ≥ 2 (LP-5) → IF

After all nodes are fathomed, we select the best of the integer solutions:(LP-2). The next slide illustrates this result...


OutlineIntroduction



Branch-and-Bound Algorithm: LP-1...5Tree diagram

Node 1

LP− 1

Node 2

LP− 2

Node 3

LP− 3

Node 4

LP− 4

Node 5

LP− 5

x1 = 2, x2 = 1.5,

z0 = 9

x1 = 2, x2 = 1,

z0 = 8

x1 = 1.5, x2 = 2,

z0 = 8.5

x1 = 1, x2 = 2,

z0 = 7

(Optimal)

(Fathomed)

(Infeasible)

x2 ≤ 1 x2 ≥ 2

x1 ≤ 1 x1 ≥ 2


OutlineIntroduction



Steps of branch and bound method (MIP)Initial process

Initial solution bounding...

Solve MIP as LP-1 w/o integer restrictions, yielding Z1.

Take Z1 as an upper bound on the optimum integer solution.

Partition feasible region of (LP-1) by branching on an integervariable at a fractional value.

Seek a feasible integer solution to serve as an initial lower bound

Partitioning rules...

Select IV with largest fractional value in the LP solution

Assign priorities on IVs. Branch on top priority one.

(a) Represents an important decision in model(b) Cost/profit coeff. is very large(c) Value high based on past experience

Branch on arbitrary choice.

Next we illustrate how this might proceed...Jim Thrasher Structured Problems and Algorithms

OutlineIntroduction



Steps of branch and bound methodFathoming and solution rules

Selecting a branch node...

1. Use Optimal Value of Obj F Branch from node w/max Z.

2. Last-In-First-Out Rule Branch from most recently solved LP.

Finding first integer solution...(this might be the hardest step)

DO Choose IV, xj w/fractional value

Solve LP In feasible region w/xj integer constrained

Add node Check if integer solution obtained yet

WHILE No integral solution for Z found.

Set lwr bound Zlb is the highest-valued int. sol.

Continue fathoming nodes that might yield Z > Zlb, until all pathsblocked. This is illustrated in the next frame (Fig. 11.6)...


OutlineIntroduction



Branch-and-Bound Algorithm: LP-1...7Tree diagram (general)

Node 1

LP− 1

Node 2

LP− 2

Node 4

LP− 4

Node 5

LP− 5

Node 3

LP− 3

Node 6

LP− 6

Node 7

LP− 7

Fractionalz0 = z1

Fractionalz0 = z2

Fractionalz0 = z3

Integer

z0 = z4

Fractionalz0 = z6

Fractionalz0 = z7

Infeasible

xj ≤ bβjc xj ≥ dβje

xi ≤ bβic xi ≥ dβie xk ≤ bβkc xk ≥ dβke


OutlineIntroduction



Steps of branch and bound methodFathoming and solution rules

Solution Time

The time it takes to solve the Integer Problem is very sensitive to howthe problem is formulated initially!

Guidelines based on practical experience at IBM Research:

1 Keep set of IV’s as small as possible: No int constraints if xi > 20.

2 Provide tight lwr/upr bound on integer variables when possible.

3 Freely add new (integer) constraints to reduce total solution time.

4 Accept first int solution that is within 0.03 of continuous optimum.

5 Choose branching variables based on experience-dictated importance.

Branch-and-bound can also be used to solve nonlinear IP. See Gupta andRavindran for implementation of B-and-B using generalized reducedgradient (GRG). They also provide a good survey (ca. 1981) of B-and-Bheuristics.


OutlineIntroduction



Branch and bound for nonlinear programsGeneral method [Gupta and Ravindran, 1985]

The general branch and bound method also handles convex nonlinearmixed integer programs (NLMIP).

Nonlinear mixed integer program definition

Maximize: f (x),

Subject to: gi (x) ≥ 0,

hk(x) = 0,

xj ∈ Z

x = (x1, x2, . . . , xn) ∈ Rn,

i = 1, 2, . . . ,NI ,

k = 1, 2, . . . ,NE ,

j = 1, 2, . . . ,m ≤ n

The first m of x1, x2, . . . , xn are identified as IV.Selecting which values to restrict to integers depends on theproblem. The nonlinearity may increase difficulty.Solving NLMIP ignoring integer restrictions satisfies first ordersufficient optimality conditions.Hence a Kuhn-Tucker point is a global extremum.


OutlineIntroduction


QP ProblemQP ApplicationsQP Solution MethodsQP Solution by Complementary Pivot MethodExample of Complementary Pivot Algorithm


1 Introduction




OutlineIntroduction



What is a quadratic program?QP definition

A QP problem is an optimization problem involving a quadratic objectivefunction and linear constraints.

QP problem (scalar)

Minimize: f (x) =n∑

j=1

cjxj +n∑

i=1

n∑j=1

xijqijxij

Subject to:n∑

j=1

aijxj = bi for i = 1, . . . ,m

xj ≥ 0

QP problem (vector)

Minimize: f (x) = cx + xTQx c(1×n), x(n×1), Q(n×n)

Subject to: Ax = b b(m×1), A(m×n)

x ≥ 0Jim Thrasher Structured Problems and Algorithms

OutlineIntroduction




1 Introduction




OutlineIntroduction



Applications of QP 1Portfolio Selection with risk minimization

Selecting a portfolio of securities to generate a ’maximum return’ while’minimizing risk’ results in a Quadratic Program formulation.

QP for risk-averse portfolio

Minimize: Z = xTQx Minimize overall risk.

Subject to:N∑j=1

xj ≤ C A minimum requred cash reserve.

xj ≥ 0 Only positive amounts invested in xj

µTx ≥ R Total return must be at least $ R

Q(N×N) = [qij ] is the variance-covariance matrix of the N securities.

µT is the average annual return for the securities.


OutlineIntroduction



Applications of QP 2Constrained regression problem

The basic regression problem requires minimizing an error vector thatarises from the stochastic relation between independent and dependentvariables of the problem.

L-S regression problem

Minimize: f (x) = eTIe e(n×1)

Subject to: Y = Xβ + e X(n×m), β(m×1)

Types of additional constraints applied:

βj ≥ 0 and∑βj = 1.

Error terms weighted: eTAe where A is the weighting matrix


OutlineIntroduction




1 Introduction




OutlineIntroduction



Kuhn-Tucker Conditions

Quadratic programs are a restriction of general nonlinear programs. TheQP optimization is of the form:

KTC for QP

Minimize: f (x) = cx + xTQx

Subject to: g(x) = x ≥ 0 Inequality constraints.

h(x) = Ax − b = 0 Equality constraints.

The associated Kuhn-Tucker conditions for QPs:

Solve: c + xT(Q + QT)− u − vA = 0

Ax = b x ≥ 0

ux = 0 u ≥ 0

u ≥ 0 v unrestricted in sign


OutlineIntroduction



QP: Kuhn-Tucker ConditionsKTC Example 11.6

Illustration of Kuhn-Tucker conditions for a Quadratic Program.Minimize: f (x) = −6x1 + 2x2

1 − 2x1x2 + 2x22

Subject to: x1 + x2 = 2 x1, x2 ≥ 0The coefficient arrays and vectors are:

c = (−6, 0) Q =

(2 −1−1 2

)A = (1, 1) b = (2)

The Kuhn-Tucker conditions are

(−6, 0) + (x1, x2)

(4 −2−2 4

)− (u1, u2)− v1(1, 1) = 0

−6 + 4x1 − 2x2 − u1 − v1 = 0

0− 2x1 + 4x2 − u2 − v1 = 0

x1 + x2 = 2 x1, x2 ≥ 0

u1x1 = 0 u2x2 = 0

u1, u2 ≥ 0 v1 unrestricted in signJim Thrasher Structured Problems and Algorithms

OutlineIntroduction



QP: Kuhn-Tucker ConditionsKTC Example 11.6

What this example illustrates...

1 Because the constraints of QP are linear, the constraint qualificationis always satisfied → Kuhn-Tucker necessity theorem applies.

2 Therefore the Kuhn-Tucker conditions are necessary for theoptimality of QP.

3 When Q is positive (semi-)definite the objective function is convexand KTC are sufficient for optimum solution of QP.

4 Then solving KTC finds an optimal solution to QP.

In 1959, Wolfe first used phase I simplex method to solve KTC for QP. Itwas only effective in solving QP for Q positive definite. It fails toconverge for Q positive semidefinite. Lemke (1965) introduced thecomplementary pivot method which allows us to solve QP efficientlywhen Q is positive semidefinite.


OutlineIntroduction




1 Introduction




OutlineIntroduction



Complementary Pivot MethodComplementary problem

The complementary pivot method for solving QPs uses the notion ofcomplementary problem.

Complementary problem

Solve: w = Mz + q (w , z , q)(n×1), M(n×n)

Subject to: w ≥ 0, z ≥ 0

w ′z = 0

Note: no objective function to optimize. Also...

1 Solving a system of simultaneous linear equations.

2 A valid solution will be nonnegative.

3 A single nonlinear constraint.


OutlineIntroduction



Complementary Pivot MethodComplementary problem - sample

Consider a problem with

M =

1 2 34 5 66 7 8

q =

2−5−3

The complementary problem: Find w1,w2,w3, z1, z2, z3 s.t.

w1 = z1 + 2z2 + 3z3 + 2

w2 = 4z1 + 5z2 + 6z3 − 5

w3 = 6z1 + 7z2 + 8z3 − 3

constrained by:

w1,w2,w3, z1, z2, z3 ≥ 0

w1z1 + w2z2 + w3z3 = 0

The optimal solution to a QP may be obtained from an equivalentcomplementary problem formed from the KTC of the QP.


OutlineIntroduction



Complementary Pivot Method

Consider a convex quadratic problem of the form

Convex QP

Minimize: f (x) = cx + xTQx Q(n×n), symm, pos def/semidef

Subject to: Ax ≥ b and x ≥ 0

The KTC optimality conditions to the above system can be written

KTC of Convex QP

Solve: u = 2Qx − ATy + cT

v = Ax − b

Subject to: x , y , u, v ≥ b and uTx + vTy = 0

To cast it as a complementary problem, we use

w =

(uv

)z =

(xy

)M =

(2Q −AT

A 0

)q =

(cT

−b

)Jim Thrasher Structured Problems and Algorithms

OutlineIntroduction



Complementary Pivot MethodConvex QP Example 11.8

Illustration of Kuhn-Tucker conditions for a Quadratic Program.

Minimize: f (x) = −6x1 + 2x21 − 2x1x2 + 2x2

2

Subject to: −x1 − x2 ≥ −2 and x1, x2 ≥ 0

The coefficient arrays and vectors are:

A = (−1,−1) b = (−2) cT =

(−6

0

)Q =

(2 −1−1 2

)The equivalent complementary problem is given by

M(3×3) =

(Q + QT −AT

A 0

)=

4 −2 1−2 4 1−1 −1 0

, and q =

(cT

−b

)=

−602

The values of z1 and z2 correspond to the optimal values of x1 and x2

since the QP matrix Q is positive definite.


OutlineIntroduction



Complementary Pivot MethodDevelopment of method - 1

Complementary problem

Solve: w = Mz + q (w , z , q)(n×1), M(n×n)

Subject to: w ≥ 0, z ≥ 0

wTz = 0

1 A nonnegative solution (w , z) to the CP is called a feasible solution2 A feasible solution (w , z) to the CP that also satisfies the

complementarity condition wTz = 0 is called a complementarysolution.

Remarks...

wTz = 0 ≡ ∀i : wizi = 0.(wi , zi ) is called a complementary pair.If q ≥ 0, one solution is w = q, z = 0.Want at least one element of q negative for nontrivial solution.In that case, w = q, z = 0 is infeasible to CP.


OutlineIntroduction




The complementary pivot method posed by Lemke (1965)

Begin with the infeasible solution w = q, z = 0.

Introduce the artificial variable z0.

This will result in an almost complimentary solution.

Basic solution

z0 = −min(qi ) in

wi = qi + z0 zi = 0 ∀i = 1, . . . , n

This is a solution, but it is not feasible.We call it an almost complimentary solution

The original equation system must be augmented before we can applythe complementary pivot method...


OutlineIntroduction




Augmented system

Solve: w −Mz − ez0 = q

Subject to: w , z , z0 ≥ 0

wTz = 0

where e(n×1) = (1, 1, . . . , 1)T

The initial tableau becomes (qs is the most negative element of q):

Initial Tableau

Basis w1 · ws · wn z1 · zs · zn z0 q

w1 1 −m11 −m1s −m1n −1 q1

ws 1 −ms1 −mss −msn −1 qs

wn 1 −mn1 −mns −mnn −1 qn


OutlineIntroduction



Complementary Pivot MethodAlgorithm - 1

z0 replaces ws from the basis. Performing the pivot operation yields thenew tableau where

q′s = −qs q′i = qi − qs , ∀i 6= s

m′sj =−msj

−1= msj ∀j = 1, . . . , n

m′ij = −mij + msj ∀j = 1, . . . , n and i 6= s

Tableau after first pivot

Basis w1 · ws · wn z1 · zs · zn z0 q

w1 1 −1 0 −m′11 −m′1s −m′1n 0 q′1

z0 0 −1 0 −m′s1 −m′ss −m′sn 1 q′s

wn 0 −1 1 −m′n1 −m′ns −m′nn 0 q′n


OutlineIntroduction




The complementary pivot method in a finite number of pivottransformations leads from an almost complementary solution to acomplementary solution.

Notes on pivot series

1 q′i ≥ 0, i = 1, . . . , n.

2 Basic solution, w1 = q′1, . . . ,ws−1 = q′s−1, z0 = q′s , . . . ,wn = q′n, andall other vars initially 0 is an almost complementary solution.

3 The almost complementary solution becomes a complementarysolution as soon as z0 is reduced to 0.

4 In each step, maintain wizi = 0 for all i = 1, . . . , n.

5 Basic solution remains ≥ 0: qi ’s must be ≥ 0 in all tableaus.


OutlineIntroduction




Step 2 and 3

1 complementary rule: Add complement to basis of variable that justleft basis in last tableau, e.g. if ws was taken out, bring zs in.

2 Before adding new basic variable, determine which to remove using

minimum-ratio test. Find minmis>0q′i

m′is

.

3 The index from the minimum ratio test, k , determines which wk toreplace by the complementary zk .

4 (Step 3) Now that zk has entered basis, continue pivot andreplacement process until the termination condition.

Termination conditions

1 Minimum ratio obtained in row s and z0 leaves basis. After pivot,we have the complementary solution

2 Minimum ratio fails → no solution to complementary problem exists.


OutlineIntroduction



Complementary Pivot MethodSummary

Remarks

1 The complementary pivot method always terminates with acomplementary solution in a finite number of steps whenever

(a) all elements of M are positive, or(b) M has positive principal determinants (incl.M positive definite).

2 The minimum-ratio test fails, since all coeff. in the pivot column arenonpositive. This implies no solution: CP has a ray solution

See Cottle and Dantzig (1967) for a full development with proofs.


OutlineIntroduction




1 Introduction




OutlineIntroduction



Complementary Pivot MethodExample of algorithm

Tableau 2

Basis w1 w2 w3 z1 z2 z3 z0 q

w1 1 0 0 −4 2 −1 −1 −6

w2 0 1 0 2 −4 −1 −1 0

w3 0 0 1 1 1 0 −1 2

Tableau 3


z0 −1 0 0 4 −2 1 1 6

w2 −1 1 0 6 −6 0 0 6

w3 −1 0 1 6 −1 1 0 8


OutlineIntroduction




Tableau 4


z0 − 13 − 2

3 0 0 2 1 1 2

z1 − 16

16 0 1 −1 0 0 1

w3 − 16 − 5

6 1 0 4 1 0 3

Tableau 5


z0 − 14 − 1

4 − 12 0 0 1

2 1 12

z1 − 524 − 1

2414 1 0 1

4 0 74

z2 − 524 − 5

2414 0 1 1

4 0 34


OutlineIntroduction




Tableau 6


z3 − 12 − 1

2 −1 0 0 1 2 1

z1 − 112 − 1

1212 1 0 0 − 1

232

z2 − 112 − 1

1212 0 1 0 − 1

212


Documents

Structured Problems and Algorithms - Integer and quadratic