Semidefinite Programming (SDP) and the Goemans … · Semidefinite Programming (SDP) and the Goemans-Williamson MAXCUT Paper ... . Outline

Semidefinite Programming (SDP) and the Goemans-Williamson

MAXCUT Paper

Robert M. Freund

September 8, 2003

This presentation is based on: Goemans, Michel X., and David P. Williamson. Improved Approximation Algorithms for Maximum Cut and

Satisfiability Problems Using Semidefinite Programming. Journal of the ACM 42(6), November 1995, pp. 1115-1145.www.bsscommunitycollege.in www.bssnewgeneration.in www.bsslifeskillscollege.in

1www.onlineeducation.bharatsevaksamaj.net www.bssskillmission.in

WWW.BSSVE.IN

Outline

• Alternate View of Linear Programming

• Facts about Symmetric and Semidefinite Matrices

• SDP

• SDP Duality

• Approximately Solving MAXCUT using SDP and Random Vectors

• Interior-Point Methods for SDP

2003 Massachusetts Institute of Technology 2www.bsscommunitycollege.in www.bssnewgeneration.in www.bsslifeskillscollege.in


WWW.BSSVE.IN

∑

Linear Alternative Perspective

Programming

LP : minimize c · x

s.t. ai · x = bi, i = 1, . . . ,m

nx ∈ �+.

n“c · x” means the linear function “ j=1 cjxj ”

n n�+ := {x ∈ � | x ≥ 0} is the nonnegative orthant.

n is a convex cone .�+

K is convex cone if x, w ∈ K and α, β ≥ 0 ⇒ αx + βw ∈ K.



WWW.BSSVE.IN


Programming

LP : minimize c · x

s.t. ai · x = bi, i = 1, . . . ,m

nx ∈ �+.

“Minimize the linear function c · x, subject to the condition that x must solve m given equations ai · x = bi, i = 1, . . . ,m, and that

nx must lie in the convex cone K = �+.”



WWW.BSSVE.IN

∑

∑

∑ ∑ ( )


Programming LP Dual Problem...

m LD : maximize yibi

i=1 m

s.t. yiai + s = c i=1

ns ∈ �+.

For feasible solutions x of LP and (y, s) of LD, the duality gap is simply

m m

c · x − yibi = c − yiai · x = s · x ≥ 0 i=1 i=1



WWW.BSSVE.IN

∑


Programming ...LP Dual Problem

∗ ∗If LP and LD are feasible, then there exists x ∗ and (y , s ) feasible for the primal and dual, respectively, for which

m ∗ ∗ ∗ ∗ c · x − yibi = s · x = 0

i=1



WWW.BSSVE.IN

Semidefinite Cone

Facts about the

If X is an n × n matrix, then X is a symmetric positive semidefinite (SPSD) matrix if X = XT and

vTXv ≥ 0 for any v ∈ �n

If X is an n × n matrix, then X is a symmetric positive definite (SPD) matrix if X = XT and

vTXv > 0 for any v ∈ �n, v �= 0



WWW.BSSVE.IN

Semidefinite Cone

Facts about the

Sn denotes the set of symmetric n × n matrices

Sn + denotes the set of (SPSD) n × n matrices.

Sn ++ denotes the set of (SPD) n × n matrices.



WWW.BSSVE.IN

Facts about the SemidefiniteCone

Let X, Y ∈ Sn .

“X � 0” denotes that X is SPSD

“X � Y ” denotes that X − Y � 0

“X 0” to denote that X is SPD, etc.

Remark: Sn = {X ∈ Sn | X � 0} is a convex cone.+



WWW.BSSVE.IN

Facts about Eigenvalues and Eigenvectors

If M is a square n × n matrix, then λ is an eigenvalue of M with corresponding eigenvector q if

Mq = λq and q �= 0 .

Let λ1, λ2, . . . , λn enumerate the eigenvalues of M .



WWW.BSSVE.IN

( ) [ ]


The corresponding eigenvectors q1, q2, . . . , qn of M can be chosen so that they are orthonormal, namely

i)T (

qjq = 0 for i � ( i)T ( i

) = j, and q q = 1

Define: 2 nQ := q 1 q · · · q

Then Q is an orthonormal matrix:

QTQ = I, equivalently QT = Q−1



WWW.BSSVE.IN

[ ]


λ1, λ2, . . . , λn are the eigenvalues of M 1q , q2, . . . , qn are the corresponding orthonormal eigenvectors of

M 2 nQ := q 1 q · · · q

Q−1QTQ = I, equivalently QT =

Define D: λ1 0 0 0 λ2 D := . . . .

0 λn

Property: M = QDQT .



WWW.BSSVE.IN


The decomposition of M into M = QDQT is called its eigendecomposition.



WWW.BSSVE.IN

Facts about SymmetricMatrices

• If X ∈ Sn, then X = QDQT for some orthonormal matrix Q and some diagonal matrix D. The columns of Q form a set of n orthogonal eigenvectors of X, whose eigenvalues are the corresponding entries of the diagonal matrix D.

• X � 0 if and only if X = QDQT where the eigenvalues (i.e.,the diagonal entries of D) are all nonnegative.

• X 0 if and only if X = QDQT where the eigenvalues (i.e.,the diagonal entries of D) are all positive.



WWW.BSSVE.IN

∏


• If M is symmetric, then n

det(M ) = λj

j=1



WWW.BSSVE.IN

( )


• Consider the matrix M defined as follows:

P v M = T ,

v d

where P 0, v is a vector, and d is a scalar. Then M � 0 if TP −1and only if d − v v ≥ 0.

• For a given column vector a, the matrix X := aaT is SPSD, i.e., X = aaT � 0.

• If M � 0, then there is a matrix N for which M = NTN . To 1

see this, simply take N = D2QT .



WWW.BSSVE.IN

SDP X

Semidefinite Programming

Think about

Let X ∈ Sn. Think of X as:

• a matrix

• an array of n2 components of the form (x11, . . . , xnn)

• an object (a vector) in the space Sn .

All three different equivalent ways of looking at X will be useful.



WWW.BSSVE.IN

∑ ∑

Semidefinite Programming SDP

Linear Function of X

Let X ∈ Sn. What will a linear function of X look like?

If C(X) is a linear function of X, then C(X) can be written as C • X, where

n n

C • X := CijXij. i=1 j=1

There is no loss of generality in assuming that the matrix C is also symmetric.



WWW.BSSVE.IN

SDP Semidefinite Programming

Definition of SDP

SDP : minimize C • X

s.t. Ai • X = bi , i = 1, . . . ,m,

X � 0,

“X � 0” is the same as “X ∈ Sn” +

The data for SDP consists of the symmetric matrix C (which is the data for the objective function) and the m symmetric matrices A1, . . . ,Am, and the m−vector b, which form the m linear equations.



WWW.BSSVE.IN


Example...

1 0 1 0 2 8 ( ) 1 2 3

11 A1 = 0 3 7 , A2 = 2 6 0 , b = , and C = 2 9 0 ,19

1 7 5 8 0 4 3 0 7

The variable X will be the 3 × 3 symmetric matrix: x11 x12 x13 X = x21 x22 x23 , x31 x32 x33

SDP : minimize x11 + 4x12 + 6x13 + 9x22 + 0x23 + 7x33

s.t. x11 + 0x12 + 2x13 + 3x22 + 14x23 + 5x33 = 11 0x11 + 4x12 + 16x13 + 6x22 + 0x23 + 4x33 = 19

x11 x12 x13

X = x21 x22 x23 � 0.x31 x32 x33



WWW.BSSVE.IN


...Example

SDP : minimize x11 + 4x12 + 6x13 + 9x22 + 0x23 + 7x33

s.t. x11 + 0x12 + 2x13 + 3x22 + 14x23 + 5x33 = 11 0x11 + 4x12 + 16x13 + 6x22 + 0x23 + 4x33 = 19

x11 x12 x13

X = x21 x22 x23 � 0.x31 x32 x33

It may be helpful to think of “X � 0” as stating that each of the n eigenvalues of X must be nonnegative.



WWW.BSSVE.IN


LP ⊂ SDP

LP : minimize c · x s.t. ai · x = bi, i = 1, . . . ,m

nx ∈ �+.

Define: ai1 0 . . . 0 c1 0 . . . 0 0 0 . . . 0 a

.i2 . . . 0

. , i = 1, . . . , m, and C = ..c..2

. . . .. .Ai = .. . . . . . . . . . . . 0 0 . . . ain 0 0 . . . cn

SDP : minimize C • X s.t. Ai • X = bi , i = 1, . . . , m,

Xij = 0, i = 1, . . . , n, j = i + 1, . . . , n, x1 0 . . . 0 0 x2 . . . 0 . � 0,X = .. .. . . . . . . . 0 0 . . . xn



WWW.BSSVE.IN

∑

∑

∑

SDP Duality

m SDD : maximize yibi

i=1

m s.t. yiAi + S = C

i=1

S � 0.

Notice m

S = C − yiAi � 0 i=1



WWW.BSSVE.IN

∑

∑

SDP Duality

and so equivalently:


i=1

m s.t. C − yiAi � 0

i=1



WWW.BSSVE.IN

ExampleSDP Duality

( ) 1 2 31 0 1 0 2 8 11 A1 = 0 3 7 , A2 = 2 6 0 , b = , and C = 2 9 019

1 7 5 8 0 4 3 0 7

SDD : maximize 11y1 + 19y2 1 0 1 0 2 8 1 2 3 s.t. y1 0 3 7 + y2 2 6 0 + S = 2 9 0 1 7 5 8 0 4 3 0 7

S � 0



WWW.BSSVE.IN

ExampleSDP Duality

SDD : maximize 11y1 + 19y2 1 0 1 0 2 8 1 2 3 s.t. y1 0 3 7 + y2 2 6 0 + S = 2 9 0 1 7 5 8 0 4 3 0 7

S � 0 is the same as:

SDD : maximize 11y1 + 19y2

s.t. 1 − 1y1 − 0y2 2 − 0y1 − 2y2 3 − 1y1 − 8y2 2 − 0y1 − 2y2 9 − 3y1 − 6y2 0 − 7y1 − 0y2 � 0. 3 − 1y1 − 8y2 0 − 7y1 − 0y2 7 − 5y1 − 4y2



WWW.BSSVE.IN

∑

∑

Weak Duality SDP Duality

Weak Duality Theorem: Given a feasible solution X of SDP and a feasible solution (y, S) of SDD, the duality gap is

m

C • X − yibi = S • X ≥ 0 . i=1

If m

C • X − yibi = 0 , i=1

then X and (y, S) are each optimal solutions to SDP and SDD, respectively, and furthermore, SX = 0.



WWW.BSSVE.IN

SDP Duality Strong Duality

∗ ∗Strong Duality Theorem: Let z and zD denote the optimal P

objective function values of SDP and SDD, respectively. Suppose that there exists a feasible solution X of SDP such that X 0, and that there exists a feasible solution (ˆ ˆˆ y,S) of SDD such that S 0. Then both SDP and SDD attain their optimal values, and

∗ ∗ zP = zD .



WWW.BSSVE.IN

SDP

Some Important Weaknesses of

• There may be a finite or infinite duality gap.

• The primal and/or dual may or may not attain their optima.

• Both programs will attain their common optimal value if bothprograms have feasible solutions that are SPD.

• There is no finite algorithm for solving SDP .

• There is a simplex algorithm, but it is not a finite algorithm. There is no direct analog of a “basic feasible solution” for SDP .



WWW.BSSVE.IN

The MAX CUT Problem

M. Goemans and D. Williamson, Improved Approximation Algorithms for Maximum Cut and Satisf iability Problems using Semidef inite Programming, J. ACM 42 1115-1145, 1995.



WWW.BSSVE.IN

The MAX CUT Problem

G is an undirected graph with nodes N = {1, . . . , n} and edge set E.

Let wij = wji be the weight on edge (i, j), for (i, j) ∈ E.

We assume that wij ≥ 0 for all (i, j) ∈ E.

The MAX CUT problem is to determine a subset S of the nodes N for which the sum of the weights of the edges that cross from S to its complement ¯ S := N \ S).S is maximized (



WWW.BSSVE.IN

∑ ∑

The MAX CUT Formulations

Problem

The MAX CUT problem is to determine a subset S of the nodes N for which the sum of the weights wij of the edges that cross from S to its complement ¯ S := N \ S).S is maximized (

¯Let xj = 1 for j ∈ S and xj = −1 for j ∈ S.

n n 1MAXCUT : maximizex 4 wij(1 − xixj )

i=1 j=1

s.t. xj ∈ {−1, 1}, j = 1, . . . , n.



WWW.BSSVE.IN

∑ ∑


Problem

n n 1MAXCUT : maximizex 4 wij(1 − xixj)

i=1 j=1

s.t. xj ∈ {−1,1}, j = 1, . . . , n.

Let Y = xxT .

Then Yij = xixj i = 1, . . . , n, j = 1, . . . , n.



WWW.BSSVE.IN

∑ ∑


Problem

Also let W be the matrix whose (i, j)th element is wij for i = 1, . . . , n and j = 1, . . . , n. Then

n n MAXCUT : maximizeY,x

1 wij (1 − Yij)4 i=1 j=1

s.t. xj ∈ {−1,1}, j = 1, . . . , n

Y = xxT .



WWW.BSSVE.IN

∑ ∑


Problem


1 wij (1 − Yij)4 i=1 j=1

s.t. xj ∈ {−1, 1}, j = 1, . . . , n

Y = xxT .



WWW.BSSVE.IN

∑ ∑


Problem

The first set of constraints are equivalent to Yjj = 1, j = 1, . . . , n.


1 wij (1 − Yij)4 i=1 j=1

s.t. Yjj = 1, j = 1, . . . , n

Y = xxT .



WWW.BSSVE.IN

∑ ∑


Problem


1 wij (1 − Yij)4 i=1 j=1

s.t. Yjj = 1, j = 1, . . . , n

Y = xxT .

Notice that the matrix Y = xxT is a rank-1 SPSD matrix.



WWW.BSSVE.IN

∑ ∑


Problem

We relax this condition by removing the rank-1 restriction:

n n 1RELAX : maximizeY 4 wij (1 − Yij )

i=1 j=1

s.t. Yjj = 1, j = 1, . . . , n

Y � 0.

It is therefore easy to see that RELAX provides an upper bound on MAXCUT, i.e.,

MAXCUT ≤ RELAX.



WWW.BSSVE.IN

∑ ∑

( )

The MAX CUT Computing a Good Solution

Problem

n n 1RELAX : maximizeY 4 wij (1 − Yij )

i=1 j=1

s.t. Yjj = 1, j = 1, . . . , n

Y � 0.

Let Y solve RELAX

Factorize ˆ = V T ˆY ˆ V

vT ˆˆ v1 ˆ ˆ Yij = V T ˆ = i vjV = [ˆ v2 · · · vn] and ˆ ˆ V ij



WWW.BSSVE.IN

( )


Problem

Let ˆY solve RELAX

Factorize ˆ = V T ˆY ˆ V

vT ˆˆ v1 ˆ ˆ Yij = V T ˆ = i vjV = [ˆ v2 · · · vn] and ˆ ˆ V ij

Let r be a random uniform vector on the unit n-sphere Sn

S := {i | rT vi ≥ 0}

S := {i | rT vi < 0}



WWW.BSSVE.IN


Problem

Proposition:

vT ˆ( vi) � vj)

) =

arccos(ˆ vj )iP sign(rT ˆ = sign(rT ˆ . π



WWW.BSSVE.IN

Vi

Vj ^

0

The MAX CUT Problem

Computing a Good Solution



WWW.BSSVE.IN

The MAX CUT Problem


Let r be a random uniform vector on the unit n-sphere Sn

S := {i | rT vi ≥ 0}

S := {i | rT vi < 0}

Let E[Cut] denote the expected value of this cut.

Theorem: E[Cut] ≥ 0.87856 × MAXCUT



WWW.BSSVE.IN

∑ ( )

∑


Problem

vi) � vj )E[Cut] = 1 wij × P sign(rT ˆ = sign(rT ˆ2 i,j

T1 ∑ arccos( i ˆv vj )= 2 wij π

i,j

1 ∑ arccos(Yij )= 2 wij π i,j

= 1 wij arccos(Yij )2π i,j



WWW.BSSVE.IN

∑

( )

( )


Problem

E[Cut] = 1 wij arccos(Yij )2π i,j

= ∑

wij 1 − ˆ 2 arccos( ˆ1 Yij )Yij Yij4 π 1− i,j ∑ 2 arccos(t)wij 1 − ˆ≥ 1 Yij min−1≤t≤1 π4 1−t i,j

2 θ= RELAX × min0≤θ≤π π 1−cos θ ≥ RELAX × 0.87856



WWW.BSSVE.IN

The MAX CUT Problem


So we have

MAXCUT ≥ E[Cut] ≥ RELAX × 0.87856 ≥ MAXCUT × 0.87856

This is an impressive result, in that it states that the value of the semidefinite relaxation is guaranteed to be no more than 12.2% higher than the value of NP -hard problem MAXCUT.



WWW.BSSVE.IN

The Logarithmic Barrier Function for SPD Matrices

Let X � 0, equivalently X ∈ Sn .+

X will have n nonnegative eigenvalues, say λ1(X), . . . , λn(X) ≥ 0 (possibly counting multiplicities).

∂Sn = {X ∈ Sn | λj(X) ≥ 0, j = 1, . . . , n, +

and λj(X) = 0 for some j ∈ {1, . . . , n}}.



WWW.BSSVE.IN

∑ ∏


∂Sn = {X ∈ Sn | λj(X) ≥ 0, j = 1, . . . , n, +

and λj(X) = 0 for some j ∈ {1, . . . , n}}. A natural barrier function is:

n n

B(X) := − ln(λi(X)) = − ln λi(X) = − ln(det(X)). j=1 j=1

This function is called the log-determinant function or the logarithmic barrier function for the semidefinite cone.



WWW.BSSVE.IN

∑ ∏

( ) ( )

∑


n n

B(X) := − ln(λi(X)) = − ln λi(X) = − ln(det(X)). j=1 j=1

¯Quadratic Taylor expansion at X = X:

¯ ¯ X−1 ¯ 2DX−1 ¯ 2DX−1 B(X + αD) ≈ B(X) + α ¯ • D +

1 α2 X−1 ¯ 2 • X−1 ¯ 2 .

2

B(X) has the same remarkable properties in the context of interior-point methods for SDP as the barrier function

n− j=1 ln(xj ) does in the context of linear optimization.



WWW.BSSVE.IN

∑

∑

∑

Interior-point Primal and Dual SDP

Methods for SDP

SDP : minimize C • X s.t. Ai • X = bi , i = 1, . . . , m,

X � 0 and


i=1 m

s.t. yiAi + S = C i=1 S � 0 .

If X and (y, S) are feasible for the primal and the dual, the duality gap is: m

C • X − yibi = S • X ≥ 0 . i=1

Also, S • X = 0 ⇐⇒ SX = 0 .



WWW.BSSVE.IN

∑ ∏


Methods for SDP

n n

B(X) = − ln(λi(X)) = − ln λi(X) = − ln(det(X)) . j=1 j=1

Consider:

BSDP (µ) : minimize C • X − µ ln(det(X))

s.t. Ai • X = bi , i = 1, . . . , m,

X 0.

Let fµ(X) denote the objective function of BSDP (µ). Then:

−∇fµ(X) = C − µX−1



WWW.BSSVE.IN

∑


Methods for SDP

BSDP (µ) : minimize C • X − µ ln(det(X))

s.t. Ai • X = bi , i = 1, . . . , m,

X 0.

∇fµ(X) = C − µX−1

Karush-Kuhn-Tucker conditions for BSDP (µ) are:

Ai • X = bi , i = 1, . . . , m, X 0, m C − µX−1 = yiAi.

i=1



WWW.BSSVE.IN

∑


Methods for SDP

Ai • X = bi , i = 1, . . . ,m, X 0,

m C − µX−1 = yiAi. i=1

Define S = µX−1 ,

which implies XS = µI ,



WWW.BSSVE.IN

∑


Methods for SDP

and rewrite KKT conditions as:

Ai • X = bi , i = 1, . . . ,m, X 0 m yiAi + S = C i=1

XS = µI.



WWW.BSSVE.IN

∑

∑ ∑ ∑ ∑


Methods for SDP


XS = µI.

If (X, y, S) is a solution of this system, then X is feasible for SDP , (y, S) is feasible for SDD, and the resulting duality gap is

n n n n

S • X = SijXij = (SX)jj = (µI)jj = nµ. i=1 j=1 j=1 j=1



WWW.BSSVE.IN

∑


Methods for SDP


XS = µI.

If (X, y, S) is a solution of this system, then X is feasible for SDP , (y, S) is feasible for SDD, the duality gap is

S • X = nµ.



WWW.BSSVE.IN

Interior-point Methods for SDP

Primal and Dual SDP

This suggests that we try solving BSDP (µ) for a variety of values of µ as µ → 0.

Interior-point methods for SDP are very similar to those for linear optimization, in that they use Newton’s method to solve the KKT system as µ → 0.



WWW.BSSVE.IN

Website for SDP

A good website for semidefinite programming is:

http://www-user.tu-chemnitz.de/ helmberg/semidef.html.



WWW.BSSVE.IN

Differential Evolution:a stochastic nonlinear optimization algorithm by Storn and Price, 1996

Presented by David CraftSeptember 15, 2003

This presentation is based on: Storn, Rainer, and Kenneth Price. Differential Evolution – A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces. Journal of Global Optimization 11, 1997, pp. 341-359.www.bsscommunitycollege.in www.bssnewgeneration.in www.bsslifeskillscollege.in


WWW.BSSVE.IN

The highlights ofDifferential Evolution (DE)

A population of solution vectors are successively updated by addition, subtraction, and component swapping, until the population converges, hopefully to the optimum.

No derivatives are used.

Very few parameters to set.

A simple and apparently very reliable method.

www.bsscommunitycollege.in www.bssnewgeneration.in www.bsslifeskillscollege.in


WWW.BSSVE.IN

DE: the algorithm

Start with NP randomly chosen solution vectors.

For each i in (1, …NP), form a ‘mutant vector’

vi = xr1+F.(xr2-xr3)

Where r1, r2, and r3 are three mutually distinct randomly drawn indices from (1, …NP), and also distinct from i, and 0<F<=2.



WWW.BSSVE.IN

DE: forming the mutant vectorvi = xr1+F.(xr2-xr3)

. xr1. xr3

. xr2

. .

.

.

Solution space

. vi

.xi



WWW.BSSVE.IN

DE: From old points to mutants

..

.

.. .

.

.

.

.

.. .

..

..



WWW.BSSVE.IN

DE: Crossover xi and vi to form the trial vector

.

.

Possible trial vectors

original x

mutant v



WWW.BSSVE.IN

DE: Crossover xi and vi to form the trial vector ui

xi = (xi1, xi2, xi3, xi4, xi5)vi = (vi1, vi2, vi3, vi4, vi5)ui = (__, __, __, __, __)For each component of vector, draw a random number in U[0,1]. Call this randj. Let 0<=CR<1 be a cutoff. If randj<=CR, uij= vij, else uij= xij.

To ensure at least some crossover, one component of uiis selected at random to be from vi .



WWW.BSSVE.IN

DE: Crossover xi and vi to form the trial vector ui

xi = (xi1, xi2, xi3, xi4, xi5)vi = (vi1, vi2, vi3, vi4, vi5)

So, for example, maybe we have

ui = (vi1, xi2, xi3, xi4, vi5)

Index 1 randomlyselected as definite crossover

rand5<=CR, so it crossed over too



WWW.BSSVE.IN

DE: Selection

If the objective value COST(ui) is lower than COST(xi), then ui replaces xi in the next generation. Otherwise, we keep xi.



WWW.BSSVE.IN

Numerical verificationMuch of the paper is devoted to trying the algorithm on many functions, and comparing the algorithm to representative algorithms of other classes. These classes are:

•Annealing algorithms•Evolutionary algorithms•The method of stochastic differential equations

Summary of tests: DE is the only algorithm which consistently found the optimal solution, and often with fewer function evaluations than the other methods.



WWW.BSSVE.IN

Numerical verification: exampleThe fifth De Jong function, or “Shekel’s Foxholes”

(See equation 10 on page 348 of the Differential Evolutionpaper.)



WWW.BSSVE.IN

The rest of the talk…

• Why is DE good?

• Variations of DE.

• How do we deal with constraints?

• An example from electricity load management.



WWW.BSSVE.IN

Why is DE good?

•Simple vector subtraction to generate ‘random’ direction.•More variation in population (because solution has not converged yet) leads to more varied search over solution space.

•∆ = (xr2-xr3) [discuss: size and direction]

•Annealing versus “self-annealing”.



WWW.BSSVE.IN

Variations of DE

xr1 : instead of random, could use best

(xr2-xr3) : instead of single difference, could use more vectors, for more variation.

for example (xr2-xr3+xr4-xr5)

Crossover: something besides bernoulli trials…



WWW.BSSVE.IN

Dealing with constraints

• Penalty methods for ‘difficult’ constraints.• Simple projection back to feasible set for

l<=x<=u type constraints. • Or, random value U[l,u] (when, why?)



WWW.BSSVE.IN

Example: Appliance Job SchedulingHourly electricity prices(cents/kWh):

Power requirements for 3 different jobs (kW):

Start time constraints.www.bsscommunitycollege.in www.bssnewgeneration.in www.bsslifeskillscollege.in


WWW.BSSVE.IN

Example: Appliance Job SchedulingObjective: find start times for each job which minimize cost.

Cost includes a charge on the maximum power used throughout the day. This couples the problems!

1min ( ) ( )

. . 1,...,

Ji ii

i i i

t x D x

s t a x u i J=

+

≤ ≤ =∑

where

( ) ( ) ( , )i i

i

x l

i i i ixt x p t e t x dt

+= ∫ Cost of job i

started at time xi

[0, ]( ) max ( , )t T i iD x r e t x∈= ⋅ ∑ Demand chargewww.bsscommunitycollege.in www.bssnewgeneration.in www.bsslifeskillscollege.in


WWW.BSSVE.IN

Convergence for different F

Other settings: CR=0.3, NP=6www.bsscommunitycollege.in www.bssnewgeneration.in www.bsslifeskillscollege.in


WWW.BSSVE.IN

Appliance Job Scheduling: Solution

Solution

Total energy profile

Electricity price over time



WWW.BSSVE.IN

Wrap-up

•DE is widely used, easy to implement, extensions and variations available, but no convergence proofs.

•More information:DE homepage: practical advice (e.g. start with NP=10*D and CR=0.9, F=0.8), source codes, etc.http://www.icsi.berkeley.edu/~storn/code.html

DE bibliography, 1995-2002. Almost entirely DE applications.http://www.lut.fi/~jlampine/debiblio.htm



WWW.BSSVE.IN

http://www.icsi.berkeley.edu/~storn/code.html

http://www.lut.fi/~jlampine/debiblio.htm

I. Integer programming part of Clarkson-paper

II. Incremental Linear Programming, Section 9.10.1 in Randomized Algorithms-book

presented by Jan De MotSeptember 29, 2003

1/23

This presentation is based on: Clarkson, Kenneth L. Las Vegas Algorithms for Linear and Integer Programming When the Dimensionis Small. Journal of the ACM 42(2), March 1995, pp. 488-499. Preliminary version in Proceedings of the 29th Annual IEEESymposium on Foundations of Computer Science, 1988.

and Chapter 9 of: Motwani, Rajeev, and Prabhakar Raghavan. Randomized Algorithms. Cambridge, UK: Cambridge UniversityPress, 1995.



WWW.BSSVE.IN

Outline

Part I: Integer Linear Programming (ILP)• Previous work• Algorithm for solving Integer Linear Programs [Clarkson 1995]

based on the mixed algorithm for LP (Susan)– Concept– Running Time Analysis

Part II: Incremental Linear Programming• Concept• SeideLP [Seidel 1991]• BasisLP [Sharir and Welzl 1992]

2/23www.bsscommunitycollege.in www.bssnewgeneration.in www.bsslifeskillscollege.in


WWW.BSSVE.IN

Part I: Integer Linear Programming



WWW.BSSVE.IN

Previous Work

• [Lenstra 1983] showed how to solve an ILP in polynomial time when the numbers of variables is fixed.

• Subsequent improvements (e.g. by [Frank and Tardos 1987]) show that the fasted deterministic algorithm requires operations on -bit numbers.

• Running time of new ILP algorithm:This is substantially faster than Lenstra’s for



WWW.BSSVE.IN

ILP Problem

• Find the optimum of:

where and



WWW.BSSVE.IN

Notation and Preliminaries

• Let:– denote the set of constraints defined by and – denote the optimal solution of the ILP defined on

(not the corresponding LP relaxation).• Assume:

– Bounded solution by adding to a new set of constraints :

where and where we use a result by [Schrijver 1986]: if an ILP has finite solution, then every coordinate of that optimum has size no more than where is the facet complexity of

– Unique solution by choosing the lexicographically largest point achieving the optimum value.



WWW.BSSVE.IN

ILP Algorithm: Concept

• First it is established that an optimum is determined by a smallset ([Bell 1977] and [Scarf 1977]):Lemma: There is a set with and with

• ILP algorithms are variations on the LP algorithms, with sample sizes using rather than and using Lenstra’s algorithm in the base case.

• Here, we convert the mixed algorithm for LPs to a mixed algorithm for ILPs, establishing the right sample sizes and criteria for successful iterations in both the recursive and iterative part of the mixed algorithm.



WWW.BSSVE.IN

ILP Algorithm: Details

• Lemma 2, related to the LP recursive algorithm, needs to be redone due to the fact that is not unique.

• Reminder: why do we need lemma 2?We want to make sure the set of violated constraints does not become too big.

• Lemma 2 (ILP version): Let and let be a random subset of size with Let be the set of constraints violated by Then with probability

• Other necessary lemma’s remain valid or can be adapted easily, yielding the following essential parameters for the ILP mixed algorithm:– Recursive part: use Lenstra’s algorithm for

and require for a successful iteration.– Iterative part: with a corresponding bound

of



WWW.BSSVE.IN

ILP Algorithm: Proof of Lemma 2 (ILP version)

• Proof. Lemma 2 (ILP version): With probability

• Assume is empty. For not empty: similar proof.Let and let denote the number of constraints in violated by We know that for some with

We want to find such that the probability that is less then This probability is bounded above by:

which is no more than:



WWW.BSSVE.IN

ILP Algorithm: Proof of Lemma 2 (cont’d)

which is again no more than:

and using elementary bounds, this quantity is less thanfor



WWW.BSSVE.IN

ILP Algorithm: Running Time

• We have the following theorem:The ILP algorithm requires expected

row operations on -bit vectors, and

expected operations on -bit numbers, as where the constant factors do not depend on or



WWW.BSSVE.IN

Part II: Incremental Linear Programming



WWW.BSSVE.IN

Incremental LP

• Randomized incremental algorithms for LP• Concept:

– add constraints in random order,– after adding each constraint, determine the optimum of the

constraints added so far.• Two algorithms will be discussed:

– SeideLP– BasisLP



WWW.BSSVE.IN

Algorithm SeideLP

Input: A set of constraints Output: The optimum of the LP defined by

0. if output 1. Pick a random constraint

Recursively find2.1. if does not violate output to be the optimum 2.2. else project all the constraints of onto and recursively solve this new linear programming problem;



WWW.BSSVE.IN

SeideLP: Running Time

• Let denote an upper bound on the expected running time for a problem with constraints in dimensions.

• Then:

– First term: cost of recursively solving the LP defined by the constraints

– Second term: checking whether violates – Third term (with probability ): cost of projecting + recursively

solving smaller LP.• Theorem: There is a constant such that the recurrence

satisfies the solution



WWW.BSSVE.IN

SeideLP: Further Discussion

• In Step 2.2. we completely discard any information obtained from the solution of the LP

• From the above figure, it follows we must consider all constraints in

• But: Can we use to “jump-start” the recursive call in step 2.2.?

• RESULT: Algorithm BasisLP



WWW.BSSVE.IN

Algorithm BasisLP

Input:Output: A basis for

0. If output1. Pick a random constraint

BasisLP( );2.1. if does not violate output2.2. else output BasisLP( Basis( ));

Basis returns a basis for a set of or fewer constraints.



WWW.BSSVE.IN

BasisLP: Why does it work?

• Each invocation of Basis occurs when the violation test in 2.1. fails (i.e. does violate ).

• What is the probability that we fail a violation test?– Let– Remember: – Pr( violates the optimum of )– This probability decreases further if contains some of the

constraints of – This was indeed the motivation for modifying SeideLP to BasisLP.



WWW.BSSVE.IN

BasisLP: Running Time

• Notation:– Given , we call

enforcing in if

– Let denote minus the number of constraints that are enforcing in is called the hidden dimension of

• Lemma 1: If is enforcing in then (i) and (ii) is extreme in all such that

• So, the probability that a violation occurs can be bounded by

• We establish that the decreases by at least 1 at each recursive call in step 2.2. It turns out is likely to decrease much faster.

• Theorem: The expected running time of BasisLP is



WWW.BSSVE.IN

BasisLP: Analysis Details

• Proof of Lemma 1. If is enforcing in then– (i)

We have which can not be true if were a subset of

– (ii) is extreme in all such thatAssume the contrary:

a contradiction.



WWW.BSSVE.IN

BasisLP: Analysis Details (Cont’d)• Lemma 2: Let and let be an

extreme constraint in Let be a basis of Then:(i) Any constraint that is enforcing in is also enforcing in (ii) is enforcing in (iii)

Proof:– (i)

then:– (ii) Since is extreme in– (iii) Follows readily.

• So, the numerator of decreases by at least 1 at each execution.



WWW.BSSVE.IN

BasisLP: Analysis Details (Cont’d)

• Show that this decrease is likely to be faster.• Given and a random we bound the

probability that violates If it does, check the probability distribution of the resulting hidden dimension.

• Lemma 3: Let be the extreme constraints of that are not in numbered so that

Then, for all and for is enforcing inBasis (proof: immediate from lemma 2.)

• In other words: when then all of will be enforcing and the arguments of the recursive call will have hidden dimension

• Observation: since any is equally likely to be is uniformly distributed on the integers in and the resulting hidden dimension is uniformly distributed on the integers in



WWW.BSSVE.IN

BasisLP: Analysis Details (Cont’d)

• Let denote the maximum expected number of violationtests for a call to BasisLP with arguments where and

• We get:

• This yields: and consequently the expected running time of BasisLP is

Augmenting the analysis with Clarkson’s sampling technique improves the running time of the mixed algorithm to



WWW.BSSVE.IN

Las Vegas Algorithms for Linear (and Integer) Programmingwhen the Dimension is Small

Kenneth L. Clarkson presented by Susan Martonosi

September 29, 2003

This presentation is based on: Clarkson, Kenneth L. Las Vegas Algorithms for Linear and Integer Programming When the Dimensionis Small. Journal of the ACM 42(2), March 1995, pp. 488-499. Preliminary version in Proceedings of the 29th Annual IEEESymposium on Foundations of Computer Science, 1988.



WWW.BSSVE.IN

Outline

• Applications of the algorithm

• Previous work

• Assumptions and notation

• Algorithm 1: “Recurrent Algorithm”

• Algorithm 2: “Iterative Algorithm”

• Algorithm 3: “Mixed Algorithm”

• Contribution of this paper to the field

1



WWW.BSSVE.IN

Applications of the Algorithms

Algorithms give a bound that is “good” in n (number of constraints), but “bad” in d (dimension). So we require the problem to have a small dimension.

• Chebyshev approximation: fitting a function by a rational function where both the numerator and denominator have relatively small degree. The dimension is the sum of the degrees of the numerator and denominator.

• Linear separability: separating two sets of points in d-dimensional space by a hyperplane

• Smallest enclosing circle problem: find a circle of smallest radius that encloses points in d dimensional space

2



WWW.BSSVE.IN

Previous work

• Megiddo: Deterministic algorithm for LP in O(22dn)

• Clarkson; Dyer: O(3d2 n)

• Dyer and Frieze: Randomized algo. with expected time no better than O(d3dn)

• This paper’s “mixed” algo.: Expected time √ O(d2 n) + (log n)O(d)d/2+O(1) + O(d4 n log n) as n → ∞

3



WWW.BSSVE.IN

Assumptions

• Minimize x1 subject to Ax ≤ b

• The polyhedron F(A, b) is non-empty and bounded and 0 ∈ F(A, b)

• The minimum we seek occurs at a unique point, which is a vertex of F(A, b) – If a problem is bounded and has multiple optimal solutions with optimal value

∗ x1, choose the one with the minimum Euclidean norm ∗ min{‖x‖2|x ∈ F(A, b), x1 = x1}

• Each vertex of F(A, b) is defined by d or fewer constraints

4



WWW.BSSVE.IN

Notation

Let:

• H denote the set of constraints defined by A and b

• O(S) be the optimal value of the objective function for the LP defined on S ⊆ H

• “Each vertex of F (A, b) is defined by d or fewer constraints” implies that ∃B(H) ⊂ H of size d or less such that O(B(H)) = O(H). We call this subset B(H) the basis of H. All other constraints in H\B(H) are redundant.

• a constraint h ∈ H be called extreme if O(H\h) < O(H) (these are the constraints in B(H)).

5



WWW.BSSVE.IN

Algorithm 1: Recursive

• Try to eliminate redundant constraints • Once our problem has a small number of constraints (n ≤ 9d2), then use Simplex

to solve it. • Build up a smaller set of constraints that eventually include all of the extreme

constraints and a small number of redundant constraints √ – Choose r = d n unchosen constraints of H\S at random – Recursively solve the problem on the subset of constraints, R ∪ S – Determine which remaining constraints (V ) are violated by this optimal solution √ – Add V to S if it’s not too big (|V | ≤ 2 n). – Otherwise, if V is too big, then pick r new constraints We stop once V is empty: we’ve found a set S ∪R such that no other constraints in H are violated by its optimal solution. This optimal solution x is thus optimal for the original problem.

6



WWW.BSSVE.IN

Recursive Algorithm

Input: A set of constraints H. Output: The optimum B(H)

1. S ← ∅; Cd ← 9d2

2. If n ≤ Cd return Simplex(H) 2.1 else repeat: √

choose R ⊂ H\S at random, with |R| = r = d nx ←Recursive(R ∪ S)V ← {h ∈ H| vertex defined by x violates h}√if |V | ≤ 2 n then S ← S ∪ Vuntil V = ∅

2.2 return x

7



WWW.BSSVE.IN

Recursive Algorithm: Proof Roadmap

Questions:

• How do we know that S doesn’t get too large before it has all extreme constraints?

• How do we know we will find a set of violated constraints V that’s not too big (i.e. the loop terminates quickly)?

Roadmap: Lemma 1. If the set V is nonempty, then it contains a constraint of B(H).Lemma 2. Let S ⊆ H and let R ⊆ H\S be a random subset of size r, with |H\S| =m. Let V ⊂ H be the set of constraints violated by O(R ∪ S). Then the expected sizeof V is no more than d(m−r+1) .r−d

And we’ll use this to show the following Lemma:

8



WWW.BSSVE.IN

Lemma 3. The probability that any given execution of the loop body is ”successful” √(|V | ≤ 2 n for this recursive version of the algorithm) is at least 1/2, and so on average, two executions or less are required to obtain a successful one

This will leave us with a running time

√ T (n, d) ≤ 2dT (3d n, d) + O(d2 n) for n > 9d2 .

9



WWW.BSSVE.IN

Recursive Algorithm: Proof of Lemma 1

Proof. Lemma 1: When V is nonempty, it contains a constraint of B(H).

Suppose on the contrary that V �= ∅ contains no constraints of B(H).

L Let a point x � y if (x1, ‖x‖2) ≤ (y1, ‖y‖2) (x is better than y).

∗Let x ∗ (T ) be the optimal solution over a set of constraints T . Then x (R ∪S) satisfies ∗all the constraints of B(H) (it is feasible), and thus x ∗ (R ∪ S) � x (B(H)).

∗ ∗However, since R ∪ S ⊂ H, we know that x ∗ (R ∪ S) � x (H) = x (B(H)). Thus, ∗ x (R ∪ S) has the same obj. fcn value and norm as x ∗ (B(H)). By the uniqueness of

∗ ∗this point, x ∗ (R ∪ S) = x (B(H)) = x (H), and V = ∅. Contradiction! So, every time V is added to S, at least one extreme constraint of H is added (so we’ll do this at most d times).

10



WWW.BSSVE.IN


Proof. Lemma 2: The expected size of V is no more than d(m−r+1) . r−d

First assume problem nondegenerate.

Let CH = {x ∗ (T ∪ S)|T ⊆ H\S}, subset of optima.

Let CR = {x ∗ (T ∪ S)|T ⊆ R}

The call Recursive(R ∪ S) returns an element x ∗ (R ∪ S):

• an element of CH

• unique element of CR satisfying every constraint in R.

11



WWW.BSSVE.IN

�


Choose x ∈ CH and let vx = number of constraints in H violated by x. � ∗ � E[|V |] = E[ x∈CH

vxI(x = x (R ∪ S))] = vxPxx∈CH

where

∗ 1 if x = x ∗ (R ∪ S)I(x = x (R ∪ S)) =

0 otherwise

and Px = P (x = x ∗ (R ∪ S))

How to find Px?

12



WWW.BSSVE.IN

� �


∗Let N = number of subsets of H\S of size r s.t. x ∗ (subset) = x (R ∪ S).

m NThen N = Px and Px = m . r ( r )

∗To find N , note that x ∗ (subset) ∈ CH and x (subset) = x ∗ (R ∪ S) only if

∗ • x (subset) ∈ CR as well ∗ • x (subset) satisfies all constraints of R

∗Therefore, N = No. of subsets of H\S of size r s.t. x ∗ (subset) ∈ CR and x (subset) satisfies all constraints of R.

13



WWW.BSSVE.IN


∗For some such subset of H\S of size r and such that x ∗ (subset) = x (R ∪ S), let T ∗be the minimal set of constraints such that x ∗ (subset) = x (T ∪ S).

∗ • x (subset) ∈ CR implies T ⊆ R

• nondegeneracy implies T is unique and |T | ≤ d

Let ix = |T |. ∗ ∗In order to have x ∗ (T ∪ S) = x (R ∪ S) (and thus x ∗ (subset) = x (R ∪ S)), when

constructing our subset we must choose:

• the ix constraints of T ⊆ R

• r − ix constraints from H\S\T \V

14



WWW.BSSVE.IN

� � r−ix ) m−r+1(m−vx−ix(m−vx−ix

Therefore, N = m−vx−ix and Px = (m ≤ r−d r−ix−1 ) (mr−ix r ) r )

(m−vx−ix � r−ix−1 ) ≤ dm−r+1E[|V |] ≤ m−r+1 x∈CH

vx (mr−dr ) r−d

(where the summand is E[No. of x ∈ CR violating exactly one constraint in R] ≤ d)

For the degenerate case, we can perturb the vector b by adding (ε, ε2, ..., εn) and show that the bound on |V | holds for this perturbed problem, and that the perturbed problem has at least as many violated constraints as the original degenerate problem.

15



WWW.BSSVE.IN


Proof. Lemma 3: P(successful execution) ≥ 1/2; E[Executions til 1st success] ≤ 2.

√Here, P(unsuccessful execution) = P (|V | > 2 n)

√ √ 2E[|V |] ≤ 2dm−r+1 = 2n−d n+1 (since r = d

√ n) ≤ 2 n√

r−d n−1

√So, P(unsuccessful execution)= P (|V | > 2 n) ≤ P (|V | > 2E[|V |]) ≤ 1/2, by the Markov Inequality.

P(successful execution) ≥ 1/2, and the expected number of loops until our first successful execution is less than 2.

16



WWW.BSSVE.IN

Recursive Algorithm: Running Time

As long as n > 9d2 ,

• Have at most d+1 augmentations to S (succesful iterations), with expected 2 tries until success √ √ • With each success, S grows by at most 2 n, since |V | ≤ 2 n

• After each success, we run the Recursive algorithm on a problem of size |S ∪R| ≤√ √ √ 2d n + d n = 3d n

• After each recursive call, we check for violated constraints, which takes O(nd) each of at most d + 1 times

√ T (n, d) ≤ 2(d + 1)T (3d n, d) + O(d2 n), for n > 9d2

17



WWW.BSSVE.IN

Algorithm 2: Iterative

• Doesn’t call itself, calls Simplex directly each time

• Associates weight wh to each constraint which determines the probability with which it is selected

• Each time a constraint is violated, its weight is doubled

• Don’t add V to a set S; rather reselect R (of size 9d2) over and over until it includes the set B(H)

18



WWW.BSSVE.IN

Algorithm 2: Iterative

Input: A set of constraints H. Output: The optimum B(H)

1. ∀h ∈ H, wh ← 1; Cd = 9d2

2. If n ≤ Cd, return Simplex(H) 2.1 else repeat:

choose R ⊂ H at random, with |R| = r = Cd

x ←Simplex(R)V ← {h ∈ H| vertex defined by x violates h}

w(H)if w(V ) ≤ 29d−1 then for h ∈ V , wh ← 2wh

until V = ∅

2.2 return x

19



WWW.BSSVE.IN

Iterative Algorithm: Analysis

• Lemma 1: “If the set V is nonempty, then it contains a constraint of B(H)” still holds (proof as above with S = ∅).

• Lemma 2: “Let S ⊆ H and let R ⊆ H\S be a random subset of size r, with |H\S| = m. Let V ⊂ H be the set of constraints violated by O(R ∪ S). Then the expected size of V is no more than d(m−r+1) ” still holds with the following r−d changes. Consider each weight-doubling as the creation of multinodes. So “size” of a set is actually its weight. So we have S = ∅, and thus |H\S| = m = w(H).

+1 ≤ w(H)This gives us E[w(V )] ≤ d(w(H)−9d2

9d−19d2−d

• Lemma 3: If we define a “successful iteration” to be w(V ) ≤ 2w(H) , then Lemma 3 9d−1 holds, and the probability that any given execution of the loop body is ”successful” is at least 1/2, and so on average, two executions or less are required to obtain a successful one.

20



WWW.BSSVE.IN

� �

� �

′

Iterative Algorithm: Running Time

The Iterative Algorithm runs in O(d2 n log n)+(d log n)O(d)d/2+O(1) expected time, as n → ∞, where the constant factors do not depend on d.

First start by showing expected number of loop iterations = O(d log n)

• By Lemma 3.1, at least one extreme constraint h ∈ B(H) is doubled during a successful iteration

• Let d′ = |B(H)|. After kd′ successful executions w(B(H)) = 2nh ,h∈B(H)

where nh is the number of times h entered V and thus h∈B(H) nh ≥ kd′

• h∈B(H) wh ≥ h∈B(H) 2k = d′2k

2• When members of V are doubled, increase in w(H) = w(V ) ≤ 9d−1 , so after kd′

successful iterations, we have ′ 2kd

9d2 −1)

kd 9d−1w(H) ≤ n(1 + ≤ ne

21



WWW.BSSVE.IN

� �

• V sure to be empty when w(B(H)) > w(H) (i.e. P (Choose B(H)) > 1). This gives us:

k > ln(n/d′) , or kd′ = O(d log n) successful iterations = O(d log n) iterations.ln 2− 2d

9d−1

Within a loop:

• Can select a sample R in O(n) time [Vitter ’84] • Determining violated constraints, V , is O(dn)

2Cd• Simplex algorithm takes dO(1) time per vertex, times �d/2� vertices [?]. Using

Stirling’s approximation, this gives us O(d)d/2+O(1) for Simplex

Total running time:

O(d log n) ∗ [O(dn) + O(d)d/2+O(1)] = O(d2 n log n) + (d log n)O(d)d/2+O(1)

22



WWW.BSSVE.IN

Algorithm 3: Mixed

• Follow the Recursive Algorithm, but rather than calling itself, call the Iterative Algorithm instead √ • Runtime of Recursive: T (n, d) ≤ 2(d + 1)T (3d n, d) + O(d2 n), for n > 9d2 � √ • In place of T (3d (n), substitute in runtime of Iterative algorithm on 3d n constraints √ • Runtime of Mixed Algorithm: O(d2 n)+(d2 log n)O(d)d/2+O(1)+O(d4 n log n)

23



WWW.BSSVE.IN

Contributions of this paper to the field

• Leading term in dependence on n is O(d2 n), an improvement over O(d3dn)

• Algorithm can also be applied to integer programming (Jan’s talk) • Algorithm was later applied as overlying algorithm to “incremental” algorithms

(Jan’s talk) to give a sub-exponential bound for linear programming (rather than using Simplex once n ≤ 9d2, use an incremental algorithm)

24



WWW.BSSVE.IN

Polytopes, their diameter, and randomized simplex

Presentation by: Dan StratilaOperations Research Center

Session 4: October 6, 2003

Based primarily on:Gil Kalai. A subexponential randomized simplex algorithm (extended abstract).

In STOC. 1992. [Kal92a].

and on:Gil Kalai. Linear programming, the simplex algorithm and simple polytopes.

Math. Programming (Ser. B), 1997. [Kal97].



WWW.BSSVE.IN

Structure of the talk

1. Introduction to polytopes, linear programming, and the simplex method.

2. A few facts about polytopes.

3. Choosing the next pivot. Main result in this talk.

4. Subexponential randomized simplex algorithms.

5. Duality between two subexponential simplex algorithms.

6. The Hirsch conjecture, and applying randomized simplex to it.

7. Improving diameter results using an oracle for choosing pivots.

1



WWW.BSSVE.IN

Polytopes and polyhedra

A polyhedron P ⊆ Rd is the intersection of finitely many halfspaces, or in matrix notation P := {x ∈ Rd : Ax ≤ b}, where A ∈ Rn×d and b ∈ Rn . A polytope is a bounded polyhedron.

Dimension of polyhedron P is dim(P ) := dim(aff(P )), where aff(P ) is the affine hull of all points in P .

A polyhedron P ∈ Rd with dim(P ) = k is often called a k-polyhedron. If d = k, P called full-dimensional. (Most of the time we assume full-dimensional d-polyhedra, not concerned much about the surrounding space.)

An inequality ax ≤ β, where a ∈ Rd and β ∈ R, is called valid if ax ≤ β for all x ∈ P .

2



WWW.BSSVE.IN

Vertices, edges, ..., facets

A face F of P is the intersection of P with a valid inequality ax ≤ β, i.e. F := {x ∈ P : ax = β}. Faces of dimension d − 1 are called facets, 1 ... edges, and 0 ... vertices. Verticesare points, ⇔ basic feasible solutions (algebraic), or extreme points (linear cost).

Since 0x ≤ 0 is valid, P is a d-dimensional face of P . 0x ≤ 1 is valid too, so ∅ is a face of P , and we define its dimension to be −1.

Some vertices are connected by edges, so we can define a graph G = (V (G), E(G)), where V (G) = {v : v ∈ vert(P )} and E(G) = {(v, w) ∈V (G)2 : ∃ edge E of P s.t. v ∈ E, w ∈ E}. For unbounded polyhedra often a ∞ node is introduced in V (G), and we add graph arcs (v, ∞) whenever v ∈ E where E is an unbounded edge of P .

3



WWW.BSSVE.IN

Example of a 3-polytope

1

2

5

3

7

86

4

F

1

2

3

4

5

6

7

8

F

Figure 1: A 3-polytope (left) and its graph (right). Four vertices, three edges, and facet F are shown in corresponding colors.

4



WWW.BSSVE.IN

Linear programming and the simplex method

A linear programming problem max{cx : Ax ≤ b} is the problem of maximizing a linear function over a polyhedron.

• If problem bounded (cost of feas. sol. finite), optimum can be achieved at some vertex v.

If problem unbounded, can find edge E of P = {x ∈ Rd : Ax ≤ b} s.t. cx is• unbounded on the edge.

• If problem bounded, vertex v is optimal ⇔ cv ≥ cw for all w adjacent to v (for all (v, w) ∈ E(G)).

Geometrically, the simplex method starts at a vertex (b.f.s.) and moves from one vertex to another along a cost-increasing edge (pivots) until it reaches an optimal vertex (optimal b.f.s).

5



WWW.BSSVE.IN

Vertices as intersections of facets

Any polytope can be represented by its facets P = {x ∈ Rd : Ax ≤ b}, or by its vertices P = conv({v : v ∈ vert(P )}). If vertices are given, then LP is trivial—just select the best one. Most of the time, facets are given. Number of vert. exponential in number of facets makes generating all vertices from the facets impractical.

Represent a vertex v as intersection of d facets. Any vertex is situated at the intersection of at least d facets; any non-empty intersection of d facets yields a vertex.

dWhen situated at a vertex v given by ∩i=1Fi, easy to find all adjacent vertices. Remove each facet Fi, and intersect with all other facets not in {F1, . . . , Fd}. Except when...

6



WWW.BSSVE.IN

Degeneracy and simple polytopes

When a vertex is at the intersection of > d facets, procedure above may leave us at the same vertex. Worse, sometimes need such changes before can move away from a vertex in cost-increasing direction.

This is (geometric) degeneracy. In standard form degenerate vertices yield degenerate b.f. solutions. Other “degenerate” b.f. solutions may appear because of redundant constraints.

If all vertices of P belong to at most d facets (⇒ exactly d), P is called simple. Simple polytopes correspond to non-degenerate LPs, and have many properties [Zie95, Kal97].

We restrict ourselves to simple polytopes. Ok for two reasons: 1) any LP can be suitably perturbed to become non-degenerate; 2) perturbation can be made implicit in the algorithms.

7



WWW.BSSVE.IN

A few facts about polytopes

Disclaimer: results not used or related to subexponential simplex pivot rules (mainresult in this talk).

The f -vector: fk(P ) :=# of k-faces of P .

Degrees: let degc(v) w.r.t. to some objective function c be the # of neighboringvertices w with cw < cv.

The h-vector: hk,c(P ) :=# of vertices of degree k w.r.t. objective c in P .

Note: there is always one vertex of degree d, and one of degree 0.

Property: hk,c(P ) = hk(P ), independent of c.

8



WWW.BSSVE.IN

hk,c(P ) = hk(P ), proof (1/2)

Proof. Count p := |{(F, v) : F is a k-face of P , v is max. on F }, in two ways.

Pick facets. Because c in general position ⇒ v unique for each F , hence p = fk(P ).

On the other hand, pick a vertex v, and assume degc(v) = r. Let T = {(v, w) : cv > cw}, by definition T = r.| | For simple polytopes, each vertex v has d adjacent edges, and any k of them define a k-face F that includes v.

�T

�So, # of k-facets that contain v as local maximum is

|k| =

�r�

.k

9



WWW.BSSVE.IN

hk,c(P ) = hk(P ), proof (2/2)

Summing over all v ∈ vert(P ), we obtain fk(P ) = �d

�r�

. r=k hr,c(P ) k

Equations linearly independent in hr,c. This completely determines hr,c(P ) in terms of fk(P ). But fk(P ) independent of c, so same true for hr(P ).

10



WWW.BSSVE.IN

The Euler Formula and Dehn-Sommerville Relations

r=k(−1)r−kfr(P ) .We can expess hk(P ) =�d

�r�

k

We know that h0(P ) = hd(P ) = 1, hence f0(P )−f1(P )+ · · ·+(−1)dfd(P ) = 1, d+ (−1)d−1fd−1(P ) = 1− (−1) .or f0(P )− f1(P ) + · · ·

In 3 dimensions, V − E + F = 2.

Back to hk,c(P ), note that if degc(v) = k then deg−c(v) = d − k.

Because of independence of c, we obtain the Dehn-Sommerville Relations: hk(P ) = hd−k(P ).

11



WWW.BSSVE.IN

Cyclic polytopes and the upper bound theorem

A cyclic d-polytope with n vertices is defined by n scalars t1, . . . , tn asconv({(ti, t2

i , . . . , tid) : i = 1, d}). Can use other curves too.

All cyclic d-polytopes with n vertices have same structure, denote by C(d, n).

The polar C∗(d, n) := {x ∈ (Rd)∗ : xv ≤ 1,∀v ∈ C(d, n)} is a simple polytope.

Property: C(d, n) has the maximum number of k-facets for any polytope withn vertices.

The polar C∗(d, n) has the maximum number of k-facets for any polytope with n facets (the face lattice).

Exact expression for fk−1 elaborate, but a simple one is fk−1 =�min{d,k}

�d− i

�

hi(P ). For more interesting details, see [Zie95]. i=0 k − i

12



WWW.BSSVE.IN

Abstract objective functions and the combinatorial structure

An abstract objective function assigns a value to every vertex of a simple polytope P , s.t. every non-empty face F of P has a unique local maximum vertex.

AOFs are gen. of linear objective functions. Most results here apply.

The combinatorial structure of a polytope is all the information on facet inclusion,e.g. all vertices, all edges and the vertices they are composed of, all 3-facets and their composition, etc.

Lemma: Given graph G(P ) of simple polytope P , connected subgraph H = (V (H), E(H)) with k vertices defines a k-face if and only if ∃ AOF s.t. all vertices in V (H) come before all vertices in V (G(P )) \ V (H).

Property: The combinatorial structure of any simple polytope is determined by its graph.

13



WWW.BSSVE.IN

Main result in this talk—context

In the simplex algorithm we often make choices on which vertex to move to next. Criteria for choosing the next vertex are called pivot rules.

In the early days, “believed” simple rules guarantee a polynomial number of vertices in path. Klee and Minty [KM72] have shown exponential behaviour.

After that, not known even if LP can be solved in polynomial time at all, until [Kha79]. But still,

• Finding a pivot rule (deterministic or randomized) that would yield a polynomial number of vertex changes—open since simplex introduced.

For some f(n), exponential: f(n) ∈ Ω(kn), k > 1. Polynomial: f(n) ∈ O(nk) for some fixed k ≥ 1. Subexponential f(n k) for any fixed k ≥ 1 and f(n) �∈ Ω(kn) for any fixed k > 1.

) �∈ O(n

14



WWW.BSSVE.IN

Main result in this talk

Shortly before a different technique in [SW92], shorty aftewards a subexponential analysis for it in [MSW96].

• The first randomized pivot rule that yields subexponential expected path length (presented from [Kal92a, Kal97]).

Expectation over internal random choices of algorithm; applicable to all LP instances.

Immediate application to diameter of polytopes and the Hirsch conjecture (more about diameters and the Hirsch conjecture later).

15



WWW.BSSVE.IN

Algorithm 1

Simplest randomization (Dantzig, others): next vertex random with equal prob. among neighboring cost-increasing vertices. Hard to analyize in general; Gartner, Henk and Ziegler show quadratic lower bounds on Klee-Minty cubes.

Reminder: P Given P = {x ∈ Rd : Ax ≤ b}, so in LP terms: d = # of variables, n = # of constraints. Also given c ∈ Rd .

A1-1 (parameter r, start vertex v):

1. Find vertices on r facets F1, F2, . . . , Fr s.t. ∀Fi, cv < max{cx : x ∈ Fi}. 2. Choose a facet Fk at random from F1, F2, . . . , Fr with equal probability. 3. Solve max{cx : x ∈ Fk} recursively. Let the optimum vertex be w. 4. Finish solving the problem from w recursively.

How is this a simplex algorithm?

16



WWW.BSSVE.IN

A1-1: Implementation of step 1

For step 1, easy to find the first d facets. For the rest r − d facets, let k := d, z := v and proceed as follows:

1. Solve an LP from z with only the k facets recursively. Let result be z.

2. If z feasible for original problem, optimum found, A1-1 terminates.

3. Otherwise, first edge E on path that leaves P gives new facet F . Let z be the point in E ∩ F . If r facets, stop; otherwise go to step 1.

Up to now we are tracing a path along the vertices of the original problem.

17



WWW.BSSVE.IN

A1-1: Implementation of steps 2 and 3

First, note that when solving max{cx : x ∈ Fk}, tracing a path along vertices of Fk. This is also a path along vertices of P , since we are working with Fk in its dimension.

If k = r or k = r − 1, then last vertex ∈ Fk, can continue our path in step 3.

But if k < r − 1, then backtracking from the last vertex found when discovered. Not “honest” simplex.

Easy to fix. Since facet Fk is chosen uniformly among facets F1, . . . , Fr, this can be done by choosing uniformly among Fi1, . . . , Fir , where i1, . . . , ir order in which facets encountered by step 1.

So, generate random k before step 1, and stop once reached k-th facet (Kalai also offers another workaround).

18



WWW.BSSVE.IN

A1-1: Implementation of step 4

A facet F is active w.r.t. a vertex v if ∃w ∈ vert(F ) s.t. cw > cv.

Apply algorithm recursively from w using only those facets which are active. At most n− 1 such facets (Fk from step 3 cannot be active).

Complexity analysis

Let f1(P, c) := E[# of pivots when solving max{cx : x ∈ P} by A1]. Let f1(d, n) := max

�f1({x ∈ Rd : Ax ≤ b}, c) : A ∈ Rn×d, c ∈ Rd, b ∈ Rn

�.

First part of analysis: probabilistic reasoning to obtain a recurrence relation on f1(d, n).

Second part: solving the recurrence (using generating functions).

19



WWW.BSSVE.IN

A1-1: Analysis of step 1

It takes f1(d, i) to solve an LP in d variables with i facets using A1-1. So step 1 takes at most

�ri=1 f1(d, i).

In step 2, note that there is at least one vertex in the path for each random number generated.

In step 3, the expected complexity is f1(d− 1, n− 1).

After step 3, we only need to consider the active facets w.r.t. w. How many? Assume facets F1, . . . , Fr are ordered according to their top vertex. Then selecting facet i ⇒ at most n− i− 1 active facets w.r.t. w.

So with probability 1 we will have n− i− 1 active facets, for i = 1, 2, . . . , r. r

Rec.: f1(d, n) = 2�r

f1(d, i)+f1(d− 1, n− 1)+ 1 �n−1 �l f1(d, n− i).i=d r n−r−1 i=d

20



WWW.BSSVE.IN

(The real) Algorithm 1

Before analyzing the recurrence, we improve A1-1.

1A1 (parameter c > 2, start vertex v):

1. Starting from v, find vertices on r active facets F1, F2, . . . , Fr. 2. Choose a facet Fk at random from F1, F2, . . . , Fr with equal probability. 3. Solve max{cx : x ∈ Fk} recursively. Let the optimum vertex be w. 4. Let l := |{F : F active w.r.t. If l > (1 − c)n then let v := w and go to w}|.

step 1; otherwise, finish solving recursively from w.

Let r := max �

n 2 , d

�. What is probability of not returning to step 1? If r = n

easily geometric with ratio = P ( no return) = P (l < (1 − c)n) = 1 − c.

In general, analysis more complicated.

21



WWW.BSSVE.IN

�

A1: the recurrence

2r� 1 1

r�f1(d, n) ≤ f1(d, i) + f (d − 1, n − 1) + f1(d, n − i)i.

1 − c 1 − c (1 − c)n i=d i=�cn�

(1) �

d + logb n1Taking b = 1−c, we get a bound of f1(d, n) ≤ bd(6n)logb n logb n

.

1Taking c = 1 − , we obtain √d

f1(d, n) ≤ n 16√

d . (2)

22



WWW.BSSVE.IN

Algorithm 2

Delete step 4, repeat steps 1–3 until step 1 detects an optimal vertex. Equivalent to setting c := 1 in 1.

A2 (start vertex v):1. Starting from v, find vertices on r active facets F1, F2, . . . , Fr. If unable to

find r distinct active facets ⇒ opt. vertex found. 2. Choose a facet Fk at random from F1, F2, . . . , Fr with equal probability. 3. Solve max{cx : x ∈ Fk} recursively. Let the optimum vertex be w. 4. Delete inactive facets, set v := w, and go to step 1.

Recurrence is:

f2(d, n) ≤ f2(d− 1, n− 1) + n/2�

g(d, i) +2

n/2�

n i=d i=1

g(d, n− i). (3)

23



WWW.BSSVE.IN

A2: the bounds

d log dBy solving the recurrences, we get f2(d, Kd) ≤ 2C

√Kd , and f2(d, n) ≤ n

C�

.

When co-dimension (m := n − d) is small the following bound is very useful: 2C√

m log d .

Next: the interesting A3.

24



WWW.BSSVE.IN

Algorithm 3

A3 (start vertex v):

1. From d facets containing v, select a facet F0 at random, with equal probability. 2. Apply A3 to F0 recursively, and let w be the optimum. 3. Set v := w and go to step 1.

Simple! This algorithm is the dual of the algorithm discovered by Sharir and Welzl [SW92] (more about this later).

For now, note that in a simple polytope, there can be at most 1 non-active facet adjacent to any vertex v, unless v is optimal.

25



WWW.BSSVE.IN

A3: the recursion

1First, if all facets active, with probability d the chosen facet yields n − i active facets at step 3, for i = 1, . . . , d.

1Second, if one facet inactive, with probability d−1 the chosen facet yields n − i facets at step 3, for i = 1, . . . , d− 1.

Second alternative is worse, so we factor it in and obtain recursion f3(d, n) ≤f(d− 1, n− 1) + 1 �d−1

f(d, n− i).i=1d−1

This yields bound f3(d, n) ≤ eC√

n log d . A4, which we do not present now, gives eC√

d log n, better, like A2.

26



WWW.BSSVE.IN

Subexponential behaviour

2 4 6 8 10 12 14

500

1000

1500

2000

2500

200 400 600 800 1000 1200

5·108

1·109

1.5·109

2·109

Figure 2: Asymptotic behaviour of the exponential 2d , the polynomial d3 and the subexponential 2

√d .

27



WWW.BSSVE.IN

The duality of A4 and the Sharir-Welzl algorithm

Following [Gol95], we show what the Sharir-Welzl algorithm (Algorithm B) [MSW96] does to the polytope of the dual LP. Reminder: algorithm B was called BasisLP in the second part of Session 3.

Unlike before, we’ll use lots of traditional LP terminology. Let H be a set of constraints, and B a set of constraints that define a basis.

B (set of constraints H, basis C with C ⊆ H):

1. Begin at C. 2. Choose random constraint h ∈ H \ C 3. Solve LP recursively with constraints H \ {h}, from C. Let result be B.4. If B violates h, then form new basis C � := basis(B, h); otherwise optimum

found. 5. Let C := C � and go to step 1.

28



WWW.BSSVE.IN

The dual problems

Primal (we run B on this problem):

max{cx : Ax ≤ b}. (4)

Dual (we see what B does to this problem):

min{yb : y ≥ 0, yA = c}. (5)

We will imagine the dual problem in the yA = c space, so only the inequality constraints y ≥ 0 define facets; yA = c is simply an affine transformation of the space.

29



WWW.BSSVE.IN

The correspondence, up to step 3

C := initial feasible basis � C := initial feasible vertex. (Not true.)

Choosing random h ∈ H \ C � choosing random facet yh ≥ 0 that contains C:

• We know, by complementary slackness, that un-tight constraints in the primal correspond to 0-level variable components in the dual. Only the yi ≥ 0 constraints define facets in the dual polytope. • Active constraint at a vertex C defines a facet that constains C.•

Solve recursively the LP with constrains H \h starting from C � solve recursively LP on facet yh = 0 starting from C.

If we remove a constraint in the primal, this is the same as requiring yh = 0 • in the dual.

30



WWW.BSSVE.IN

The correspondence, step 4

In both cases we obtain a new basis (� vertex) B.

If B violates h � B not optimal: infeasible primal solutions correspond to suboptimal dual solutions.

C � := basis(B, h) is a pivot operation � C � := move away along an unique edge from yh = 0. But, there is no “move along an edge” in A3!

Slight adjustment to A3, in order to achieve perfect duality: in step 1, pick a random facet among d active facets.

After we found optimum w on facet F0, this facet is not active (since w optimum on it). Moreover, this facet is defined by d − 1 of the edges at w (in a simple polytope, every d − 1 edges at a vertex define a facet, and conversely).

31



WWW.BSSVE.IN

The correspondence, steps 4 and 5

Hence only one remaining edge can be cost-improving ⇒ any simplex algorithm will take it. So, our algorithm takes it when it tries to find the d-th facet.

But this implies that the choice of d facets available to A3 is exactly the same as the choice of d facets available after moving along the unique edge.

In steps 5 and 1 this yields the same choice of un-tight constraints in the primal!

So, a variant of the Sharir-Welzl algorithm B when followed on the dual polytope is exactly the same as the (slightly modified) Kalai algorithm A3.

32



WWW.BSSVE.IN

The Hirsch conjecture

The diameter δ(G(P )) of a polytope is the diameter of its graph, i.e. the longest shortest path between any pair (v, w) of vertices. Denote by δ( �G(P )) the longest shortest path in the cost-function directed graph of P .

Let Δ(d, n) := max{δ(G(P )) : P is a d-polytope with n facets }. Let H(d, n) := max{δ(G(P )) : P is a d-polytope with n facets, c is any cost function }. Clearly, the simplex algorithm cannot guarantee a better performance on P than δ(G(P )). Moreover, Δ(d, n) ≤ H(d, n).

Conjecture (Hirsch, [Dan63]): Δ(P ) ≤ n − d.

False for unbounded polyhedra [KW67]. Lower bound of Δ(d, n) ≥ n− d+ �d/5�. Still best lower bound!

33



WWW.BSSVE.IN

Status of Hirsch conjecture for polytopes

Still open!

Exponential bound Δ(d, n) ≤ n2d−3 [Larman, 1970].

Until recently (w.r.t. 1992) no sub-exponential bound known.

Bounds of n2 log d+3 and nlog d+1 in [Kal92b, KK92] respectively.

How randomized pivot rules affect the Hirsch conjecture? A randomized simplexalgorithm gives only hope that a deterministic algorithm with the same complexitymay be devised.

But, because E[...] is over choices, at least one of these choices (even if we don’tknow it), yields a path of length less that E[...].

34



WWW.BSSVE.IN

A friendly oracle

So all our bounds on the number of simplex pivots, immediately apply to H(d, n) (since simplex takes only monotone paths) and hence to Δ(d, n).

In algorithms A1–A4 we spend at most O(d2n) for each pivot, and generate at most 1 random number per pivot.

What if we allow much more time per pivot? Result will still apply to the Hirsch conjecture. Do not want algorithms such as “construct the graph, find the shortest path, then parse it”, since analysis is equivalent to Hirsch conjecture.

But, can still make use of a more powerful oracle that makes choices at each pivot step. Works from within Algorithm 4.

35



WWW.BSSVE.IN

Algorithm 4

A4 (start vertex v):

1. From the active facets w.r.t v, select a facet F0 at random, with equal probability.

2. Find vertices recursively until reached F0, or until optimum found. 3. Solve recursively on F0. Let result be v. 4. Go to step 1.

As mentioned, bound of eC√

d log n .

Now instead of selecting F0 at random, order all facets in increasing order F1, . . . , Fn of max{cx : x ∈ Fi}. Select F0 s.t. max{x : x ∈ F0} above the median (i > �n/2� in the ordering).

Let f4(d, n) be the number of steps using the oracle.

36



WWW.BSSVE.IN

A4: the recursion

At most f4(d, n/2) pivots for step 2. Pivots from step 4 (counting everything that happens) is again f4(d, n/2). Clearly, step 3 takes f4(d − 1, n − 1). Hence:

f4(d, n) ≤ 2f4(d, �n/2�) + f4(d − 1, n − 1) + 1. (6)

To solve this, let φ(d, t) := 2tf4(d, 2t). Then, from (6), we obtain φ(d, t) ≤φ(d − 1, t) + φ(d, t − 1).

By simple combinatorial reasoning (counting all paths to the bottom), this yields�d + t

� �log n + d

�φ(d, t) ≤ . So f4(d, n) ≤ n

d log n .

�a + b

�

Finally, by combinatorics a

≤ ab (or ba), we obtain f4(d, n) ≤ nlog d+1 .

37



WWW.BSSVE.IN

How smart the oracle?

Not too smart:

1. Solve max{cx : x ∈ Fi} for each facet Fi using some polynomial LP algorithm.

2. Rank all values.

Only needs to be done once per instance. Hence bound nlog d+1 can be achievedby a polynomial-pivot-time deterministic simplex algorithm!

Not combinatorial; overkill.

38



WWW.BSSVE.IN

References

[Dan63] George B. Dantzig. Linear programming and extensions. Princeton University Press, Princeton, N.J., 1963.

[Gol95] Michael Goldwasser. A survey of linear programming in randomized subexponential time. ACM SIGACT News, 26(2):96–104, 1995.

[Kal92a] Gil Kalai. A subexponential randomized simplex algorithm (extended abstract). In Proceedings of the twenty-fourth annual ACM symposium on Theory of computing, pages 475–482. ACM Press, 1992.

[Kal92b] Gil Kalai. Upper bounds for the diameter and height of graphs of convex polyhedra. Discrete Comput. Geom., 8(4):363–372, 1992.

[Kal97] Gil Kalai. Linear programming, the simplex algorithm and simple polytopes. Math. Programming, 79(1-3, Ser. B):217–233, 1997.

[Kha79] L. G. Khachiyan. A polynomial algorithm in linear programming. Dokl. Akad. Nauk SSSR, 244(5):1093– 1096, 1979.

[KK92] Gil Kalai and Daniel J. Kleitman. A quasi-polynomial bound for the diameter of graphs of polyhedra. Bull. Amer. Math. Soc. (N.S.), 26(2):315–316, 1992.

[KM72] Victor Klee and George J. Minty. How good is the simplex algorithm? In Inequalities, III (Proc. Third Sympos., Univ. California, Los Angeles, Calif., 1969; dedicated to the memory of Theodore S. Motzkin), pages 159–175. Academic Press, New York, 1972.

[KW67] Victor Klee and David W. Walkup. The d-step conjecture for polyhedra of dimension d < 6. Acta Math., 117:53–78, 1967.

39



WWW.BSSVE.IN

[MSW96] J. Matousek, M. Sharir, and E. Welzl. A subexponential bound for linear programming. Algorithmica, 16(4-5):498–516, 1996.

[SW92] Micha Sharir and Emo Welzl. A combinatorial bound for linear programming and related problems. In STACS 92 (Cachan, 1992), volume 577 of Lecture Notes in Comput. Sci., pages 569–579. Springer, Berlin, 1992.

[Zie95] Gunter M. Ziegler. Lectures on polytopes, volume 152 of Graduate Texts in Mathematics. Springer-Verlag, New York, 1995.

40



WWW.BSSVE.IN

Minimization by Random Search Techniques by Solis and Wets

and

an Intro to Sampling Methods

Presenter: Michele Aghassi

October 27, 2003

This presentation is based on: Solis, F. J., and R. J-B. Wets. Minimization by Random Search Techniques. Mathematical

Operations Research 6, 1981, pp. 19-30.



WWW.BSSVE.IN

Recap of Past Sessions

• LP – Kalai (1992, 1997)

∗ use randomized pivot rules – Motwani and Raghavan (1995), Clarkson (1998, 1995)

∗ solve on a random subset of constraints, recursively – Dunagan and Vempala (2003): LP Feasibility (Ax ≥ 0, 0 �=0)

∗ Generate random vectors and test for feasibility ∗ If not, try moving in deterministic (w.r.t. random vector already selected) direction to achieve feasibility

• NLP – Storn and Price (1997): Unconstrained NLP

∗ Heuristic ∗ Select random subsets of solution population vectors ∗ Perform addition, subtraction, component swapping and test for obj func improvement

1



WWW.BSSVE.IN

Motivation

What about provably convergent algorithms for constrained NLPs?

• Random search techniques first proposed in the 1950s • pre-1981 proofs of convergence were highly specific and involved

• Solis and Wets, 1981: Can we give more general sufficient conditions for convergence, unifying the past results in the literature?

• Solis and Wets paper interesting more from a unifying theoretical standpoint • Computational results of the paper relatively unimpressive

2



WWW.BSSVE.IN

Outline

• Part I: Solis and Wets paper – Motivation for using random search – Appropriate goals of random search algorithms – Conceptual Algorithm encompassing several concrete examples – Sufficient conditions for global search convergence, and theorem – Local search methods and sufficient conditions for convergence, and theorem – Defining stopping criteria – Some computational results

• Part II: Intro to Sampling Methods – Traditional Methods – Hit-and-run algorithm

3



WWW.BSSVE.IN

Why Use Random Search Techniques?

Let f : Rn → R, S ⊆ Rn .

(P) min f (x)

s.t. x ∈ S

• Function characteristics difficult to compute (e.g. gradients, etc.) • Function is “bumpy” • Need global minimum, but there are lots of local minima • Limited computer memory

4



WWW.BSSVE.IN

What is an Appropriate Goal?

• Problems – Global min may not exist – Finding min may require exhaustive examination (e.g. min occurs at point at which f singularly discontinuous)

• Response Definition 1. α is the Essential Infimum of f on S iff

α = inf {t | v(x ∈ S | f(x) < t) > 0} ,

where v denotes n-dimensional volume or Lebesgue measure. Optimality region for P is given by (

{x∈ S | f(x) < α+ ε} , α finite =Rε,M {x∈ S | f(x) < −M} , α = −∞,

for a given “big” M > 0

5



WWW.BSSVE.IN

What is Random Search?

Conceptual Algorithm:

1. Initialize: Find x0 ∈ S. Set k := 0

2. Generate ξk ∈ Rn (random) from distribution µk k+13. Set x = D(xk, ξk). Choose µk+1. Set k := k + 1. Go to step 1.

“ ˛ ” k ˛ 0 1 k−1

µk(A) = P x ∈ A ˛ x , x , . . . , x

This captures both

• Local search =⇒ supp(µk) is bounded and v(S ∩ supp(µk)) < v(S)

• Global search =⇒ supp(µk) is such that v(S ∩ supp(µk)) = v(S)

6



WWW.BSSVE.IN

Sufficient Conditions for Convergence

˘ (H1) D s.t. f(xk)

¯∞ nonincreasing

k=0

f(D(x, ξ)) ≤ f(x)

ξ ∈ S =⇒ f(D(x, ξ)) ≤ min {f(x), f(ξ)}

(H2) Zero probability of repeatedly missing any positive-volume subset of S.

∞ Y ∀A ⊆ S s.t. v(A) > 0, (1 − µk(A)) = 0

k=0

i.e. sampling strategy given by µk cannot consistently ignore a part of S with positive volume (Global search methods satisfy (H2))

7



WWW.BSSVE.IN

Example Satisfying (H1) and (H2), I

Due to Gaviano [2].

k k kD(x , ξ ) = (1 − λk)x + λkξ

k where h i k k k k

λk = arg min f ((1 − λ)x + λξ ) | (1 − λ)x + λξ ∈ S λ∈[0,1]

k µk unif on n-dim sphere with center x and r ≥ 2diam(S).

Why? ˘ • (H1) satisfied since f (xk) ¯∞

nonincreasing by construction k=0

• (H2) satisfied because sphere contains S

8



WWW.BSSVE.IN

Example Satisfying (H1) and (H2), II

Due to Baba et al. [1]. (

kξk, ξk ∈ S and f(ξk) < f(x )k kD(x , ξ ) = kx , o.w.

k µk ∼ N (x , I)

Why? ˘ • (H1) satisfied since f(xk) ¯∞

nonincreasing by construction k=0

k • (H2) satisfied because S contained in support of N (x , I)

9



WWW.BSSVE.IN

Global Search Convergence Theorem

k¯∞

Theorem 1. Suppose f measurable, S ⊆ Rn measurable, (H1), (H2), and xk=0

generated by the algorithm. Then “ ” k

lim P x ∈ Rε,M = 1 k→∞

k ∈ Rε,M =⇒ x� �Proof. By (H1), x � ∈ Rε,M , ∀� < k

“ ” k−1 Y ` ´kP x ∈ S\Rε,M ≤ 1 − µ�(Rε,M )

�=0

“ ” “ ” k−1 Y ´k kP x ∈ Rε,M = 1 − P x ∈ S\Rε,M ≥ 1 − 1 − µ�(Rε,M )

�=0

“ ” k−1 Y ´k1 ≥ lim P x ∈ Rε,M ≥ 1 − lim 1 − µ�(Rε,M ) = 1,

k→∞ k→∞ �=0

where last equality follows from (H2).

10



WWW.BSSVE.IN

Local Search Methods

• Easy to find examples for which the algorithm will get trapped at local minimum

• Drastic sufficient conditions ensure convergence to optimality region, but are very difficult to verify

For instance

(H3) ∀x0 ∈ S ˘ L0 = x ∈ S | f (x) ≤ f (x0)

¯ is compact and

∃γ > 0 and η ∈ (0, 1] (possibly depending on x0) s.t., ∀k and ∀x ∈ L0, `ˆ ˜ ˆ ˜´ µk D(x, ξ) ∈ Rε,M ∪ dist(D(x, ξ), Rε,M ) < dist(x, Rε,M ) − γ ≥ η.

Iff and S are “nice,” local search methods demonstrate better convergence behavior.

11



WWW.BSSVE.IN

Example Satisfying (H3), I

• int(S) �= ∅

• ∀α ∈ R, S ∩ {x | f(x) ≤ α} convex and compact Happens whenever f quasi-convex and either S compact or f has bounded level sets

• ξk chosen via uniform distribution on hypersphere with center xk and radius ρk 0 1 • ρk is a function of x , x , . . . , xk−1 and ξ1, . . . , ξk−1 such that ρ = infk ρk > 0

• (ξk, ξk ∈ Sk k

D(x , ξ ) = kx , o.w.

Proof. L0 compact convex since level sets are. Rε,M has nonempty interior since S does. ∴ can draw ball contained in interior of Rε,M .

v(region I) Now take γ = ρ and η = > 02 v(hypersphere with radius ρ)

12



WWW.BSSVE.IN

Example Satisfying (H3)

ρ/2

x

y

L

R

ρ

Mε,

0

k

k

ρ z

I II

v(region II) > v(region I) = η. v(hypersphere with radius ρk) v(hypersphere with radius ρ)

13



WWW.BSSVE.IN

��

�

�

Local Search Convergence Theorem, I

Theorem 2. Suppose f is a measurable function, S ⊆ Rn is a measurable, and (H1) ˘ k¯∞

and (H3) are satisfied. Let x be a sequence generated by the algorithm. Then, k=0 “ ”

klim P x ∈ Rε,M = 1. k→∞

Proof. Let x0 be the initial iterate used by the algorithm. By (H1), all future iterates in L0 ⊇ Rε,M . L0 is compact. Therefore ∃p ∈ Z s.t. γp > diam(L0). “ ” “ ” P x�+p ∈ Rε,M, x �∈ Rε,M

P x �+p ∈ Rε,M | x � = ` ∈ Rε,M P x� �∈ Rε,M “ ”

≥ P x �+p ∈ Rε,M, x �∈ Rε,M “ ≥ P x �∈ Rε,M, dist(x k, Rε,M ) ≤ γ(p − (k − �)),

k = �, . . . , � + p)

≥ ηp by repeated Bayes rule and (H3)

14



WWW.BSSVE.IN

Local Search Convergence Theorem, II

` xkp ´

Claim: P �∈ Rε,M ≤ (1 − ηp)k , ∀k ∈ {1, 2, . . . }

By induction “ ” ` (k = 1) P xp ∈ Rε,M

≥ P xp ∈ Rε,M, x 0 �∈ Rε,M ≥ ηp

“ ” “ ” “ ” ∈ Rε,M | x(k−1)p ∈ Rε,M P x(k−1)pxkp ∈ Rε,M = P xkp � � �(Genl k) P � ∈ Rε,M h “ ”i

x(k−1)p ∈ Rε,M 1 − ηp´k−1≤ 1 − P xkp ∈ Rε,M | �

≤ ` 1 − η

p´ ` 1 − η

p´k−1

“ ” “ ” ∴ P xkp+� ∈ Rε,M ≥ P xkp ∈ Rε,M ≥ 1 −

` 1 − η

p´k, � = 0, 1, . . . , p − 1

15



WWW.BSSVE.IN

Stopping Criteria

˘k¯∞ k• So far, we gave a conceptual method for generating x such that f (x ) → k=0

essential inf plus buffer • In practice, need stopping criterion

• Easy to give stopping criterion if have LB on µk(Rε,M ) (unrealistic) • How to do this without knowing a priori essential inf or Rε,M ? • Has been shown that even if S compact and convex and f ∈ C2, each step of alg leaves unsampled square region of nonzero measure, over which f can be redefined so that global min is in unsampled region

• “search for a good stopping criterion seems doomed to fail”

16



WWW.BSSVE.IN

′

Rates of Convergence

• Measured by distributional characteristics of number of iters or function evals required to reach essential inf (e.g. mean)

• Solis and Wets tested 3 versions of the conceptual alg (1 local search, 2 global search) on various problems (constrained and unconstrained)

• They report results only for

min x x x∈Rn

3with stopping criterion ‖xk‖ ≤ 10−

• Found that mean number of function evals required ∝ n.

17



WWW.BSSVE.IN

Conclusion and Summary of Part I

• Why use random search techniques? • How to handle pathological cases? (essential infimum, optimality region) • Conceptual Algorithm unifies past examples in the literature • Global and local search methods • Sufficient conditions for convergence and theorems • Issue of stopping criteria • Computational results

18



WWW.BSSVE.IN

Part II: Traditional Sampling Methods

• Transformation method – easier to generate Y than X, but well-behaved transformation between the two

• Acceptance-rejection method – Generate a RV and subject it to a test (based on a second RV) in order to determine acceptance

• Markov-regression – Generate random vector component-wise, using marginal distributions w.r.t. components generated already

Impractical because complexity increases rapidly with dimension.

19



WWW.BSSVE.IN

Part II: Approximate Sampling Methods

• Perform better computationally (efficient) • generates a sequence of points, whose limiting distribution is equal to target distribution

Hit-and-Run: Generate random point in S, a bounded open subset of Rd, according to some target distribution π.

1. Initialize: select starting point x0 ∈ S. n := 0. 2. Randomly generate direction θn in Rd, according to distribution ν (corresponds to randomly generating a point on a unit sphere).

n3. Randomly select step size from λn ∈ {λ | x + λθn ∈ S} according to distribution L(xn, θn)

n+1 n4. Set x := x + λnθn . n := n + 1. Repeat.

e.g. generate point according to uniform distribution on S: use all uniform distributions

20



WWW.BSSVE.IN

Further Reading

References

[1] Baba, N., T. Shoman, and Y. Sawaragi. “A Modified Convergence Theorem for a Random Optimization Algorithm,” Information Science, 13 (1977).

[2] Gaviano, M. “Some General Results on the Convergence of Random Search Algorithms in Minimization Problems.” In Towards Global Optimization, eds. L. Dixon and G. Szego. Amsterdam.

[3] Solis, Francisco J. and Roger J.B. Wets. “Minimization by Random Search Techniques,” Mathematics of Operations Research, 6: 19 - 30 (1981).

[4] H.E. Romeijn, Global Optimization by Random Walk Sampling Methods, Thesis Publishers, Amsterdam, 1992.

21



WWW.BSSVE.IN

Pure Adaptive Search in Global Optimizationby

Z. B. Zabinsky and R. L. Smith

Presented by Michael Yee

November 3, 2003

Outline

• Pure Random Search versus Pure Adaptive Search

• Relationship to Solis and Wetz (1981)

• Distribution of improvement in objective function value

• Performance bounds

Pure Adaptive Search [1]

Global Optimization Problem

• Problem (P):minx∈S

f(x)

where x ∈ Rn, S is convex, compact subset of Rn, and f continuous over S

• f satisfies Lipschitz condition, i.e., |f(x)− f(y)| ≤ kf‖x− y‖, ∀x, y ∈ S

• x∗ = arg minx∈S f(x)

• y∗ = f(x∗) = minx∈S f(x)

• y∗ = maxx∈S f(x)


Pure Random Search (PRS)

• Generate sequence of independent, uniformly distributed points

X1, X2, . . . ,

in the feasible region S. Denote their associated objective function values by

Y1 = f(X1), Y2 = f(X2), . . .

• When stopping criterion met, best point generated so far is taken asapproximation to true optimal solution


This presentation is based on: Zabinsky, Zelda B., and Robert L. Smith. Pure Adaptive Searchin Global Optimization. Mathematical Programming 55, 1992, pp. 323-338.



WWW.BSSVE.IN

Pure Adaptive Search (PAS)

Step 0. Set k = 0, and S0 = S

Step 1. Generate Xk+1 uniformly distributed in Sk, and set Wk+1 = f(Xk+1)

Step 2. If stopping criterion met, STOP. Otherwise, set

Sk+1 = {x : x ∈ S and f(x) < Wk+1},

Increment k, Goto Step 1.


Solis and Wetz

• Give sufficient conditions for convergence of random global search methods

• Experimental support for linear relation between function evaluations anddimension

• PAS satisfies H1 since objective function values are increasing

• PAS satisfies H2 since the optimal solution is always in the restricted feasibleregion


Importance of Strict Improvement

• What if consecutive points were allowed to have equal objective functionvalues?

• Let S be a unit hypersphere, with f(x) = 1 on S except for a depression on ahypersphere of radius ε, Sε, where f(x) drops to value 0 at the center of theε-ball Sε

• Then, P (random point is in Sε) = volume(Sε)/volume(S) = εn

• Thus, PAS could have expected number of iterations that is exponential indimension (if strict improvement were not enforced)


Some Notation

• Let p(y) = P (Yk ≤ y), for k = 1, 2, . . . and y∗ ≤ y ≤ y∗

• For PRS,p(y) = v(S(y))/v(S),

where S(y) = {x : x ∈ S and f(x) ≤ y} and v(·) is Lebesgue measure

• Note that for PAS,

P (Wk+1 ≤ y|Wk = z) = v(S(y))/v(S(z)) = p(y)/p(z),

for k = 1, 2, . . . and y∗ ≤ y ≤ z ≤ y∗




WWW.BSSVE.IN

Connection Between PAS and PRS

Definition. Epoch i is said to be a record of the sequence {Yk, k = 0, 1, 2, . . .} ifYi < min(Y0, Y1, . . . , Yi−1). The corresponding value Yi is called a record value.

Lemma 1. For the global optimization problem (P), the stochastic process{Wk, k = 0, 1, 2, . . .} ∼ {YR(k), k = 0, 1, 2, . . .}, where R(k) is the kth record ofthe sequence {Yk, k = 0, 1, 2, . . .}. In particular,

P (Wk ≤ y) = P (YR(k) ≤ y), for k = 0, 1, 2, . . . , and y∗ ≤ y ≤ y∗


Proof of Lemma 1

Proof. First, we show that the conditional distributions are equal.

P (YR(k+1) ≤ y|YR(k) = x) = P (YR(k)+1 ≤ y|YR(k) = x)+P (YR(k)+2 ≤ y, YR(k)+1 ≥ x|YR(k) = x) + · · ·

= P (YR(k)+1 ≤ y)+P (YR(k)+2 ≤ y)P (YR(k)+1 ≥ x) + · · ·

= P (Y1 ≤ y)∑∞

i=0 P (Y1 ≥ x)i

= P (Y1≤y)1−P (Y1≥x)

= v(S(y))/v(S(x))

= P (Wk+1 ≤ y|Wk = x).


Next, we use induction to show that the unconditional distributions are equal.By definition, R(0) = 0 and Y0 = W0 = y∗, thus YR(0) = W0.

For the base case k = 1,

P (YR(1) ≤ y) = P (YR(1) ≤ y|Y0 = y∗)

= P (W1 ≤ y|W0 = y∗)

= P (W1 ≤ y), for all y∗ ≤ y ≤ y∗

Thus, YR(1) ∼ W1.


For k > 1, suppose that YR(i) ∼ Wi for i = 1, 2, . . . , k. Then,

P (YR(k+1) ≤ y) = E[P (YR(k+1) ≤ y|YR(k))]

=∫ x

0P (YR(k+1) ≤ y|YR(k) = x) dFYR(k)

(x)

=∫ x

0P (Wk+1 ≤ y|Wk = x) dFWk

(x)

= E[P (Wk+1 ≤ y|Wk)]

= P (Wk ≤ y), for all y∗ ≤ y ≤ y∗

Thus, YR(k+1) ∼ Wk+1.

Finally, since the two sequences are equal in conditional and marginaldistribution, they are equal in joint distribution. 2




WWW.BSSVE.IN

Linear versus Exponential

Theorem 1. Let k and R(k) be respectively the number of PAS and PRSiterations needed to attain an objective function value of y or better, fory∗ ≤ y ≤ y∗. Then

R(k) = ek+o(k), with probability 1,

where limk→∞ o(k)/k = 0, with probability 1.

Proof. Use general fact about records that limk→∞ln R(k)

k = 1, with probability1... 2


Relative Improvement

Definition. Let Zk = (y∗− Yk)/(Yk − y∗) be the relative improvement obtainedby the kth iteration of PRS.

Then, the cumulative distribution function F of Zk is given by

F (z) = P (Zk ≤ z)

= P (Yk ≥ (y∗ + zy∗)/(1 + z))

={

0 if z < 0,1− p((y∗ + zy∗)/(1 + z)) if 0 ≤ z < ∞.

Note also that the random variables Zk are iid and nonnegative.


Relative Improvement Process

Lemma 2. Let Z1, Z2, . . . denote a sequence of iid nonnegative continuousrandom variables with density f and cdf F . Let M(z) denote the number ofrecord values (in the max sense) of {Zi, i = 1, 2, . . .} less than or equal to z.

Then {M(z), z ≥ 0} is a nonhomogeneous Poisson process with intensityfunction λ(z) = f(z)/(1− F (z)) and mean value function m(z) =

∫ z

0λ(z) ds.

Theorem 2. Let N(z) be the number of PAS iterations achieving a relativeimprovement at most z for z ≥ 0. Then {N(z), z ≥ 0} is a nonhomogeneousPoisson process with mean value function

m(z) = ln(1/p((y∗ + zy∗)/(1 + z))), for 0 ≤ z < ∞.


Distribution of Objective Function Values

Theorem 3. P (Wk ≤ y) =∑k−1

i=0p(y)(ln(1/p(y)))i

i!

Proof. The events {Wk < y} and {N((y∗− y)/(y− y∗)) < k} are equivalent, so

P (Wk ≤ y) = P (Wk < y) = P (N((y∗ − y)/(y − y∗)) < k),

and by previous theorem N(z) is a Poisson random variable with mean

m(z) = ln(1/p((y∗ + zy∗)/(1 + z))),

etc. 2




WWW.BSSVE.IN

Performance Bounds

Let N∗(y) be the number of iterations require by PAS to achieve a value of y orbetter. Then

N∗(y) = N((y∗ − y)/(y − y∗)) + 1

Corollary 1. The cumulative distribution of N∗(y) is given by

P (N∗(y) ≤ k) =k−1∑i=0

p(y)(ln(1/p(y)))i

i!,

withE[N∗(y)] = 1 + ln(1/p(y)), V ar(N∗(y)) = ln(1/p(y))


Bounds for Lipschitz Functions

Lemma 3. For global optimization problem (P) over a convex feasible region Sin n dimensions with diameter dS = max{‖w − v‖, w, v ∈ S} and Lipschitzconstant kf ,

p(y) ≥ ((y − y∗)/kfdS)n, for y∗ ≤ y ≤ y∗.

Theorem 4. For any global optimization problem (P) over a convex feasibleregion in n dimensions with diameter at most d and Lipschitz constant at mostk,

E[N∗(y)] ≤ 1 + [ln(kd/(y − y∗))]n

andV ar(N∗(y)) ≤ [ln(kd/(y − y∗))]n

for y∗ ≤ y ≤ y∗.


Conclusions

• Complexity of PRS is exponentially worse than that of PAS

• General performance bounds using theory from stochastic processes

• Specific performance bounds for Lipschitz functions : linear in dimension!

• But is this too good to be true?!




WWW.BSSVE.IN

This presentation is based on: Zabinsky, Zelda B., and Robert L. Smith. Pure Adaptive Search in Global

Optimization. Mathematical Programming 55, 1992, pp. 323-338.



WWW.BSSVE.IN



WWW.BSSVE.IN



WWW.BSSVE.IN



WWW.BSSVE.IN



WWW.BSSVE.IN

Documents

Semidefinite Programming (SDP) and the Goemans … · Semidefinite Programming (SDP) and the Goemans-Williamson MAXCUT Paper ... . Outline