The Semidefinite Programming - Fundamentals

Semidefinite Programming

The Semidefinite Programming - Fundamentals

K. Subramani1

1Lane Department of Computer Science and Electrical EngineeringWest Virginia University

April 14, 2014


Outline

Outline

1 Mathematical Programming Frameworks

2 Rudiments of Semidefinite Programming

3 Separating hyper-planes and vector

programs


Outline

Outline




programs


Outline

Outline




programs


Mathematical Programming Frameworks

The Linear Programming Problem

Feasibility version (Equality Form)

A ·x = b

x ≥ 0

Algebraic Interpretation

Can b be expressed as a positive linearcombination of the columns of A?

Geometric Interpretation

Does b fall in the cone determined by thecolumns of A?





A ·x = b

x ≥ 0









A ·x = b

x ≥ 0









A ·x = b

x ≥ 0







The Linear Programming Problem (contd.)

a2

ba1

Figure: Geometric Interpretation of Linear Programming

Fact

Linear programming is a one person game.




a2

ba1


Fact





a2

ba1


Fact




The Fundamental Duality Theorem

Farkas’ Lemma

Either ∃ x ∈ Rn+ A ·x = b or (mutually exclusively)

∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.

Certifying algorithm

A certifying algorithm can either produce x ∈ Rn+, such that A ·x = b , or y ∈ Rm such that

y ·A≥ 0 and y ·b < 0.


Either b falls in the cone determined by the columns of A, or there exists a y that forms an acuteangle with each of the columns of A while simultaneously forming an obtuse angle with b.

Note that in the second case the hyperplane perpendicular to y separates b from the columns ofA.




Farkas’ Lemma


∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.



y ·A≥ 0 and y ·b < 0.







Farkas’ Lemma


∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.


A certifying algorithm can either produce x ∈ Rn+, such that A ·x = b ,

or y ∈ Rm such thaty ·A≥ 0 and y ·b < 0.







Farkas’ Lemma


∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.



y ·A≥ 0 and y ·b < 0.







Farkas’ Lemma


∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.



y ·A≥ 0 and y ·b < 0.


Either b falls in the cone determined by the columns of A,

or there exists a y that forms an acuteangle with each of the columns of A while simultaneously forming an obtuse angle with b.





Farkas’ Lemma


∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.



y ·A≥ 0 and y ·b < 0.







Farkas’ Lemma


∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.



y ·A≥ 0 and y ·b < 0.






The Fundamental Duality Theorem (contd.)

a2

a2

b

y

Figure: Geometric Interpretation of Farkas’ Lemma




a2

a2

b

y





a2

a2

b

y




Algebraic Extensions of Linear Programming

Additional one person games

(a) Bilinear programming -

min z =n

∑i=1

ci · (xi · yi)

A ·x≤ b, C ·y≤ d

(b) Quadratic programming -

min z =n

∑i=1

ci · (xi)2 +

n

∑i=1

di · xi

A ·x ≤ b

(c) Polynomial programming - Replace the quadratic function with a polynomial function.






min z =n

∑i=1

ci · (xi · yi)

A ·x≤ b, C ·y≤ d


min z =n

∑i=1

ci · (xi)2 +

n

∑i=1

di · xi

A ·x ≤ b







min z =n

∑i=1

ci · (xi · yi)

A ·x≤ b, C ·y≤ d


min z =n

∑i=1

ci · (xi)2 +

n

∑i=1

di · xi

A ·x ≤ b







min z =n

∑i=1

ci · (xi · yi)

A ·x≤ b, C ·y≤ d


min z =n

∑i=1

ci · (xi)2 +

n

∑i=1

di · xi

A ·x ≤ b




Complexity of Quadratic Programming

Theorem

Quadratic Programming is NP-hard.

Proof.

1 Let φ = φ1 ∧φ2 . . .φm denote a 3SAT formula over the n variables {x1,x2, . . .xn}.2 Let C+

j denote the set of literals that appear in uncomplemented form in clause Cj .

Likewise, let C−j denote the set of literals that appear in complemented form in clause Cj .

3 Consider the following quadratic program:

z = min ∑ni=1 xi · (1− xi)

subject to ∑i∈C+j

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

4 It is not hard to see that φ is satisfiable if and only if z is zero.




Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.

1 Let φ = φ1 ∧φ2 . . .φm denote a 3SAT formula over the n variables {x1,x2, . . .xn}.

2 Let C+j denote the set of literals that appear in uncomplemented form in clause Cj .



z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)

subject to

∑i∈C+j

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i





Theorem


Proof.





z = min ∑ni=1 xi · (1− xi)


xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i




Quadratically Constrained Programs


A quadratically constrained program (QCP) is the problem of optimizing (minimizing ormaximizing) a quadratic function of integer valued variables, subject to quadratic constraints onthese variables.

If each monomial in the objective function, as well as in each of the constraints, is of degree 0(i.e., is a constant) or 2, then we will say that this is a strict QCP. If the program variables arediscrete, the program is said to be a quadratically constrained integer program.

Example

max ∑1≤i<j≤n yi · yj

subject to yi · yj = 1, ∀yi ,yj

yi ∈ Z, ∀yi







Example



yi ∈ Z, ∀yi






If each monomial in the objective function, as well as in each of the constraints, is of degree 0(i.e., is a constant) or 2, then we will say that this is a strict QCP.

If the program variables arediscrete, the program is said to be a quadratically constrained integer program.

Example



yi ∈ Z, ∀yi







Example



yi ∈ Z, ∀yi







Example



yi ∈ Z, ∀yi







Example



yi ∈ Z, ∀yi







Example



yi ∈ Z, ∀yi







Example



yi ∈ Z, ∀yi



Vector Programs

Definition

A vector program is defined over n vector variables in ℜn, say v1, ...,vn, and is the problem ofoptimizing (minimizing or maximizing) a linear function of the inner products vi ·vj, 1≤ i ≤ j ≤ n,subject to linear constraints on these inner products.

Note

1 A vector program can be thought of as being obtained from a linear program by replacingeach variable with an inner product of a pair of these vectors.

2 A strict QCP over n integer variables, defines a vector program over n vector variables inℜn as follows:Establish a correspondence between the n integer variables and the n vector variables andreplace each degree 2 term with the corresponding inner product.

3 For the strict QCP example above, the vector program is:

max ∑1≤i<j≤n vi ·vj

subject to vi ·vj = 1, ∀vi,vj

vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi



Vector Programs

Definition


Note


2 A strict QCP over n integer variables, defines a vector program over n vector variables inℜn as follows:

Establish a correspondence between the n integer variables and the n vector variables andreplace each degree 2 term with the corresponding inner product.




vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi



Vector Programs

Definition


Note






vi ∈ℜn, ∀vi


Rudiments of Semidefinite Programming

Semidefinite Matrix

Definition

An n×n real, symmetric matrix A is said to be positive semidefinite, written A�0, if one of thefollowing three equivalent conditions holds:

1 (∀x ∈ℜn) xT ·A ·x≥ 0.

2 All eigenvalues of A are non-negative real numbers.3 There exists an n×n real matrix W, such that A = WT ·W. (Cholesky factorization).



Semidefinite Matrix

Definition


1 (∀x ∈ℜn) xT ·A ·x≥ 0.




Semidefinite Matrix

Definition


1 (∀x ∈ℜn) xT ·A ·x≥ 0.




Semidefinite Matrix

Definition


1 (∀x ∈ℜn) xT ·A ·x≥ 0.




Semidefinite Matrix

Definition


1 (∀x ∈ℜn) xT ·A ·x≥ 0.

2 All eigenvalues of A are non-negative real numbers.

3 There exists an n×n real matrix W, such that A = WT ·W. (Cholesky factorization).



Semidefinite Matrix

Definition


1 (∀x ∈ℜn) xT ·A ·x≥ 0.

2 All eigenvalues of A are non-negative real numbers.3 There exists an n×n real matrix W, such that A = WT ·W.

(Cholesky factorization).



Semidefinite Matrix

Definition


1 (∀x ∈ℜn) xT ·A ·x≥ 0.




Observations on semidefiniteness

Note

1 Any real, symmetric matrix A can be decomposed as A = L ·D ·LT, where D is a diagonalmatrix with the diagonal entries denoting the eigenvalues of A. This decomposition can bedone in polynomial time.

2 A is positive semidefinite, if and only if all the entries of D are non-negative.3 The decomposition of A as A = WT ·W is not polynomial time, since the entries could be

irrational. However, the decomposition can be approximated to any desired degree byapproximating the square roots of the diagonal entries in D.

4 We will assume that A can in fact be decomposed as A = WT ·W in polynomial time; theinaccuracy will be absorbed into the approximation factor.

5 The sum of two positive semidefinite matrices is also positive semidefinite and so is anyconvex combination.




Note









Note

1 Any real, symmetric matrix A can be decomposed as A = L ·D ·LT, where D is a diagonalmatrix with the diagonal entries denoting the eigenvalues of A.

This decomposition can bedone in polynomial time.








Note









Note


2 A is positive semidefinite, if and only if all the entries of D are non-negative.

3 The decomposition of A as A = WT ·W is not polynomial time, since the entries could beirrational. However, the decomposition can be approximated to any desired degree byapproximating the square roots of the diagonal entries in D.






Note



irrational.

However, the decomposition can be approximated to any desired degree byapproximating the square roots of the diagonal entries in D.






Note









Note




4 We will assume that A can in fact be decomposed as A = WT ·W in polynomial time;

theinaccuracy will be absorbed into the approximation factor.





Note









Note








Frobenius Inner Product

Definition

Let C = {cij}k×l and let Y = {yij}k×l .

C•Y = c11 · y11 + c12 · y12 + · · ·cij · yij + ·+ ckl · ykl

=k

∑i=1

l

∑j=1

cij · yij

Note

1 C•Y is an inner product of C and Y.

2 C•Y is precisely trace(CT ·Y).3 C•Y is a linear function of variables yij with coefficients cij .




Definition



=k

∑i=1

l

∑j=1

cij · yij

Note






Definition



=k

∑i=1

l

∑j=1

cij · yij

Note






Definition


C•Y =

c11 · y11 + c12 · y12 + · · ·cij · yij + ·+ ckl · ykl

=k

∑i=1

l

∑j=1

cij · yij

Note






Definition



=k

∑i=1

l

∑j=1

cij · yij

Note






Definition



=k

∑i=1

l

∑j=1

cij · yij

Note






Definition



=k

∑i=1

l

∑j=1

cij · yij

Note






Definition



=k

∑i=1

l

∑j=1

cij · yij

Note






Definition



=k

∑i=1

l

∑j=1

cij · yij

Note


2 C•Y is precisely trace(CT ·Y).

3 C•Y is a linear function of variables yij with coefficients cij .




Definition



=k

∑i=1

l

∑j=1

cij · yij

Note






The Model

1 Let Mn denote the cone of symmetric n×n matrices. If A ∈Mn, we use A� 0 to denotethe fact that A is positive semidefinite.

2 Let C,D1,D2, . . . ,Dk ∈Mn and let d1,d2, . . .dk ∈ℜ. The semidefinite programmingproblem (denoted by S) is defined as follows:

max C•Y

subject to Di •Y = di , 1≤ i ≤ k

Y� 0,

Y ∈Mn




The Model



max C•Y


Y� 0,

Y ∈Mn




The Model

1 Let Mn denote the cone of symmetric n×n matrices.

If A ∈Mn, we use A� 0 to denotethe fact that A is positive semidefinite.


max C•Y


Y� 0,

Y ∈Mn




The Model



max C•Y


Y� 0,

Y ∈Mn




The Model


2 Let C,D1,D2, . . . ,Dk ∈Mn and let d1,d2, . . .dk ∈ℜ.

The semidefinite programmingproblem (denoted by S) is defined as follows:

max C•Y


Y� 0,

Y ∈Mn




The Model



max C•Y


Y� 0,

Y ∈Mn




The Model



max C•Y


Y� 0,

Y ∈Mn




The Model



max C•Y


Y� 0,

Y ∈Mn




The Model



max C•Y


Y� 0,

Y ∈Mn




The Model



max C•Y


Y� 0,

Y ∈Mn



Properties of semidefinite programs

Observation

1 Semidefinite programming generalizes linear programming. In the semidefinite program S,if the matrices C,D1,D2, . . . ,Dk are all diagonal, then S becomes a linear programmingproblem.

2 Inequality constraints can be added, without affecting the form of S.3 The set of feasible solutions of S forms a convex set (since the convex combination of

semidefinite matrices is semidefinite).

4 Let A ∈ℜn×n be an infeasible point. Let C ∈ℜn×n. A hyperplane C•Y≤ b is called aseparating hyperplane for A, if all feasible points satisfy it and point A does not satisfy it.

5 Most effective for maximization problems involving constraints between pairs of variables.6 Leads to reasonably good approximation ratios, although in most cases, not the best

possible.




Observation






possible.




Observation

1 Semidefinite programming generalizes linear programming.

In the semidefinite program S,if the matrices C,D1,D2, . . . ,Dk are all diagonal, then S becomes a linear programmingproblem.





possible.




Observation






possible.




Observation


2 Inequality constraints can be added, without affecting the form of S.

3 The set of feasible solutions of S forms a convex set (since the convex combination ofsemidefinite matrices is semidefinite).



possible.




Observation


2 Inequality constraints can be added, without affecting the form of S.3 The set of feasible solutions of S forms a convex set

(since the convex combination ofsemidefinite matrices is semidefinite).



possible.




Observation






possible.




Observation




4 Let A ∈ℜn×n be an infeasible point. Let C ∈ℜn×n.

A hyperplane C•Y≤ b is called aseparating hyperplane for A, if all feasible points satisfy it and point A does not satisfy it.


possible.




Observation






possible.




Observation





5 Most effective for maximization problems involving constraints between pairs of variables.

6 Leads to reasonably good approximation ratios, although in most cases, not the bestpossible.




Observation






possible.


Separating hyper-planes and vector programs

Separating hyperplane theorem

Theorem

Let S be a semidefinite programming problem, and let A be a point in ℜn×n. We can determine,in polynomial time, whether A is feasible for S and, if it is not, find a separating hyperplane.

Proof.

If A is feasible, then A should be symmetric, positive semidefinite, and satisfy all the linearconstraints.Clearly, the above checks can be carried out in polynomial time.

The separating hyperplane can be found as follows:

1 If A is not symmetric, then aij > aji , for some i, j . Then, yij ≤ yji is a separating hyperplane.(Violation of Y ∈Mn.)

2 If A is not positive semidefinite, then it has a negative eigenvalue, say λ . Let v be thecorresponding eigenvector. Now (v ·vT)•Y = vT ·Y ·v≥ 0 is a separating hyperplane.(Violation of Y� 0.)

3 If any of the linear constraints is violated, it directly yields a separating hyperplane.(Violation of Di •Y = di , for some i = 1,2, . . . ,k .)




Theorem

Let S be a semidefinite programming problem, and let A be a point in ℜn×n.

We can determine,in polynomial time, whether A is feasible for S and, if it is not, find a separating hyperplane.

Proof.









Theorem


Proof.









Theorem


Proof.









Theorem


Proof.

If A is feasible, then A should be

symmetric, positive semidefinite, and satisfy all the linearconstraints.Clearly, the above checks can be carried out in polynomial time.








Theorem


Proof.

If A is feasible, then A should be symmetric,

positive semidefinite, and satisfy all the linearconstraints.Clearly, the above checks can be carried out in polynomial time.








Theorem


Proof.

If A is feasible, then A should be symmetric, positive semidefinite,

and satisfy all the linearconstraints.Clearly, the above checks can be carried out in polynomial time.








Theorem


Proof.

If A is feasible, then A should be symmetric, positive semidefinite, and satisfy all the linearconstraints.

Clearly, the above checks can be carried out in polynomial time.








Theorem


Proof.









Theorem


Proof.









Theorem


Proof.



1 If A is not symmetric, then aij > aji , for some i, j .

Then, yij ≤ yji is a separating hyperplane.(Violation of Y ∈Mn.)






Theorem


Proof.



1 If A is not symmetric, then aij > aji , for some i, j . Then, yij ≤ yji is a separating hyperplane.

(Violation of Y ∈Mn.)






Theorem


Proof.









Theorem


Proof.




2 If A is not positive semidefinite, then it has a negative eigenvalue, say λ .

Let v be thecorresponding eigenvector. Now (v ·vT)•Y = vT ·Y ·v≥ 0 is a separating hyperplane.(Violation of Y� 0.)





Theorem


Proof.




2 If A is not positive semidefinite, then it has a negative eigenvalue, say λ . Let v be thecorresponding eigenvector.

Now (v ·vT)•Y = vT ·Y ·v≥ 0 is a separating hyperplane.(Violation of Y� 0.)





Theorem


Proof.




2 If A is not positive semidefinite, then it has a negative eigenvalue, say λ . Let v be thecorresponding eigenvector. Now (v ·vT)•Y = vT ·Y ·v≥ 0 is a separating hyperplane.

(Violation of Y� 0.)3 If any of the linear constraints is violated, it directly yields a separating hyperplane.

(Violation of Di •Y = di , for some i = 1,2, . . . ,k .)




Theorem


Proof.









Theorem


Proof.








Equivalence between vector programs and semidefinite programs

Lemma

Let V be a vector program on n n-dimensional vector variables v1,v2, . . .vn. Define thecorresponding semidefinite program, S, over n2 variables yij ,1≤ i, j ≤ n, as follows:

1 Replace each inner product vi ·vj occurring in V by the variable yij .

2 The objective function and constraints are now linear in the yij ’s.3 Additionally, require that matrix Y , whose (i, j)th entry is yij , be symmetric and positive

semidefinite.

Vector program V is equivalent to semidefinite program S.




Lemma




semidefinite.





Lemma

Let V be a vector program on n n-dimensional vector variables v1,v2, . . .vn.

Define thecorresponding semidefinite program, S, over n2 variables yij ,1≤ i, j ≤ n, as follows:



semidefinite.





Lemma




semidefinite.





Lemma




semidefinite.





Lemma



2 The objective function and constraints are now linear in the yij ’s.

3 Additionally, require that matrix Y , whose (i, j)th entry is yij , be symmetric and positivesemidefinite.





Lemma




semidefinite.





Lemma




semidefinite.




Proof of Equivalence

Proof.

1 Let a1,a2, . . .an denote a feasible solution to the vector program V .

2 Let W be the matrix whose columns are a1,a2, . . .an.3 A = WT ·W is a feasible solution to the SDP S, having the same objective function value.

4 Let A be a feasible solution to S. Since A is positive semidefinite, there must exist an n×nmatrix W, such that A = WT ·W.

5 Let a1,a2, . . .an denote the columns of W.6 These columns form a feasible solution to the vector program V , having the same objective

function value.




Proof.





function value.




Proof.





function value.




Proof.


2 Let W be the matrix whose columns are a1,a2, . . .an.

3 A = WT ·W is a feasible solution to the SDP S, having the same objective function value.



function value.




Proof.





function value.




Proof.



4 Let A be a feasible solution to S.

Since A is positive semidefinite, there must exist an n×nmatrix W, such that A = WT ·W.


function value.




Proof.





function value.




Proof.




5 Let a1,a2, . . .an denote the columns of W.

6 These columns form a feasible solution to the vector program V , having the same objectivefunction value.




Proof.





function value.

Documents

The Semidefinite Programming - Fundamentals