The Semidefinite Programming - Fundamentals

Semidefinite Programming

K. Subramani1

1Lane Department of Computer Science and Electrical EngineeringWest Virginia University

April 14, 2014

Outline

1 Mathematical Programming Frameworks

2 Rudiments of Semidefinite Programming

3 Separating hyper-planes and vector

programs

Outline

programs

Outline

programs

Mathematical Programming Frameworks

The Linear Programming Problem

Feasibility version (Equality Form)

A ·x = b

x ≥ 0

Algebraic Interpretation

Can b be expressed as a positive linearcombination of the columns of A?

Geometric Interpretation

Does b fall in the cone determined by thecolumns of A?

A ·x = b

x ≥ 0

A ·x = b

x ≥ 0

A ·x = b

x ≥ 0

The Linear Programming Problem (contd.)

Figure: Geometric Interpretation of Linear Programming

Linear programming is a one person game.

The Fundamental Duality Theorem

Farkas’ Lemma

Either ∃ x ∈ Rn+ A ·x = b or (mutually exclusively)

∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.

Certifying algorithm

A certifying algorithm can either produce x ∈ Rn+, such that A ·x = b , or y ∈ Rm such that

y ·A≥ 0 and y ·b < 0.

Either b falls in the cone determined by the columns of A, or there exists a y that forms an acuteangle with each of the columns of A while simultaneously forming an obtuse angle with b.

Note that in the second case the hyperplane perpendicular to y separates b from the columns ofA.

Farkas’ Lemma

∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.

y ·A≥ 0 and y ·b < 0.

Farkas’ Lemma

∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.

A certifying algorithm can either produce x ∈ Rn+, such that A ·x = b ,

or y ∈ Rm such thaty ·A≥ 0 and y ·b < 0.

Farkas’ Lemma

∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.

y ·A≥ 0 and y ·b < 0.

Farkas’ Lemma

∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.

y ·A≥ 0 and y ·b < 0.

Either b falls in the cone determined by the columns of A,

or there exists a y that forms an acuteangle with each of the columns of A while simultaneously forming an obtuse angle with b.

Farkas’ Lemma

∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.

y ·A≥ 0 and y ·b < 0.

Farkas’ Lemma

∃ y ∈ Rm y ·A≥ 0 and y ·b < 0.

y ·A≥ 0 and y ·b < 0.

The Fundamental Duality Theorem (contd.)

Figure: Geometric Interpretation of Farkas’ Lemma

Algebraic Extensions of Linear Programming

Additional one person games

(a) Bilinear programming -

min z =n

∑i=1

ci · (xi · yi)

A ·x≤ b, C ·y≤ d

(b) Quadratic programming -

min z =n

∑i=1

ci · (xi)2 +

∑i=1

di · xi

A ·x ≤ b

(c) Polynomial programming - Replace the quadratic function with a polynomial function.

min z =n

∑i=1

ci · (xi · yi)

A ·x≤ b, C ·y≤ d

min z =n

∑i=1

ci · (xi)2 +

∑i=1

di · xi

A ·x ≤ b

min z =n

∑i=1

ci · (xi · yi)

A ·x≤ b, C ·y≤ d

min z =n

∑i=1

ci · (xi)2 +

∑i=1

di · xi

A ·x ≤ b

min z =n

∑i=1

ci · (xi · yi)

A ·x≤ b, C ·y≤ d

min z =n

∑i=1

ci · (xi)2 +

∑i=1

di · xi

A ·x ≤ b

Complexity of Quadratic Programming

Theorem

Quadratic Programming is NP-hard.

Proof.

1 Let φ = φ1 ∧φ2 . . .φm denote a 3SAT formula over the n variables {x1,x2, . . .xn}.2 Let C+

j denote the set of literals that appear in uncomplemented form in clause Cj .

Likewise, let C−j denote the set of literals that appear in complemented form in clause Cj .

3 Consider the following quadratic program:

z = min ∑ni=1 xi · (1− xi)

subject to ∑i∈C+j

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

4 It is not hard to see that φ is satisfiable if and only if z is zero.

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

1 Let φ = φ1 ∧φ2 . . .φm denote a 3SAT formula over the n variables {x1,x2, . . .xn}.

2 Let C+j denote the set of literals that appear in uncomplemented form in clause Cj .

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

subject to

∑i∈C+j

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Theorem

Proof.

z = min ∑ni=1 xi · (1− xi)

xi +∑i∈C−j(1− xi)≥ 1, ∀Cj

0≤ xi ≤ 1, ∀i

Quadratically Constrained Programs

A quadratically constrained program (QCP) is the problem of optimizing (minimizing ormaximizing) a quadratic function of integer valued variables, subject to quadratic constraints onthese variables.

If each monomial in the objective function, as well as in each of the constraints, is of degree 0(i.e., is a constant) or 2, then we will say that this is a strict QCP. If the program variables arediscrete, the program is said to be a quadratically constrained integer program.

Example

max ∑1≤i<j≤n yi · yj

subject to yi · yj = 1, ∀yi ,yj

yi ∈ Z, ∀yi

Example

yi ∈ Z, ∀yi

If each monomial in the objective function, as well as in each of the constraints, is of degree 0(i.e., is a constant) or 2, then we will say that this is a strict QCP.

If the program variables arediscrete, the program is said to be a quadratically constrained integer program.

Example

yi ∈ Z, ∀yi

Example

yi ∈ Z, ∀yi

Example

yi ∈ Z, ∀yi

Example

yi ∈ Z, ∀yi

Example

yi ∈ Z, ∀yi

Example

yi ∈ Z, ∀yi

Vector Programs

Definition

A vector program is defined over n vector variables in ℜn, say v1, ...,vn, and is the problem ofoptimizing (minimizing or maximizing) a linear function of the inner products vi ·vj, 1≤ i ≤ j ≤ n,subject to linear constraints on these inner products.

1 A vector program can be thought of as being obtained from a linear program by replacingeach variable with an inner product of a pair of these vectors.

2 A strict QCP over n integer variables, defines a vector program over n vector variables inℜn as follows:Establish a correspondence between the n integer variables and the n vector variables andreplace each degree 2 term with the corresponding inner product.

3 For the strict QCP example above, the vector program is:

max ∑1≤i<j≤n vi ·vj

subject to vi ·vj = 1, ∀vi,vj

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Vector Programs

Definition

2 A strict QCP over n integer variables, defines a vector program over n vector variables inℜn as follows:

Establish a correspondence between the n integer variables and the n vector variables andreplace each degree 2 term with the corresponding inner product.

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Vector Programs

Definition

vi ∈ℜn, ∀vi

Rudiments of Semidefinite Programming

Semidefinite Matrix

Definition

An n×n real, symmetric matrix A is said to be positive semidefinite, written A�0, if one of thefollowing three equivalent conditions holds:

1 (∀x ∈ℜn) xT ·A ·x≥ 0.

2 All eigenvalues of A are non-negative real numbers.3 There exists an n×n real matrix W, such that A = WT ·W. (Cholesky factorization).

Semidefinite Matrix

Definition

1 (∀x ∈ℜn) xT ·A ·x≥ 0.

Semidefinite Matrix

Definition

1 (∀x ∈ℜn) xT ·A ·x≥ 0.

Semidefinite Matrix

Definition

1 (∀x ∈ℜn) xT ·A ·x≥ 0.

Semidefinite Matrix

Definition

1 (∀x ∈ℜn) xT ·A ·x≥ 0.

2 All eigenvalues of A are non-negative real numbers.

3 There exists an n×n real matrix W, such that A = WT ·W. (Cholesky factorization).

Semidefinite Matrix

Definition

1 (∀x ∈ℜn) xT ·A ·x≥ 0.

2 All eigenvalues of A are non-negative real numbers.3 There exists an n×n real matrix W, such that A = WT ·W.

(Cholesky factorization).

Semidefinite Matrix

Definition

1 (∀x ∈ℜn) xT ·A ·x≥ 0.

Observations on semidefiniteness

1 Any real, symmetric matrix A can be decomposed as A = L ·D ·LT, where D is a diagonalmatrix with the diagonal entries denoting the eigenvalues of A. This decomposition can bedone in polynomial time.

2 A is positive semidefinite, if and only if all the entries of D are non-negative.3 The decomposition of A as A = WT ·W is not polynomial time, since the entries could be

irrational. However, the decomposition can be approximated to any desired degree byapproximating the square roots of the diagonal entries in D.

4 We will assume that A can in fact be decomposed as A = WT ·W in polynomial time; theinaccuracy will be absorbed into the approximation factor.

5 The sum of two positive semidefinite matrices is also positive semidefinite and so is anyconvex combination.

1 Any real, symmetric matrix A can be decomposed as A = L ·D ·LT, where D is a diagonalmatrix with the diagonal entries denoting the eigenvalues of A.

This decomposition can bedone in polynomial time.

2 A is positive semidefinite, if and only if all the entries of D are non-negative.

3 The decomposition of A as A = WT ·W is not polynomial time, since the entries could beirrational. However, the decomposition can be approximated to any desired degree byapproximating the square roots of the diagonal entries in D.

irrational.

However, the decomposition can be approximated to any desired degree byapproximating the square roots of the diagonal entries in D.

4 We will assume that A can in fact be decomposed as A = WT ·W in polynomial time;

theinaccuracy will be absorbed into the approximation factor.

Frobenius Inner Product

Definition

Let C = {cij}k×l and let Y = {yij}k×l .

C•Y = c11 · y11 + c12 · y12 + · · ·cij · yij + ·+ ckl · ykl

∑i=1

∑j=1

cij · yij

1 C•Y is an inner product of C and Y.

2 C•Y is precisely trace(CT ·Y).3 C•Y is a linear function of variables yij with coefficients cij .

Definition

∑i=1

∑j=1

cij · yij

Definition

∑i=1

∑j=1

cij · yij

Definition

C•Y =

c11 · y11 + c12 · y12 + · · ·cij · yij + ·+ ckl · ykl

∑i=1

∑j=1

cij · yij

Definition

∑i=1

∑j=1

cij · yij

Definition

∑i=1

∑j=1

cij · yij

Definition

∑i=1

∑j=1

cij · yij

Definition

∑i=1

∑j=1

cij · yij

Definition

∑i=1

∑j=1

cij · yij

2 C•Y is precisely trace(CT ·Y).

3 C•Y is a linear function of variables yij with coefficients cij .

Definition

∑i=1

∑j=1

cij · yij

The Model

1 Let Mn denote the cone of symmetric n×n matrices. If A ∈Mn, we use A� 0 to denotethe fact that A is positive semidefinite.

2 Let C,D1,D2, . . . ,Dk ∈Mn and let d1,d2, . . .dk ∈ℜ. The semidefinite programmingproblem (denoted by S) is defined as follows:

max C•Y

subject to Di •Y = di , 1≤ i ≤ k

Y� 0,

Y ∈Mn

The Model

max C•Y

Y� 0,

Y ∈Mn

The Model

1 Let Mn denote the cone of symmetric n×n matrices.

If A ∈Mn, we use A� 0 to denotethe fact that A is positive semidefinite.

max C•Y

Y� 0,

Y ∈Mn

The Model

max C•Y

Y� 0,

Y ∈Mn

The Model

2 Let C,D1,D2, . . . ,Dk ∈Mn and let d1,d2, . . .dk ∈ℜ.

The semidefinite programmingproblem (denoted by S) is defined as follows:

max C•Y

Y� 0,

Y ∈Mn

The Model

max C•Y

Y� 0,

Y ∈Mn

The Model

max C•Y

Y� 0,

Y ∈Mn

The Model

max C•Y

Y� 0,

Y ∈Mn

The Model

max C•Y

Y� 0,

Y ∈Mn

The Model

max C•Y

Y� 0,

Y ∈Mn

Properties of semidefinite programs

Observation

1 Semidefinite programming generalizes linear programming. In the semidefinite program S,if the matrices C,D1,D2, . . . ,Dk are all diagonal, then S becomes a linear programmingproblem.

2 Inequality constraints can be added, without affecting the form of S.3 The set of feasible solutions of S forms a convex set (since the convex combination of

semidefinite matrices is semidefinite).

4 Let A ∈ℜn×n be an infeasible point. Let C ∈ℜn×n. A hyperplane C•Y≤ b is called aseparating hyperplane for A, if all feasible points satisfy it and point A does not satisfy it.

5 Most effective for maximization problems involving constraints between pairs of variables.6 Leads to reasonably good approximation ratios, although in most cases, not the best

possible.

Observation

possible.

Observation

1 Semidefinite programming generalizes linear programming.

In the semidefinite program S,if the matrices C,D1,D2, . . . ,Dk are all diagonal, then S becomes a linear programmingproblem.

possible.

Observation

possible.

Observation

2 Inequality constraints can be added, without affecting the form of S.

3 The set of feasible solutions of S forms a convex set (since the convex combination ofsemidefinite matrices is semidefinite).

possible.

Observation

2 Inequality constraints can be added, without affecting the form of S.3 The set of feasible solutions of S forms a convex set

(since the convex combination ofsemidefinite matrices is semidefinite).

possible.

Observation

possible.

Observation

4 Let A ∈ℜn×n be an infeasible point. Let C ∈ℜn×n.

A hyperplane C•Y≤ b is called aseparating hyperplane for A, if all feasible points satisfy it and point A does not satisfy it.

possible.

Observation

possible.

Observation

5 Most effective for maximization problems involving constraints between pairs of variables.

6 Leads to reasonably good approximation ratios, although in most cases, not the bestpossible.

Observation

possible.

Separating hyper-planes and vector programs

Separating hyperplane theorem

Theorem

Let S be a semidefinite programming problem, and let A be a point in ℜn×n. We can determine,in polynomial time, whether A is feasible for S and, if it is not, find a separating hyperplane.

Proof.

If A is feasible, then A should be symmetric, positive semidefinite, and satisfy all the linearconstraints.Clearly, the above checks can be carried out in polynomial time.

The separating hyperplane can be found as follows:

1 If A is not symmetric, then aij > aji , for some i, j . Then, yij ≤ yji is a separating hyperplane.(Violation of Y ∈Mn.)

2 If A is not positive semidefinite, then it has a negative eigenvalue, say λ . Let v be thecorresponding eigenvector. Now (v ·vT)•Y = vT ·Y ·v≥ 0 is a separating hyperplane.(Violation of Y� 0.)

3 If any of the linear constraints is violated, it directly yields a separating hyperplane.(Violation of Di •Y = di , for some i = 1,2, . . . ,k .)

Theorem

Let S be a semidefinite programming problem, and let A be a point in ℜn×n.

We can determine,in polynomial time, whether A is feasible for S and, if it is not, find a separating hyperplane.

Proof.

Theorem

Proof.

Theorem

Proof.

Theorem

Proof.

If A is feasible, then A should be

symmetric, positive semidefinite, and satisfy all the linearconstraints.Clearly, the above checks can be carried out in polynomial time.

Theorem

Proof.

If A is feasible, then A should be symmetric,

positive semidefinite, and satisfy all the linearconstraints.Clearly, the above checks can be carried out in polynomial time.

Theorem

Proof.

If A is feasible, then A should be symmetric, positive semidefinite,

and satisfy all the linearconstraints.Clearly, the above checks can be carried out in polynomial time.

Theorem

Proof.

If A is feasible, then A should be symmetric, positive semidefinite, and satisfy all the linearconstraints.

Clearly, the above checks can be carried out in polynomial time.

Theorem

Proof.

Theorem

Proof.

Theorem

Proof.

1 If A is not symmetric, then aij > aji , for some i, j .

Then, yij ≤ yji is a separating hyperplane.(Violation of Y ∈Mn.)

Theorem

Proof.

1 If A is not symmetric, then aij > aji , for some i, j . Then, yij ≤ yji is a separating hyperplane.

(Violation of Y ∈Mn.)

Theorem

Proof.

Theorem

Proof.

2 If A is not positive semidefinite, then it has a negative eigenvalue, say λ .

Let v be thecorresponding eigenvector. Now (v ·vT)•Y = vT ·Y ·v≥ 0 is a separating hyperplane.(Violation of Y� 0.)

Theorem

Proof.

2 If A is not positive semidefinite, then it has a negative eigenvalue, say λ . Let v be thecorresponding eigenvector.

Now (v ·vT)•Y = vT ·Y ·v≥ 0 is a separating hyperplane.(Violation of Y� 0.)

Theorem

Proof.

2 If A is not positive semidefinite, then it has a negative eigenvalue, say λ . Let v be thecorresponding eigenvector. Now (v ·vT)•Y = vT ·Y ·v≥ 0 is a separating hyperplane.

(Violation of Y� 0.)3 If any of the linear constraints is violated, it directly yields a separating hyperplane.

(Violation of Di •Y = di , for some i = 1,2, . . . ,k .)

Theorem

Proof.

Theorem

Proof.

Equivalence between vector programs and semidefinite programs

Let V be a vector program on n n-dimensional vector variables v1,v2, . . .vn. Define thecorresponding semidefinite program, S, over n2 variables yij ,1≤ i, j ≤ n, as follows:

1 Replace each inner product vi ·vj occurring in V by the variable yij .

2 The objective function and constraints are now linear in the yij ’s.3 Additionally, require that matrix Y , whose (i, j)th entry is yij , be symmetric and positive

semidefinite.

Vector program V is equivalent to semidefinite program S.

semidefinite.

Let V be a vector program on n n-dimensional vector variables v1,v2, . . .vn.

Define thecorresponding semidefinite program, S, over n2 variables yij ,1≤ i, j ≤ n, as follows:

semidefinite.

2 The objective function and constraints are now linear in the yij ’s.

3 Additionally, require that matrix Y , whose (i, j)th entry is yij , be symmetric and positivesemidefinite.

semidefinite.

Proof of Equivalence

Proof.

1 Let a1,a2, . . .an denote a feasible solution to the vector program V .

2 Let W be the matrix whose columns are a1,a2, . . .an.3 A = WT ·W is a feasible solution to the SDP S, having the same objective function value.

4 Let A be a feasible solution to S. Since A is positive semidefinite, there must exist an n×nmatrix W, such that A = WT ·W.

5 Let a1,a2, . . .an denote the columns of W.6 These columns form a feasible solution to the vector program V , having the same objective

function value.

Proof.

function value.

Proof.

function value.

Proof.

2 Let W be the matrix whose columns are a1,a2, . . .an.

3 A = WT ·W is a feasible solution to the SDP S, having the same objective function value.

function value.

Proof.

function value.

Proof.

4 Let A be a feasible solution to S.

Since A is positive semidefinite, there must exist an n×nmatrix W, such that A = WT ·W.

function value.

Proof.

function value.

Proof.

5 Let a1,a2, . . .an denote the columns of W.

6 These columns form a feasible solution to the vector program V , having the same objectivefunction value.

Proof.

function value.

The Semidefinite Programming - Fundamentals

Documents

Semidefinite programming in combinatorial …...Mathematical Programming 79 (1997) 143-161 Semidefinite programming in combinatorial optimization 1 Michel X. Goemans 2 MIT, Department

On the Power of Semidefinite Programming …On the Power of Semidefinite Programming Hierarchies Prasad Raghavendra Georgia Institute of Technology, Atlanta, GA. David Steurer Microsoft

On the Power of Semidefinite Programming Hierarchies

Semidefinite Programming Based on slides by Uri Zwick Lecture 8: Magnus M. Halldorsson

HANDBOOK OF SEMIDEFINITE PROGRAMMING - GBV

A nonlinear programming algorithm for solving semidefinite

Digital Filter Design Using Semidefinite Programming

STUDIES ON OPTIMIZATION METHODS FOR NONLINEAR SEMIDEFINITE PROGRAMMING PROBLEMS · 2015-03-10 · OPTIMIZATION METHODS FOR NONLINEAR SEMIDEFINITE PROGRAMMING PROBLEMS by YUYA YAMAKAWA

Semidefinite programming relaxations for …parrilo/pubs/files/SDPrelaxations.pdfsubsection to present the basic ideas behind the convex optimization techniques used, namely semidefinite

Semidefinite Programming Lecture 22: Apr 2. Semidefinite Programming What is semidefinite programming? A relaxation of quadratic programming. A special

A Combinatorial, Primal-Dual Approach to Semidefinite Programming

Learning the Kernel Matrix with Semidefinite Programming

The Geometry of Semidefinite Programming - AIM · Semideﬁnite programming is the computational problem ... Linear programming is semideﬁnite programming for diagonal ... Algorithms

Report_Convex approximations in stochastic programming by semidefinite programming

Semidefinite Programming (SDP) and the Goemans … · Semidefinite Programming (SDP) and the Goemans-Williamson MAXCUT Paper ... . Outline

Algorithmic Conversion of Polynomial Optimization Problems of Noncommuting Variables to Semidefinite Programming Relaxations

Graph Laplacian Regularization for Large-Scale Semidefinite Programming

Introduction to Semidefinite Programs Masakazu Kojima Semidefinite Programming and Its Applications Institute for Mathematical Sciences National University

Bounding Option Prices Using Semidefinite Programming

A SEMIDEFINITE PROGRAMMING APPROACH TO THE GRAPH ...manchoso/papers/thesis.pdf · a semidefinite programming approach to the graph realization problem: theory, applications and extensions