Simultaneous Diagonalizationeaton.math.rpi.edu/faculty/Mitchell/courses/matp...2 ˜ 3 i 2 4 ⇤ 100 000 000 3 5 2 QT Q˜T 2 Q˜T 3 3 and X⇤ = h Q1 ˜ 2 Q˜ 3 i 2 4 00 0 00 0 00⇤3

Simultaneous Diagonalization

John E. Mitchell

Department of Mathematical SciencesRPI, Troy, NY 12180 USA

April 2018

Mitchell Simultaneous Diagonalization 1 / 22

SDP formulation

Outline

1 SDP formulation

2 Simultaneous diagonalization

3 Simultaneous diagonalization of optimal solutions

4 Strict complementarity


SDP formulation

Primal SDPWe write our standard form semidefinite program as

minX C • Xsubject to Ai • X = bi , i = 1, . . . ,m

X ⌫ 0(1)

where:

C is an n ⇥ n symmetric matrixAi is an n ⇥ n symmetric matrix for i = 1, . . . ,mbi is a scalar for i = 1, . . . ,m.

The parameter matrices C and Ai need not be positive semidefinite,although they are assumed to be symmetric. Recall that C • Xrepresents the Frobenius inner product between the symmetricmatrices C and X , which is equal to the trace(CX ).


SDP formulation

Dual SDP

The dual problem is

maxy2IRm,S2Sn+

bT ysubject to

Pmi=1 yiAi + S = C

S ⌫ 0(2)

Notice that the dual slack variables S = C �Pm

i=1 yiAi constitute asymmetric positive semidefinite matrix in feasible dual solutions.


Simultaneous diagonalization

Outline

1 SDP formulation







Two symmetric n ⇥ n matrices are simultaneously diagonalizable ifthey have the same eigenvectors.

LemmaIf the n ⇥ n symmetric matrices M and R are simultaneouslydiagonalizable then they commute.


1412=1211.


Proof of lemma

Let the columns of the orthogonal matrix P consist of the eigenvectorsof the matrices, so

M = P⇤PT , R = P⌅PT

for two diagonal matrices ⇤ and ⌅. Then

MR = P⇤PT P⌅PT

= P⇤⌅PT since P is orthogonal= P⌅⇤PT since diagonal matrices commute= P⌅PT P⇤PT since P is orthogonal= RM

as required.


A : diagonal.din, entries

a r eeigenvalues.

- f t p . E - p t pA O F-fr...-in)

e .I- I l

I I l

'f";D"..-11"":*.)


The converseThe converse also holds, so symmetric matrices commute if and only ifthey are simultaneously diagonalizable.

Sketch of proof of converse:Assume symmetric R, M commute.Assume R = UDUT , so diagonal entries of D are eigenvalues of R,and columns of U are eigenvectors of R. Then

RM = MR =) UDUT M = MUDUT =) DUT MU = UT MUD

since U orthogonal. For (i , j)-entry, Dii(UMUT )ij = (UMUT )ijDjj , so

UMUT =

2

6664

⇤ 0 . . . 00 ⇤ 0... . . . ...0 0 . . . ⇤

3

7775

where each block correspondsto a different eigenvalue Dii .Diagonalize M by diagonalizingdiagonal blocks / eigenspaces of R


-

ti¥Duu=u¥@Eu)

- I I 1 ) G )

I -

I-

Simultaneous diagonalization of optimal solutions

Outline

1 SDP formulation






Duality gap

Let X be feasible in (1) and (y ,S) be feasible in (2). The duality gap is

C • X � bT y =

mX

i=1

yiAi + S

!• X � bT y

= S • X +mX

i=1

yi(Ai • X )� bT y

= S • X + bT y � bT y= S • X .


M i r Cox m a x blys t . Aiox--bi f t . S=C-Ey.-Ai

X i a o S E O

n e

⇒ i

=


Frobenius product of X ⇤ and S⇤

Assume (1) and (2) have optimal solutions and their values agree.

Let X ⇤ solve (1) and (y⇤,S⇤) solve (2).

From the argument above, the Frobenius product

trace(S⇤X ⇤) = S⇤ • X ⇤ = 0. (3)


-Sun of alia, entries of

(5×44) i s z e r o .


Eigenvectors of optimal solutions

We can actually say something far stronger,namely the matrix product S⇤X ⇤ = 0, the zero matrix.

So it is not just the trace of S⇤X ⇤ that is equal to zero.

TheoremLet X ⇤ solve (1) and (y⇤,S⇤) solve (2) and assume S⇤ • X ⇤ = 0. ThenS⇤X ⇤ = 0 = X ⇤S⇤.


six#f:::-oil#

-


Proof, part 1Since S⇤ 2 Sn

+ it has an eigendecomposition,

S⇤ = Q⇤QT ,

where the columns {qi , i = 1, . . . , n} of Q are the eigenvectors of S⇤

and the diagonal entries of the diagonal matrix ⇤ are the eigenvaluesof S⇤. We now have

0 = S⇤ • X ⇤

= trace(Q⇤QT X ⇤)

= trace(⇤QT X ⇤Q) from properties of trace

=nX

i=1

⇤ii

⇣QT X ⇤Q

⌘

iisince ⇤ is diagonal

=nX

i=1

⇤ii qTi X ⇤qi


india:}

trace(ACE'X*Q)→=§(gAijCQTMQ7jiJ-

[email protected]'t.ie;

-=§{QkiX*uQti


Proof, part 2

Since X ⇤ is positive semidefinite, we have qTi X ⇤qi � 0 8i .

Since S⇤ is positive semidefinite, we have ⇤ii � 0 8i .Thus, we must have

⇤ii qTi X ⇤qi = 0 8i .

Hence, if qi is an eigenvector of S⇤ with a positive eigenvalue, we musthave

0 = qTi X ⇤qi = qT

i LLT qi = kLT qik2

where L is the Cholesky factor of X ⇤.

Thus, LT qi = 0, so X ⇤qi = 0. So qi is in the nullspace of X ⇤,so it is an eigenvector of X ⇤ with eigenvalue 0.


O = §Hi iqExqi- :3¥¥ 0 : X EO .

O- h i i - O

-l X l#EI :

-

LTgi= O = ) L Eqi i o ⇒ x * g i t0 = 0qi


Proof, part 3

We order the columns of Q as Q = [Q1,Q2],where the columns of Q1 are eigenvectors with positive eigenvalueand the columns of Q2 are eigenvectors with an eigenvalue of 0.

We can write the eigendecomposition as

S⇤ =⇥

Q1 Q2⇤ ⇤1 0

0 0

� QT

1QT

2

�

where the diagonal entries of ⇤1 are positive.

Note that X ⇤Q1 = 0, since each eigenvector qi of S⇤ with positiveeigenvalue satisfies X ⇤q1 = 0.


-

= e aQ t

-i

Q,:[q,... gr) ,w i l t r e r a n4 ( s t )

XQ,=[Xq, Xq...Xqr)


Proof, part 4

Thus,

X ⇤S⇤ = X ⇤ ⇥ Q1 Q2⇤ ⇤1 0

0 0

� QT

1QT

2

�

=⇥

0 X ⇤Q2⇤ ⇤1 0

0 0

� QT

1QT

2

�

=⇥

0 0⇤ QT

1QT

2

�

= 0, the n ⇥ n matrix of zeroes.

We also haveS⇤X ⇤ = (X ⇤S⇤)T = 0T = 0,

as required.


Im

- cyan①

since 54,#symmetric



Note that S⇤ and X ⇤ commute, with both products X ⇤S⇤ = S⇤X ⇤ = 0.

Hence, the matrices are simultaneously diagonalizable.

In general, we can construct an orthonormal basis for IRn consisting ofthree sets of vectors:

Eigenvectors of S⇤ with positive eigenvalue that are in thenullspace of X ⇤. These eigenvectors comprise the columns of amatrix Q1.A basis for the intersection of the nullspaces of X ⇤ and S⇤, whichwe denote as the columns of a matrix Q̃2, andEigenvectors of X ⇤ with positive eigenvalue that are in thenullspace of S⇤. These eigenvectors comprise the columns of amatrix Q̃3.


-
















Writing S⇤ and X ⇤ using eigenvectors

We can then write

S⇤ =h

Q1 Q̃2 Q̃3

i2

4⇤1 0 00 0 00 0 0

3

5

2

4QT

1Q̃T

2Q̃T

3

3

5

and

X ⇤ =h

Q1 Q̃2 Q̃3

i2

40 0 00 0 00 0 ⇤3

3

5

2

4QT

1Q̃T

2Q̃T

3

3

5

where the diagonal entries in ⇤1 and ⇤3 are positive.

This is the SDP version of complementary slackness.


-

Strict complementarity

Outline

1 SDP formulation






Strict complementarity for LP

In the linear programming case, there exist optimal solutions that arestrictly complementary.

The analogue of this in the SDP case is for

rank(X ⇤) + rank(S⇤) = n,

so there are no columns in Q̃2.

However, this does not always hold, as in the following example.



Example

The primal SDP is

minX2S3+

X33

subject to X11 = 1X13 + X22 + X31 = 0X12 + X33 + X21 = 0

X ⌫ 0

The unique optimal primal solution is

X ⇤ =

2

41 0 00 0 00 0 0

3

5 .


I f X , ,=0 : ⇒ XueXu--0,

[?§¥ 0= ) X , ]= K ,= Xs,= X} ,- 0

. = DX ,e-O.←

O


Dual of example

The dual problem is

maxy ,S y1

subject to S =

2

4�y1 �y3 �y2�y3 �y2 0�y2 0 1 � y3

3

5

S ⌫ 0.

The unique dual optimal solution is y⇤ = (0, 0, 0), giving

S⇤ =

2

40 0 00 0 00 0 1

3

5 .

Hence rank(X ⇤) = rank(S⇤) = 1 and rank(X ⇤)+rank(S⇤) = 2 < 3 = n.


f . c-Ey..Ai-18.%:/-g,foo!:b- y .fo,:{I-of

Documents

Simultaneous Diagonalizationeaton.math.rpi.edu/faculty/Mitchell/courses/matp...2 ˜ 3 i 2 4 ⇤ 100 000 000 3 5 2 QT Q˜T 2 Q˜T 3 3 and X⇤ = h Q1 ˜ 2 Q˜ 3 i 2 4 00 0 00 0 00⇤3