Initialization in semidefiniteprogramming via a self-dualskew-symmetric embedding

Report 96-10

E. de KlerkC. Roos

T. Terlaky

Technische Universiteit DelftDelft University of Technology

Faculteit der Technische Wiskunde en InformaticaFaculty of Technical Mathematics and Informatics

ISSN 0922-5641

Copyright c 1996 by the Faculty of Technical Mathematics and Informatics, Delft, TheNetherlands.No part of this Journal may be reproduced in any form, by print, photoprint, microfilm,or any other means without permission from the Faculty of Technical Mathematics andInformatics, Delft University of Technology, The Netherlands.

Copies of these reports may be obtained from the bureau of the Faculty of TechnicalMathematics and Informatics, Julianalaan 132, 2628 BL Delft, phone+31152784568.A selection of these reports is available in PostScript form at the Faculty’s anonymousftp-site. They are located in the directory /pub/publications/tech-reports at ftp.twi.tudelft.nl

DELFT UNIVERSITY OF TECHNOLOGYREPORT Nr. 96{10Initialization in semidefiniteprogramming via a self{dualskew{symmetric embedding.E. de Klerk, C. Roos, T. Terlaky

ISSN 0922{5641Reports of the Faculty of Technical Mathematics and Informatics Nr. 96{10Delft, January, 1996i

E. de Klerk, C. Roos and T. Terlaky,Faculty of Technical Mathematics and Informatics,Delft University of Technology,P.O. Box 5031, 2600 GA Delft, The Netherlands.e{mail: e.deklerk@twi.tudelft.nl; c.roos@twi.tudelft.nl; t.terlaky@twi.tudelft.nl

Copyright c 1996 by Faculty of Technical Mathematics and Informatics, Delft,The Netherlands.No part of this Journal may be reproduced in any form, by print, photo-print, micro�lm or any other means without written permission from Facultyof Technical Mathematics and Informatics, Delft University of Technology, TheNetherlands. ii

AbstractThe formulation of interior point algorithms for semide�nite programming hasbecome an active research area, following the success of the methods for large{scale linear programming. Many interior point methods for linear programminghave now been extended to the more general semide�nite case, but the initial-ization problem remained unsolved.In this paper we show that the initialization strategy of embedding theproblem in a self{dual skew{symmetric problem can also be extended to thesemi{de�nite case. This way the initialization problem of semi{de�nite prob-lems is solved. This method also provides solution for the initialization ofquadratic programs and it is applicable to more general convex problems withconic formulation.Key words: Semide�nite programming, complementarity, skew{symmetricembedding, initialization, self{dual problems, central path.

iii

1 IntroductionThe extension of interior point algorithms from linear programming (LP) to semide�niteprogramming (SDP) [1] has received much attention recently, as is clear from the numberof recent papers dealing with the subject (e.g. [4, 7, 9]).The question of initialization has not been satisfactorily solved up to now, and `big-M'methods are often employed in practice to obtain feasible starting points. Potra andSheng [7] describe an infeasible start method. After homogenizing the SDP problem inthe primal{dual formulation they propose an infeasible interior point method.In the LP case an elegant solution for the initialization problem is to embed the originalproblem in a skew{symmetric self{dual problem which has a known interior feasible so-lution [11, 6]. The solution of the embedding problem then yields the optimal solution tothe original problem, or detects infeasibility or unboundedness.In this paper we show how to extend this idea to semide�nite programming. To this endthe classical orthogonality properties of LP problems are also extended to SDP problems.We give a self{dual skew{symmetric embedding with known interior feasible solution onits central path, and show that an optimal solution of this problem yields the solution ofthe original problem. This means that one of the following three cases occur:� an optimal solution with zero duality gap for the original problem is obtained;� a recession direction for either the primal and/or dual problem is obtained;� one obtains a certi�cate that the optimal duality gap to the original problem isstrictly positive (This might happen if neither of the primal or dual SDP problemssatisfy the Slater regularity condition [4]).Since our embedding problem has an initial interior feasible solution on its central pathany interior point method can be applied to it.It is known [4] that QP is a special case of SDP. Thus the initialization via self{dualembedding for convex quadratic programming (QP) problems is covered as well. Via thisreformulation and embedding the problems arising from the nonlinearity of the objectiveas mentioned in [10] are avoided. A homogeneoues embedding of monotone nonlinearcomplementarity problems is discussed in [2]. An infeasible starting algorithm is proposedthere to solve the homogeneous model. Contrary to [2] our embedding problem has anonempty interior and a perfectly centered initial starting point.The paper is organized as follows. In Section 2 the primal-dual SDP pair is de�ned inthe symmetric form. First the orthogonality properties of the primal and dual solutionsare proved. Furthermore after de�ning the central path of the SDP problem it will beproved that any convergent subsequence on the central path converges to a maximally1

complementary solution. In Section 3 the skew{symmetric embedding model is presented.This problem has a trivial initial solution on its central path. We show that the limit pointof the central path of the embedding problem either provides an optimal primal{dual pairwith zero duality gap, or shows that such solutions does not exist.2 Orthogonality, Maximal Complementarity in SDPThe semi{de�nite programming problem will be considered in the symmetric form. Thusour primal problem (P) is minX;z Tr(CX)subject to Tr(AiX) � zi = bi; i = 1; : : : ;mX � 0zi � 0; i = 1; : : : ;mwith the associated dual problem (D) maxS;y bTysubject to mXi=1 yiAi + S = C;S � 0;yi � 0; i = 1; : : : ;mwhere C and the Ai's are symmetric n � n matrices, b 2 IRm, and X � 0 means Xis positive semi{de�nite. The solutions (X; z) and (y; S) will be referred to as feasiblesolutions as they satisfy the primal and dual constraints respectively. Observe that theduality gap for P and D is given byTr(CX)� bTy = Tr ( mXi=1 yiAi + S)X!� mXi=1 yi (Tr(AiX) � zi) = Tr(SX) + yTz:A basic property of LP problems known as the orthogonality property easily extends toSDP.Lemma 2.1 (Orthogonality) Let (X; z; S; y) and (X0; z0; S0; y0) be two pairs of feasiblesolutions. The following orthogonality relation holds:Tr �(X �X0)(S � S0)�+ (z � z0)T (y � y0) = 0:2

Proof:The proof uses only the feasibility conditions of P and D.Tr �(X �X0)(S � S0)�+ (z � z0)T (y � y0) =Tr (X �X0) mXi=1(y0i � yi)Ai!!+ mXi=1Tr �Ai(X �X0)(yi � y0i )� = 0: 2It is well known that the pair of problems P and D have optimal solutions (X�; z�; S�; y�)with zero duality gap (Tr(CX�)� bTy� = 0) if at least one of the primal or dual problemssatis�es the Slater regularity condition [3].The optimality conditions for P and D areTr(AiX) � zi = bi; i = 1; : : : ;m z � 0 X � 0Pmi=1 yiAi + S = C y � 0; S � 0XS = 0yizi = 0; i = 1; : : : ;m: 9>>>>>>>>=>>>>>>>>; (1)Solutions (X; z) and (S; y) satisfying the last two equality constraints are called comple-mentary. SinceX and S are symmetric positive semi{de�nite matrices the complementar-ity of X and S (XS = 0) is equivalent to Tr(XS) = 0. Complementary feasible solutionsare optimal. If the complementarity conditions are relaxed toTr(AiX)� zi = bi; i = 1; : : : ;mX yiAi + S = CXS = �Iyizi = �; i = 1; : : : ;mX;S � 0z; y � 0;with some � > 0, then this system has a unique solution X(�); z(�); S(�); y(�) [4]. Thissolution can been seen as the parametric representation of a smooth curve (the centralpath) leading to the optimal set as � ! 0. In the rest of this section we assume thatthe central path exists. This is true for the embedding problem introduced in the nextsection.We show that the central path converges to a special optimal solution, a so{called maxi-mally complementary solution, de�ned as follows.3

De�nition 2.1 An optimal solution (X�; S�; z�; y�) is called a maximally complementarysolution if z� and y� have the maximal number of strictly positive components, and if X�and S� have maximal rank.A maximally complementary solution can be obtained by a central path following algo-rithm { we prove below that all accumulation points of the central path in the optimalset are maximally complementary solutions. This result is analogous to the result thatthe central path of an LP problem converges to a strictly complementary solution [5, 6].Recall that in the LP case a strictly complementary solution exists (at least one of thecomplementary pair is positive) [6]. This is not the case for nonlinear problems, not evenfor convex quadratic problems.Theorem 2.1 (Maximal complementarity) Consider any sequence f�tg ! 0 with�t > 0 t = 1; : : :. Any convergent subsequence of the points (X(�t); z(�t); S(�t); y(�t)) onthe central path converges to a maximally complementary solution as �t ! 0.Proof:We will prove the theorem in two steps. First we prove that the limit point (x; s) of(z(�t); y(�t)) is maximally complementary.Lemma 2.2 The limit point (z; y) of the series (z(�t); y(�t)) is maximally complemen-tary. Furthermore for any maximally complementary solution (X�; S�; z�; y�) the seriesTr(S�1(�t)S�) and Tr(X�X�1(�t))are uniformly bounded.Proof:Let (X�; S�; z�; y�) be a maximally complementary solution, and (X(�); z(�); S(�); y(�))a central solution corresponding to some � > 0. By orthogonality, Lemma 2.1, one hasTr ((X(�) �X�)(S(�)� S�)) + (z(�) � z�)T (y(�)� y�) = 0:The centrality conditions imply Tr (X(�)S(�)) = n� and z(�)Ty(�) = m�. Using theoptimality conditions Tr(X�S�) = 0 and (z�)Ty� = 0, this simpli�es toTr(X(�)S�) + Tr(X�S(�)) + (z�)Ty(�) + z(�)Ty� = (m+ n)�or, using the centrality conditions once more, we haveTr(S�1(�)S�) + Tr(X�X�1(�)) + mXj=1z�j>0 z�jzj(�) + mXj=1y�j>0 y�jyj(�) = m+ n: (2)4

By feasibility each term in (2) is nonnegative and thus for the sequence �t we havelim�t!0 zj(�t) > 0 if z�j > 0 and lim�t!0 yj(�t) > 0 if y�j > 0.Now consider the term Tr(S�1(�t)S�) and Tr(X�X�1(�t)) in (2). These terms are non-negative and bounded from above because all the terms in the left hand side of (2) arenonnegative. This way the lemma is proved. 2To prove Theorem 2.1 we still have to prove that the series X(�t) and S(�t) are alsomaximally complementary, e.g. for any convergent subsequence if X = lim�t!0X(�t)and S = lim�t!0 S(�t) then X and X� have the same rank and further S and S� havethe same rank.Lemma 2.3 The matrices X and S are maximally complementary.Proof:We prove here that the rank of X is the same as the rank of X�. The proof for S proceedsanalogously. We recall from Lemma 2.2 that Tr(X(�t)�1X�) � m+ n for all �t > 0. Letthe rank ofX� be k, its eigenvalues be denoted by (��1; : : : ; ��k), and a set of its orthonormaleigenvectors be given as (e�1; : : : ; e�k). Then X� = P ���P �T where P � = (e�1; : : : ; e�k) is ann� k orthonormal and �� = diag(��1; : : : ; ��k) is a k � k diagonal matrix. Then we haveTr(X(�t)�1X�) = Tr �X(�t)�1P ���P �T� = Tr �X(�t)�12 P ���P �TX(�t)�12 � � m+ n:(3)Using this bound we deriveTr(P �TX(�t)�1P �) = Tr �X(�t)�12 P �P �TX(�t)�12 �= Tr �X(�t)�12 P ��� 12���1�� 12P �TX(�t)�12 �� k���1kTr �X(�t)�12 P ���P �TX(�t)�12 �� k���1k(m+ n):For further reference we introduce the notation K := k���1k(m + n). Since X(�t) ispositive de�nite symmetric, its orthonormal eigenvalue decomposition for each �t can begiven as X(�t) = Q(�t)TL(�T )Q(�t);where L(�t) is an n � n positive diagonal matrix and Q(�t)T = Q(�t)�1 is an n � northonormal matrix. Then X(�t)�1 = Q(�t)TL(�t)�1Q(�t) and further B(�t) = Q(�t)P �is also an orthonormal matrix. Using this we haveTr(P �TX(�t)�1P �) = Tr �P �TQ(�t)TL(�t)�1Q(�t)P ��= Tr �B(�t)TL(�t)�1B(�t)�= Tr �L(�t)�1B(�t)B(�t)T�5

= nXi=1 B(�t)iB(�t)TiL(�t)i � K: (4)Introducing the notation �(�t)i = B(�t)iB(�t)Ti we have that 0 � �(�t)i � 1 for all i andnXi=1 �(�t)i = Tr �B(�t)TB(�t)� = Tr �B(�t)B(�t)T� = k:These last two relations imply that at least k of the n �(�t)i's are larger than or equal to1n�k+1 . We can choose an appropriate subsequence (indicated again by subscript t for thesake of simplicity) where these coordinates are �xed. Denote the set of these indices byI. Then we have �(�t)i � 1n� k + 1 ; i 2 Iand by (4) L(�t)i � 1(n� k + 1)K ; i 2 I:Using the notation L = lim�t!0 L(�t) we conclude that the diagonal matrix L has at leastk nonzero diagonal elements. Since Q(�t) is orthonormal, for an appropriate subsequence(still indicated by subscript t) one has Q = lim�t!0Q(�t) orthonormal, thusX = lim�t!0X(�t) = lim�t!0Q(�t)TL(�t)Q(�t) = QTLQhas at least rank k. By noting that X� has maximal rank among the optimal solutions,one has rank(X) � rank(X�) and thus rank(X) = k. This completes the proof. 2To decide about the duality state of the problemsP andD we need the following de�nition.De�nition 2.2 We say that the primal problem P has a ray if there is a symmetricmatrix Xr � 0 such that Tr(AiXr) � 0; 8 i and Tr(CXr) < 0.Analogously, the dual problem D has a ray if there is a vector 0 � yr 2 IRm such that�Pmi=1 yriAi � 0 and bTyr > 0.Clearly if there is either a primal or dual ray, then no optimal solution to either P or Dexists. If there is a dual ray yr then P is infeasible, since by assuming primal feasibility onehas 0 � Pmi=1 Tr(AiX)yri � bTyr > 0 for any primal feasible X, which is a contradiction.If one has a primal ray Xr then D is infeasible since by assuming dual feasibility one has0 � Pmi=1Tr(AiXr)yi � Tr(CXr) < 0 for any dual feasible y, which is also a contradiction.A maximally complementary solution to our embedding problem, presented in the nextsection, will enable us to construct optimal solutions to P and D if they exist or decideabout the duality state of these problems. 6

3 Skew{symmetric Self{dual EmbeddingWe now embed the primal{dual pair of problems P and D in a larger problem. This isdone in two steps: A homogeneous skew{symmetric, self{dual SDP problem is constructedwhich is then extended to a problem with nonempty interior of the feasible region. Thefollowing skew{symmetric model can be obtained by homogenizing the optimality condi-tions of P and D. Skew{symmetric formulationFind a � > 0 such that Tr(AiX) ��bi �z = 0 8i�Pmi=1 yiAi +�C �S = 0bTy �Tr(CX) �� = 0y � 0; X � 0; � � 0; z � 0; S � 0; � � 0: 9>>>>>>>>=>>>>>>>>; (5)Note that the skew{symmetric model (5) always admits the trivial zero solution. More-over, a solution (X�; z�; S�; y�; � �; ��) to (5) with � � > 0 yields an optimal solution tothe original problem (scale the variables X� and y� by 1�� ). This is true due to the lastconstraint that requires the duality gap for the original pair of problems P and D benonpositive. As the gap is always nonnegative by weak duality, optimality of the scaledsolutions is established. Clearly if P and D have no optimal solution with zero dualitygap then the skew{symmetric problem has no solution with � � > 0.Note that the feasibility problem (5) is not only skew{symmetric but also a self{dual opti-mization problem with zero objective vector { both the right hand side and the objectivevectors are zero, and the coe�cient matrix is skew{symmetric. Due to weak duality thisproblem has no interior solution because for any feasible pair of P and D (or any multipleof it by a positive � ) one has bTy � Tr(CX). The skew{symmetric formulation musttherefore be extended.By introducing some new vectors, parameters and variables a skew{symmetric self{dualmodel with known initial interior solution is obtained. The construction is analogous tothat given in [6] for LP. Let e be the all one vector. To ease the discussions later on, weinclude the slack variables in the problem formulation as well.Self{dual embeddingminy;X;�;�;z;S;�;� ��7

subject to Tr(AiX) ��bi +��b �z = 0 8i�Pmi=1 yiAi +�C �� �C �S = 0bTy �Tr(CX) +�� �� = 0��bTy +Tr( �CX) ��� �� = ��z � 0; X � 0; y � 0; S � 0; � � 0; � � 0; � � 0; � � 0where �bi := bi + 1 � TrAi� �C := I + mXi=1Ai � C� := 1 + TrC � bT e� := 2n + 2:It is straightforward to verify that a feasible interior starting solution is given by y0 =z0 = e, X0 = S0 = I, �0 = �0 = � 0 = �0 = 1. As the formulation is self{dual (observethat { excluding the slack variables { the coe�cient matrix is skew{symmetric and theright hand side vector is just the negative of the objective vector). The abovementionedinterior solution proves existence of the central path and an optimal solution with zeroduality gap: �� = 0. This last observation also follows trivially by observing that the allzero solution is feasible and optimal.The results about the convergence of the central path to a maximally complementarysolution of the previous section apply to this problem, thus the following theorem can beproved.Theorem 3.1 For the given problems P and D the following alternatives can be distin-guished correctly by any limit point of the central path of the embedding problem.(i) Both P and D are feasible and there is a complementary optimal pair (X; y) for Pand D with Tr(CX) = bTy.(ii) Either P or D or both have a ray.(iii) Either both P and D are feasible, but at the optimum the duality gap is nonzero, orat least one of them is infeasible but neither for P nor for D a ray exists.Proof:The above self{dual problem has a skew{symmetric coe�cient matrix and it is easy tocheck that it is self{dual. Due to the selection of the parameters it has a positive solution8

as we have seen above. Hence Lemma 2.1 guarantees that a limit point of its centralpath is a maximally complementary optimal solution. Let y�; X�; � �; ��; z�; S�; ��; ��be such a limit point. By construction we also know, that �� = 0, since � > 0 and theoptimal value is zero.We have two possibilities. If � � > 0, then (X�;z�)�� and (y� ;S�)�� are maximal complementaryand optimal for P and D respectively, i.e. case (i) holds.If � � = 0 then there exists no optimal solutions of P and D with duality gap zero becauseif it would exist then the embedding problem would have an optimal solution with � = 1,thus in any maximally complementary solution � would be positive. Having � � = 0 inour maximally complementary solution we have that � = 0 in any optimal solution of theembedding problem.Now we have � � = 0 thus Tr(AiX�) � 0 for all i and ATy� � 0. If �� > 0 thenbTy� � Tr(CX�) > 0, i.e. case (ii) applies. If bTy� > 0 then y� is a dual ray. In this caseif D is feasible then P is infeasible. If Tr(CX�) < 0 then we have a primal ray. In thiscase if P is feasible then D is infeasible. If both bTy� > 0 and Tr(CX�) < 0 then we haveboth a primal and a dual ray and in this case at least one of P and D is infeasible.If � � = 0 and �� = 0 then we have Ax� � 0, ATy� � 0 and bTy� � cTx� = 0 and one hasno additional information. Due to the maximal complementarity property this case occurif neither optimal solutions with zero duality gap nor rays for both problems exist. Hereit might happen that both P and D have optimal solutions with nonzero duality gap, butit is also possible that one of them is, or both of them are infeasible. 2Remark 1: Embedding, even with points on the central path is not restricted to theuse of the all one vectors. If one has any �0 > 0, � 0 > 0, �0 > 0, �0 > 0, �0 > 0 andy0 > 0; z0 > 0, X0; S0 symmetric positive de�nite with � 0�0 = �0, �0�0 = �0, y0i z0i = �0for all i and X0S0 = �0I then we can choose�bi := � 0bi + z0i � Tr(AiX0)�i� �C := S0 +Pmi=1 y0iAi � � 0C�0� := �0 + Tr(CX0)� bTy0�0� := �0 + � 0� � Tr( �CX0) + �bTy0�0 :It is straightforward to verify that the skew{symmetric self{dual problem obtained withthe use of these vectors admits the above speci�ed initial positive solution which is on itscentral path associated to the point with �0 > 0. It is also easy to check by simple calcula-tion that � = (m+n+2)�0�2 > 0. This embedding is therefore also suitable for reoptimizationif some problem data change. 9

Remark 2: It would be interesting to study the self{dual embedding problem of thedual semide�nite programming problems of Ramana [8]. Using his dual problems, wherewithout assuming Slater condition the duality gap is always zero at optimum if both theprimal and the dual problems admit a feasible solution, hopefully case (iii) of Theorem3.1 can be excluded. To do this is the subject of further research.Remark 3: The results of the previous sections can be generalized to primal{dual convexproblems in the conic formulation. Consider the primal problem asminx f cTx j Ax� b 2 C1; x 2 C2 gand its dual problem as maxy f bTy j �ATy + c 2 C�2 ; y 2 C�1 gwhere C1; C2 are convex cones, C�1 ; C�2 are their dual cones respectively, A is an m �n matrix, b; y 2 IRm and c; x 2 IRn. These problems can be embedded in the skew{symmetric self{dual problem with nonempty interior as follows.minx;y;�;� ��subject to Ax ��b +��b 2 C1�ATy +�c ���c 2 C�2bTy �cTx +�� � 0��bTy +�cTx ��� � ��y 2 C�1 ; x 2 C2 � � 0 � � 0where �b := b+ 1 �Ae��c := e+ATy � c� := 1 + cTe� bTe� := n + 2:Clearly the cone (C1 � C�2 � IR+ � IR+ � C�1 � C2 � IR+ � IR+) is self{dual and the allone solution is an initial interior feasible solution of this system. Analogous results canbe derived as for the positive semide�nite case. Maximal complementarity of the scalarvariables (which are in IR+) can be proved the same way as in Lemma 2.2. This issu�cient to prove the validity of the above embedding. To de�ne and prove maximalcomplementarity of the general conic variables is the subject of further research.Acknowledgment: The authors are grateful to Professor Yinyu Ye. During his visit toDelft he gave useful comments on an earlier version of the paper and corrected an errorin the proof of Lemma 2.3. 10

References[1] F. Alizadeh. Combinatorial optimization with interior point methods and semi{de�nite matrices. PhD thesis, University of Minnesota, Minneapolis, USA, 1991.[2] E.D. Andersen and Y. Ye. On a homogeneous algorithm for the monotone comple-mentarity problem. Technical Report, Department of Management Sciences, Univer-sity of Iowa, Iowa City, USA, 1995.[3] M.S. Bazarraa, H.D. Sherali and C.M. Shetty, Nonlinear Programming: Theory andAlgorithms, John Wiley and Sons, New York (1993).[4] S.E. Boyd and L. Vandenberghe. Semide�nite programming. SIAM Review, 1995.[5] O. G�uler and Y. Ye. Convergence behavior of interior{point algorithms.MathematicalProgramming, 60:215{228, 1993.[6] B. Jansen, C. Roos, and T. Terlaky. The theory of linear programming : Skew sym-metric self{dual problems and the central path. Optimization, 29:225{233, 1994.[7] F.A. Potra and R. Sheng. Homogeneous interior{point algorithms for semide�niteprogramming. Reports on Computational Mathematics 82, Dept. of Mathematics,The University of Iowa, Iowa City, USA, 1995.[8] M. Ramana. An exact duality theory for semide�nite programming and its complexityimplications. DIMACS Technical Report 95{02, RUTCOR, Rutgers University, NewBrunswick, New Jersey, USA, 1995.[9] J.F. Sturm and S. Zhang. Symmetric primal{dual path following algorithms forsemide�nite programming. Technical Report 9554/A, Tinbergen Institute, ErasmusUniversity Rotterdam, 1993.[10] Y. Ye. An homogeneous and self{dual algorithm for LCP. Working Paper, Dept. ofManagement Science, University of Iowa, Iowa City, USA, 1994.[11] Y. Ye, M.J. Todd, and S. Mizuno. An O(pnL){iteration homogeneous and self{duallinear programming algorithm. Mathematics of Operations Research, 19:53{67, 1994.11