A Combinatorial, Primal-Dual A Combinatorial, Primal-Dual Approach to Semidefinite Approach to Semidefinite

ProgramsPrograms

Satyen KaleSatyen KaleMicrosoft ResearchMicrosoft Research

Joint work with Joint work with Sanjeev Arora Sanjeev Arora

Princeton UniversityPrinceton University

The Ubiquity of Semidefinite The Ubiquity of Semidefinite ProgrammingProgramming

Max Cut [GW’95]

Balanced Partitioning [ARV’04]

Graph Coloring [KMS’98, ACC’06]

(a Ç :b) Æ (:a Ç c) Æ (a Ç b) Æ (:c Ç b)

Constraint Satisfaction [ACMM’05]

1 0 0 0 1 1 0 1

SDP

Control Theory

dx(t)/dt = Ax(t)

Algorithms for SDPAlgorithms for SDP

Ellipsoid method Ellipsoid method [GLS’81][GLS’81]::• O(nO(n88)) iterations iterations

Interior point methods Interior point methods [NN’90, A’95][NN’90, A’95]::• O(√m(nO(√m(n33 + m)) + m)) time time

Lagrangian Relaxation Lagrangian Relaxation [KL’95], [AHK’05][KL’95], [AHK’05]::• Reduction to eigenvectorsReduction to eigenvectors• poly(1/poly(1/) ) dependence ondependence on , limits , limits applicability (e.g. applicability (e.g. Sparsest CutSparsest Cut))

A1 ² X ¸ b1

Am ² X ¸ bm

i wi(Ai ² X – bi) ¸ 0

Combinatorial, Primal-Dual algorithms

Primal-Dual algorithms for LP:Primal-Dual algorithms for LP:Multicommodity FlowsMulticommodity Flows

Unit capacities

Objective:Objective: Maximize total flow, Maximize total flow, while respecting edge capacitieswhile respecting edge capacities

1

1

1

1

1

1 1

1

11

1

1

11

1 + Edge weights Edge weights wwee = 1 = 1 Repeat until some Repeat until some ee reaches reaches

capacity:capacity: Find shortest pathFind shortest path p p Route Route 22/log(m) /log(m) flow on flow on pp Update Update wwee for all for all e e 22 p p as as

wwee ÃÃ w wee ¢¢ (1 + (1 + )) Output flow.Output flow.

Thm [GK’98]:Thm [GK’98]: Stops in Stops in Õ(m) Õ(m) rounds with rounds with (1 – 2(1 – 2))¢¢OPT OPT flow. Total running time is flow. Total running time is Õ(mÕ(m22).).

Primal-Dual algorithms for LP:Primal-Dual algorithms for LP:Multicommodity FlowsMulticommodity Flows

Primal-Dual algorithms for LP:Primal-Dual algorithms for LP:Multicommodity FlowsMulticommodity Flows

Thm [GK’98]:Thm [GK’98]: Stops in Stops in Õ(m) Õ(m) rounds with rounds with (1 – 2(1 – 2))¢¢OPT OPT flow. Total running time is flow. Total running time is Õ(mÕ(m22).).

1. Primal-Dual algorithm

2. Combinatorial

3. Multiplicative Weights Update Rule

Edge weights Edge weights wwee = 1 = 1 Repeat until some Repeat until some ee reaches reaches

capacity:capacity: Find shortest pathFind shortest path p p Route Route 22/log(m) /log(m) flow on flow on pp Update Update wwee for all for all e e 22 p p as as

wwee ÃÃ w wee ¢¢ (1 + (1 + )) Output flow.Output flow.

Starting point for our work:Analogous primal-dual algorithms for SDP?

Difficulties: Positive semidefiniteness hard to maintain Rounding algorithms exploit geometric structure

(e.g. negative-type metrics) Matrix operations inefficient to implement

Our work: a generic scheme that yields fast, combinatorial, primal-dual algorithms for various optimization problems using SDP

Our Results: Primal-Dual Our Results: Primal-Dual algorithmsalgorithms

ProblemProblem Previous Previous best:best:

O(√log n) O(√log n) apxapx

Our alg.:Our alg.:

O(√log n) O(√log n) apxapx

Our alg.:Our alg.:

O(log n) O(log n) apxapx

UndirectedUndirected

Sparsest CutSparsest CutÕ(nÕ(n22)) Õ(nÕ(n22)) Õ(m + nÕ(m + n1.51.5))

UndirectedUndirected

Balanced Balanced Sep.Sep.

Õ(nÕ(n22)) Õ(nÕ(n22)) Õ(m + nÕ(m + n1.51.5))

DirectedDirected

Sparsest CutSparsest CutÕ(nÕ(n4.54.5)) Õ(mÕ(m1.51.5 + n + n2+2+)) Õ(mÕ(m1.51.5))

DirectedDirected

Balanced Balanced Sep.Sep.

Õ(nÕ(n4.54.5)) Õ(mÕ(m1.51.5 + n + n2+2+)) Õ(mÕ(m1.51.5))

Min UnCutMin UnCut Õ(nÕ(n4.54.5)) Õ(nÕ(n33)) ------

Min 2CNF Min 2CNF DeletionDeletion

Õ(nÕ(n4.54.5)) Õ(nmÕ(nm1.5 1.5 + n+ n33)) ------

[AHK’04]

[AHK’04]

Our Results: Primal-Dual Our Results: Primal-Dual algorithmsalgorithms

ProblemProblem Previous Previous best:best:

O(√log n) O(√log n) apxapx

Our alg.:Our alg.:

O(√log n) O(√log n) apxapx

Our alg.:Our alg.:

O(log n) O(log n) apxapx

UndirectedUndirected

Sparsest CutSparsest CutÕ(nÕ(n22)) Õ(nÕ(n22)) Õ(m + nÕ(m + n1.51.5))

UndirectedUndirected

Balanced Balanced Sep.Sep.

Õ(nÕ(n22)) Õ(nÕ(n22)) Õ(m + nÕ(m + n1.51.5))

DirectedDirected

Sparsest CutSparsest CutÕ(nÕ(n4.54.5)) Õ(mÕ(m1.51.5 + n + n2+2+)) Õ(mÕ(m1.51.5))

DirectedDirected

Balanced Balanced Sep.Sep.

Õ(nÕ(n4.54.5)) Õ(mÕ(m1.51.5 + n + n2+2+)) Õ(mÕ(m1.51.5))

Min UnCutMin UnCut Õ(nÕ(n4.54.5)) Õ(nÕ(n33)) ------

Min 2CNF Min 2CNF DeletionDeletion

Õ(nÕ(n4.54.5)) Õ(nmÕ(nm1.5 1.5 + n+ n33)) ------

[AHK’04]

[AHK’04] Cannot be achieved by LPs even!

Our Results: Near-linear time Our Results: Near-linear time algorithm for algorithm for Max CutMax Cut

Õ(n + m) Õ(n + m) time algorithm to approximate Max Cut SDP to (1 + o(1)) factor

Previous best: Õ(nÕ(n22) ) time algorithm by Klein, Lu ‘95

Prototype MW for LPs: Prototype MW for LPs: Winnow Winnow [L’88][L’88]

a1 ¢ x ¸ a2 ¢ x ¸

am ¢ x ¸

x ¸ 0i xi = 1

MW for LP:Init: x = (1/n, …, 1/n)Update:xj = xj ¢ exp(- aij)/ = norm. factor

OracleFind i s.t. ai ¢ x < -

x

ai

Thm: Finds x in O(2log(n)/2)iterations.= maxij |aij|

---

000

Convex comb. of constraints allowed:i yi ai ¢ x < -, where yi ¸ 0, i yi = 1. Hence Primal-Dual.

Analysis uses as potential fn.

MW algorithm: boosting, hard-core sets, zero-sum games, flows, portfolio, …

Prototype Matrix MW for SDPsPrototype Matrix MW for SDPs

A1 ² X ¸ A2 ² X ¸

Am ² X ¸

X º 0Tr(X) = 1

Matrix MW for SDP:Init: X = I/nUpdate:X = exp(- t

s=1 Ais)/

= norm. factor

OracleFind it s.t. Ait

² X < -

X

Ait

Thm: Finds X in O(2log(n)/2)iterations.= maxi kAik

---

000

Convex comb. of constraints allowed:i yi Ai²X < -, where yi ¸ 0, i yi = 1. Hence Primal-Dual.

Analysis uses as potential fn.

The Matrix ExponentialThe Matrix Exponential

Matrix MW for SDP:Init: X = I/nUpdate:X = exp(- t

s=1 Ais)/

= norm. factor

Matrix exponential: exp(A) = I + A + A2/2! + A3/3!

+ …

Always PSD:Always PSD: exp(A) exp(A) ºº 0 0

Tricky beast:Tricky beast:

exp(A+B) = exp(A) exp(B)exp(A+B) = exp(A) exp(B)

Computation:Computation: O(nO(n33) ) timetime Usually, can use Usually, can use J-L lemmaJ-L lemma Only need Only need exp(A)v exp(A)v

products:products: Õ(m) Õ(m) timetime

Golden-Thompson inequality:Golden-Thompson inequality:Tr(exp(A+B)) Tr(exp(A+B)) ·· Tr(exp(A)exp(B)) Tr(exp(A)exp(B))

Approximation via SDP Approximation via SDP RelaxationsRelaxations

Input

0-1 QuadraticProgram

SDP:Vectorvars

Reduction

SDP Solver

RoundingAlgorithm

SDP Opt

v1

vn

v2

RelaxationPrimal-Dual SDP

Primal-Dual SDP FrameworkPrimal-Dual SDP Framework

Input

0-1 QuadraticProgram

SDP:Vectorvars

Reduction

RelaxationRoundingAlgorithm

v1

vn

v2

MatrixMWX

y1,…, ym

Primal-Dual SDP

Approximating Balanced Approximating Balanced SeparatorSeparator

Cut Cut (S, S’)(S, S’) is is cc-balanced if -balanced if

|S|, |S’| |S|, |S’| ¸̧ cn cn Min Min cc-Balanced Separator:-Balanced Separator: cc--

balanced cut of min capacitybalanced cut of min capacity

Numerous applications:Numerous applications: Divide-and-conquer Divide-and-conquer

algorithmsalgorithms Markov chainsMarkov chains Geometric embeddingsGeometric embeddings ClusteringClustering Layout problemsLayout problems ……

G = (V, E)G = (V, E)

S

S’

SDP for Balanced SeparatorSDP for Balanced Separator

¼¼kkvvii – v – vjjkk22 = 0 = 0 if if ii, , jj on same on same side,side,

= 1= 1 otherwise otherwise

SDP:SDP:

min min i,j i,j 22 E E ¼ ¼kkvvii – v – vjjkk22

88i: i: kkvviikk22 = 1 = 1

88ijk: ijk: kkvvii–v–vjjkk2 2 + + kkvvjj–v–vkkkk22 ¸̧ kkvvii–v–vkkkk22

i, ji, j ¼ ¼kkvvii – v – vjjkk2 2 ¸̧ c(1-c)n c(1-c)n22

S

S’

G = (V, E)G = (V, E)

vi = -1 vi = 1

Implementing OracleImplementing Oracle

Primal:Primal:

minmin i,j i,j 22 E E ¼ ¼kkvvii – v – vjjkk22

88i: i: kkvviikk22 = 1 = 1

88ijk: ijk: kkvvii–v–vjjkk2 2 + + kkvvjj–v–vkkkk22 ¸̧ kkvvii–v–vkkkk22

i, ji, j ¼ ¼kkvvii – v – vjjkk2 2 ¸̧ c(1-c)n c(1-c)n22

Oracle: given X (thus, vi), : check first and third constraints inequalities: dual vars , multicommodity flow s.t.

1. total flow from any node is · d2. no capacity is exceeded3. i,j fijkvi – vjk2 ¸

fij = total flow between i, j

Bounds eigenvalues via demand graph Laplacian

Thm: Given vi, :1. Max-flow ) desired flow or cut of value O(log(n)¢).2. Multicommodity flow ) desired flow or cut of value O(√log(n)¢).

Conclusions and Future WorkConclusions and Future Work

Matrix MW algorithm:Matrix MW algorithm: more applications in more applications in Solving SDPs:Solving SDPs: e.g. e.g. Min Linear ArrangementMin Linear Arrangement Quantum algorithms:Quantum algorithms: density matrix is a central density matrix is a central

conceptconcept Learning:Learning: e.g. online variance minimization e.g. online variance minimization [WK [WK

COLT’06]COLT’06], online PCA , online PCA [WK NIPS’06][WK NIPS’06] Other applications?Other applications?

LinearLinear time algorithm for time algorithm for Sparsest CutSparsest Cut?? Our algorithm runs in Our algorithm runs in Õ(m + nÕ(m + n1.51.5) ) time for an time for an O(log n) O(log n)

approximationapproximation

Thank you!Thank you!