Download pdf - Parameterized Complexity of Matrix Factorization - IBM · Parameterized Complexity of Matrix Factorization David P. Woodruff IBM Almaden. .. . Based on joint works with Ilya Razenshteyn,

Parameterized Complexity of Matrix Factorization

David P. Woodruff

IBM Almaden

. .

. .

Based on joint works with Ilya Razenshteyn, Zhao Song, Peilin Zhong

Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 1 / 50

Outline

Frobenius norm factorization can be done in polynomial time

Intractable matrix problems

I Nonnegative matrix factorization(NMF)

F Arora-Ge-Kannan-Moitra’12, Moitra’13

I Weighted low rank approximation(WLRA)

F Razenshteyn-Song-W’16

I `1 low rank approximation(`1LRA)

F Song-W-Zhong’16


Outline

Frobenius norm factorization can be done in polynomial timeIntractable matrix problems


F Arora-Ge-Kannan-Moitra’12, Moitra’13




F Song-W-Zhong’16


Outline



F Arora-Ge-Kannan-Moitra’12, Moitra’13I Weighted low rank approximation(WLRA)



F Song-W-Zhong’16


Outline


I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra’12, Moitra’13




F Song-W-Zhong’16


Outline




F Razenshteyn-Song-W’16I `1 low rank approximation(`1LRA)

F Song-W-Zhong’16


Outline



I Weighted low rank approximation(WLRA)F Razenshteyn-Song-W’16


F Song-W-Zhong’16


Outline





F Song-W-Zhong’16


Outline




I `1 low rank approximation(`1LRA)F Song-W-Zhong’16


Frobenius Norm Matrix Factorization

QuestionGiven a matrix A ∈ Rn×n and k > 1, the goal is to output two matricesU ∈ Rn×k ,V ∈ Rk×n such that

‖UV − A‖2F 6 (1 + ε) minrank−k A ′

‖A ′ − A‖2F ,

where ‖A‖2F = (∑n

i=1∑n

j=1 A2i,j)

2, ε ∈ [0,1)

Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries





‖A ′ − A‖2F ,


i=1∑n

j=1 A2i,j)

2, ε ∈ [0,1)

Useful for data compression, easier to store factorizations U,V

Can be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries





‖A ′ − A‖2F ,


i=1∑n

j=1 A2i,j)

2, ε ∈ [0,1)

Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... time

Fails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries





‖A ′ − A‖2F ,


i=1∑n

j=1 A2i,j)

2, ε ∈ [0,1)

Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorization

Not robust to noise, outliers, or missing entries





‖A ′ − A‖2F ,


i=1∑n

j=1 A2i,j)

2, ε ∈ [0,1)

Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries


. .

. .

Nonnegative Matrix Factorization


Example of NMF. .

. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50

Example of NMF. .

. .

Given : A ∈ Rn×n, n = 4, k = 2


Example of NMF. .

. .

Given : A ∈ Rn×n, n = 4, k = 2

A =

0 1 1 11 2 1 12 4 2 23 5 2 2


Example of NMF. .

. .

Given : A ∈ Rn×n, n = 4, k = 2

A =

0 1 1 11 2 1 12 4 2 23 5 2 2

Question : Are there two matrices U,V> ∈ Rn×k , such thatUV = A,U > 0,V > 0


Example of NMF. .

. .

Given : A ∈ Rn×n, n = 4, k = 2

A =

0 1 1 11 2 1 12 4 2 23 5 2 2


=

0 11 12 23 2

· [1 1 0 00 1 1 1

]


Example of NMF. .

. .

Given : A ∈ Rn×n, n = 4, k = 2

A =

0 1 1 11 2 1 12 4 2 23 5 2 2


=

0 11 12 23 2

· [1 1 0 00 1 1 1

]

= UV


Main Question, Nonnegative Matrix Factorization

QuestionGiven matrix A ∈ Rn×n and k > 1, is there an algorithm that candetermine if there exist two matrices U,V> ∈ Rn×k ,

UV = A,U > 0,V > 0.

Or, are there any hardness results?

Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learningApplications

I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.




UV = A,U > 0,V > 0.


Equivalent to computing the nonnegative rank of A, rank+(A)

Fundamental question in machine learningApplications





UV = A,U > 0,V > 0.


Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learning

Applications





UV = A,U > 0,V > 0.







UV = A,U > 0,V > 0.





Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank

I rank(A) 6 rank+(A)I Vavasis’09, determining whether rank+(A) = rank(A) is NP-hard.

.

.



I rank(A) 6 rank+(A)

I Vavasis’09, determining whether rank+(A) = rank(A) is NP-hard.

.

.




.

.




.

.

A =

1 1 0 01 0 1 00 1 0 10 0 1 1




.

.

A =

1 1 0 01 0 1 00 1 0 10 0 1 1

=

1 1 01 0 10 1 00 0 1

·1 0 0 −1

0 1 0 10 0 1 1

rank(A) = 3,




.

.

A =

1 1 0 01 0 1 00 1 0 10 0 1 1

=

1 1 01 0 10 1 00 0 1

·1 0 0 −1

0 1 0 10 0 1 1

rank(A) = 3, but rank+(A) = 4


Polynomial System Verifier

. .

. .



. .

. .[Renegar’92, Basu-Pollack-Roy’96]



. .


Given : a polynomial system P(x) over the real numbers



. .


Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, · · · , xv )



. .



m : #polynomial constraints fi(x) > 0, ∀i ∈ [m]



. .




d : maximum degree of all polynomial constraints



. .




d : maximum degree of all polynomial constraintsH : the bitsizes of the coefficients of the polynomials



. .





In (md)O(v) poly(H) time, candecide if there exists a solution to polynomial system P


Main Idea

. .

. .

1. Write minU,V>∈Rn×k ,U,V>0

‖UV − A‖2F as a polynomial system

that has poly(k) variables and poly(n) constraints and degree

2. Use polynomial system verifier to solve it


Algorithm. .


Algorithm. .

. .

Given : A ∈ Rn×n, k ∈ N


Algorithm. .

. .


Question :


Algorithm. .

. .


Question : Are there matrices U,V> ∈ Rn×k such that


Algorithm. .

. .


Question : Are there matrices U,V> ∈ Rn×k such thatUV = A


Algorithm. .

. .


Question : Are there matrices U,V> ∈ Rn×k such thatUV = AU > 0, V > 0


Algorithm. .

. .



Output :


Algorithm. .

. .



Output : Yes or No


Algorithm. .

. .



Output : Yes or Noin n2O(k)

time Arora-Ge-Kannan-Moitra’12


Algorithm. .

. .



Output : Yes or Noin n2O(k)

time Arora-Ge-Kannan-Moitra’12in 2O(k3)nO(k2) time Moitra’13


k -SUM. .


k -SUM. .

. .

k -SUM


k -SUM. .

. .

k -SUM

Given: a set of n values s1, s2, · · · , sn each in the range [0,1]


k -SUM. .

. .

k -SUM


Determine:


k -SUM. .

. .

k -SUM


Determine: if there is a set of k numbers that sum to exactly k/2


k -SUM. .

. .

k -SUM



k -SUM hardness [Patrascu-Williams’10]


k -SUM. .

. .

k -SUM




Assume: 3-SAT on n variables cannot be solved in 2o(n) time


k -SUM. .

. .

k -SUM




Assume: 3-SAT on n variables cannot be solved in 2o(n) time

then k -SUM cannot be solved in no(k) time


Hardness - Under Exponential Time Hypothesis. .



. .




. .


Output : Are there two matrices U,V> ∈ Rn×k , such thatUV = AU > 0,V > 0



. .



Assume : Exponential Time Hypothesis[Impagliazzo-Paturi-Zane’98]



. .



Assume : Exponential Time Hypothesis[Impagliazzo-Paturi-Zane’98]

Requires : nΩ(k) time


Open Problems

The upper bound is nO(k2) while the lower bound is nΩ(k) - what isthe right answer?


. .

. .

Weighted Low Rank Approximation


Weighted Low Rank Approximation. .



. .

Given : A ∈ Rn×n, W ∈ Rn×n, k ∈ N, ε > 0



. .


Output :



. .


Output : rank-k A ∈ Rn×n s.t.

AW A )− (

2

F



. .


Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε) min

rank−k A ′‖W (A ′ − A)‖2F

AW A )− (

2

F



. .




.

∑i,j

W 2i,j(A

′i,j − Ai,j)

2

AW A )− (

2

F



. .




.

∑i,j

W 2i,j(A

′i,j − Ai,j)

2

.OPT

AW A )− (

2

F



. .


Output :

..

U,V> ∈ Rn×k s.t.‖W (UV − A)‖2F 6 (1 + ε)OPT

U

V

W A )− (

2

F


Motivation. .


Motivation. .


Motivation. .


Motivation. .

. .

Action Comedy Historical Cartoon Magical


Motivation. .

. .


.

9 8


Motivation. .

. .


..


Motivation. .

. .


...


Motivation. .

. .


....


Motivation. .

. .


.....

15

15

11

11

11

11

15

15

11

11

14

14

16

16

11

11


Motivation. .

. .


.....

15

15

11

11

11

11

15

15

11

11

14

14

16

16

11

11

.

11

11

11

11

11

11

16

16

13

13

12

12

12

12

14

14


Motivation. .

. .


.....

15

15

11

11

11

11

15

15

11

11

14

14

16

16

11

11

.

11

11

11

11

11

11

16

16

13

13

12

12

12

12

14

14

.

13

13

16

16

14

14

12

12

17

17

15

15

14

14

11

11


Motivation. .

. .


.....

15

15

11

11

11

11

15

15

11

11

14

14

16

16

11

11

.

11

11

11

11

11

11

16

16

13

13

12

12

12

12

14

14

.

13

13

16

16

14

14

12

12

17

17

15

15

14

14

11

11

.

12

12

11

11

12

12

11

11

15

15

11

11

13

13

14

14


Motivation. .

. .


.....

15

15

11

11

11

11

15

15

11

11

14

14

16

16

11

11

.

11

11

11

11

11

11

16

16

13

13

12

12

12

12

14

14

.

13

13

16

16

14

14

12

12

17

17

15

15

14

14

11

11

.

12

12

11

11

12

12

11

11

15

15

11

11

13

13

14

14

.

12

12

12

12

16

16

14

14

12

12

13

13

16

16

12

12


Motivation. .

. .


..........


Motivation. .

. .


...........

8 8

8 8


Motivation. .

. .


...........

8 8

8 8

.

4 4

4 4


Motivation. .

. .


...........

8 8

8 8

.

4 4

4 4

.

1 1

1 1


Motivation. .

. .


...........

8 8

8 8

.

4 4

4 4

.

1 1

1 1

.

0 0

0 0


Motivation. .

. .


...........

8 8

8 8

.

4 4

4 4

.

1 1

1 1

.

0 0

0 0

.

8 8

8 8


Motivation. .

. .


...........

8 8

8 8

.

4 4

4 4

.

1 1

1 1

.

0 0

0 0

.

8 8

8 8

.

4 4

4 4


Motivation. .

. .


...........

8 8

8 8

.

4 4

4 4

.

1 1

1 1

.

0 0

0 0

.

8 8

8 8

.

4 4

4 4

.

2 2

2 2


Motivation. .

. .


...........

8 8

8 8

.

4 4

4 4

.

1 1

1 1

.

0 0

0 0

.

8 8

8 8

.

4 4

4 4

.

2 2

2 2

.

1 1

1 1


Matrix Completion. .



. .

Given : A ∈ R, ?n×n, Ω ⊂ [n]× [n] and k ∈ R



. .


A =

1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1

, k = 2



. .


A =

1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1

, k = 2

Output : AΩ s.t. rank(A) = k



. .


A =

1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1

, k = 2


.

A =

1 1 0 01 1 0 00 0 1 10 0 1 1

, A =

1 0 1 00 1 0 11 0 1 00 1 0 1

.



. .


A =

1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1

, k = 2


.

Algorithms : Candes-Recht’09, Candes-Recht’10, Keshavan’12Hardt-Wootters’14, Jain-Netrapalli-Sanghavi’13Hardt’15, Sun-Luo’15



. .


A =

1 ? ? 0? 1 0 ?? 0 1 ?0 ? ? 1

, k = 2


.

Algorithms : Candes-Recht’09, Candes-Recht’10, Keshavan’12Hardt-Wootters’14, Jain-Netrapalli-Sanghavi’13Hardt’15, Sun-Luo’15

Hardness : Peeters’96, Gillis-Glineur’11Hardt-Meka-Raghavendra-Weitz’14


More Real-life Datasets. .

. .

RateBeer

Documents

Amazon

Biology



. .

RateBeer

Tasters Beers Ratings

Documents

Amazon

Biology



. .

RateBeer


Documents

Words Books Frequency

Amazon

Biology



. .

RateBeer


Documents


Amazon

Users Products Ratings

Biology



. .

RateBeer


Documents


Amazon

Users Products Ratings

Biology

Genes Cells Levels


Main Results - Summary

Algorithm for Weighted low rank approximation(WLRA) problem

I W has r distinct rows and columnsI W has r distinct columnsI W has rank at most r

Hardness for Weighted low rank approximation(WLRA) problem

.

.



Algorithm for Weighted low rank approximation(WLRA) problemI W has r distinct rows and columns

I W has r distinct columnsI W has rank at most r


.

.

W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3



Algorithm for Weighted low rank approximation(WLRA) problemI W has r distinct rows and columnsI W has r distinct columns

I W has rank at most r


.

.

W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3



Algorithm for Weighted low rank approximation(WLRA) problemI W has r distinct rows and columnsI W has r distinct columnsI W has rank at most r


.

.

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3



Algorithm for Weighted low rank approximation(WLRA) problemI W has r distinct rows and columnsI W has r distinct columnsI W has rank at most r

Hardness for Weighted low rank approximation(WLRA) problem.

.


r Distinct Rows and Columns. .



. .

Given : A ∈ Rn×n, W ∈ Rn×n, r ∈ N, k ∈ N, ε > 0



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3

Output :



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3




. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3


rank−k A ′‖W (A − A ′)‖2F



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3


rank−k A ′

.

∑i,j

W 2i,j(Ai,j − A ′i,j)

2



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3

Output : rank-k A ∈ Rn×n s.t.‖W (A − A)‖2F 6 (1 + ε)

..

OPT



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3


..

OPTwith prob. 9/10



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3


..

OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r/ε) time



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3


..

OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r/ε) timefor an arbitrarily small constant γ > 0



. .


W =

1 1 1 3 3 41 1 1 3 3 42 2 2 5 5 02 2 2 5 5 02 2 2 5 5 07 7 7 4 4 6

, r = 3


..

OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r/ε) timefor an arbitrarily small constant γ > 0fixed parameter tractable


r Distinct Columns. .



. .




. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3



. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3

Output :



. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3




. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3





. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3


rank−k A ′

.

∑i,j


2



. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3


..

OPT



. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3


..

OPTwith prob. 9/10



. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3


..

OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r2/ε) time



. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3


..

OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r2/ε) timefor an arbitrarily small constant γ > 0



. .


W =

1 1 1 1 1 12 2 2 1 1 33 3 3 1 1 54 4 4 1 1 15 5 5 1 1 36 6 6 1 1 5

, r = 3


..

OPTwith prob. 9/10in O((nnz(A) + nnz(W )) · nγ) + n · 2O(k2r2/ε) timefor an arbitrarily small constant γ > 0fixed parameter tractable


Rank r. .


Rank r. .

. .



Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3


Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3

Output :


Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3



Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3




Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3


rank−k A ′

.

∑i,j


2


Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3


..

OPT


Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3


..

OPTwith prob. 9/10


Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3


..

OPTwith prob. 9/10

in nO(k2r/ε) time


Rank r. .

. .


W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

, r = 3


..

OPTwith prob. 9/10

in nO(k2r/ε) timepreviously only r=1 was known to be in polynomial time


Random-4SAT Hypothesis. .



. .

[Feige’02, Goerdt-Lanka’04]



. .


Given : random 4-SAT formula S on n variables



. .


Given : random 4-SAT formula S on n variableseach clause has 4 literals



. .


Given : random 4-SAT formula S on n variableseach clause has 4 literalseach of Θ(n4) clauses is picked ind. with prob. Θ(1/n3)m = Θ(n) is the number of clauses

(x4 ∨ x32 ∨ x9 ∨ x20)S1



. .



Any algorithm that :

(x4 ∨ x32 ∨ x9 ∨ x20)S1

(x2 ∨ x13 ∨ x5 ∨ x25)S2

· · · · · ·(x11 ∨ x19 ∨ x24 ∨ x6)Sm

S = S1 ∧ S2 ∧ · · ·∧ Sm



. .



Any algorithm that :outputs 1 with prob. 1 when S is satisfiableoutputs 0 with prob. > 1/2probability is over the input instances

(x4 ∨ x32 ∨ x9 ∨ x20)S1

(x2 ∨ x13 ∨ x5 ∨ x25)S2

· · · · · ·(x11 ∨ x19 ∨ x24 ∨ x6)Sm

S = S1 ∧ S2 ∧ · · ·∧ Sm



. .



Any algorithm that :outputs 1 with prob. 1 when S is satisfiableoutputs 0 with prob. > 1/2probability is over the input instances

Requires : 2Ω(n) time

(x4 ∨ x32 ∨ x9 ∨ x20)S1

(x2 ∨ x13 ∨ x5 ∨ x25)S2

· · · · · ·(x11 ∨ x19 ∨ x24 ∨ x6)Sm

S = S1 ∧ S2 ∧ · · ·∧ Sm


Maximum Edge Biclique(MEB). .



. .

Given : A bipartite graph G = (U,V ,E) with |U | = |V | = n



. .


Output : A k1-by-k2 complete bipartite subgraph of G



. .


Output : A k1-by-k2 complete bipartite subgraph of Gs.t. k1 · k2 is maximized



. .





. .





. .





. .





. .

[Goerdt-Lanka’04]



. .

[Goerdt-Lanka’04]

Assume : Random 4-SAT hardness hypothesis



. .

[Goerdt-Lanka’04]


there exist two constants ε1 > ε2 > 0 such that



. .

[Goerdt-Lanka’04]



Any algorithm that : distinguishes between

bipartite graphs G = (U,V ,E) with |U | = |V | = n in two cases



. .

[Goerdt-Lanka’04]





1. there is a clique of size > (n/16)2(1 + ε1)



. .

[Goerdt-Lanka’04]






2. all bipartite cliques are of size 6 (n/16)2(1 + ε2)



. .

[Goerdt-Lanka’04]







Requires :



. .

[Goerdt-Lanka’04]







Requires : 2Ω(n) time


Reduction. .


Reduction. .

. .

MEB


Reduction. .

. .

MEB Given : U,V with |U | = |V | = n


Reduction. .

. .

MEB Given : U,V with |U | = |V | = nOutput : a biclique s.t. k1k2 is maximized


Reduction. .

. .


( complement ) |E |− k1k2 is minimized


Reduction. .

. .



WLRA


Reduction. .

. .



WLRA Given : A ∈ 0,1n×n,W ∈ Rn×n


Reduction. .

. .




Output : u, v ∈ Rn s.t. ‖W (uv> − A)‖2F is minimized


Reduction. .

. .






Reduction. .

. .





Ai,j =

10

if edge (Ui ,Vj) exists

otherwise


Reduction. .

. .





Ai,j =

10

if edge (Ui ,Vj) exists

otherwiseW i,j =

1n6


Main Results - Hardness. .



. .

Weighted Low Rank Approximation Hardness



. .


Given : A ∈ Rn×n, distinct r columns W ∈ Rn×n



. .



k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)



. .



k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)

Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTwith prob. 9/10



. .



k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)


Assume : Random-4SAT Hypothesis[Feige’02, Goerdt-Lanka’04]



. .



k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)



∃ε0 > 0, for any algorithm with ε < ε0 and k > 1



. .



k ∈ N, ε > 0,W ij ∈ 0,1,2, · · · ,poly(n)



∃ε0 > 0, for any algorithm with ε < ε0 and k > 1Requires : 2Ω(r) time


Algorithmic Techniques

Tools

I Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch

Algorithm, assuming a rank r weight matrix W

I Warmup, inefficient WLRA algorithmI “Guessing a sketch”



ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]

I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch





ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]

I Multiple regression sketchAlgorithm, assuming a rank r weight matrix W




ToolsI Polynomial system verifier [Renegar’92, Basu-Pollack-Roy’96]I Lower bound on the cost [Jeronimo-Perrucci-Tsigaridas’13]I Multiple regression sketch











Algorithm, assuming a rank r weight matrix WI Warmup, inefficient WLRA algorithm

I “Guessing a sketch”




Algorithm, assuming a rank r weight matrix WI Warmup, inefficient WLRA algorithmI “Guessing a sketch”


Polynomial System Verifier (Recall)

. .

. .



. .




. .


Given : a real polynomial system P(x)



. .


Given : a real polynomial system P(x)v : #variables, x = (x1, x2, · · · , xv )



. .






. .




d : maximum degree of all polynomial constraints



. .







. .





It takes (md)O(v) poly(H) time todecide if there exists a solution to polynomial system P


Lower Bound on the Cost

. .

. .



. .

. .[Jeronimo-Perrucci-Tsigaridas’13]



. .


Given: a real polynomial system P(x)



. .



v : #variables, x = (x1, x2, · · · , xv )



. .




m : #polynomials constraints fi(x) > 0, ∀i ∈ [m]



. .





d : maximum degree of all polynomials



. .





d : maximum degree of all polynomialsH : the bitsizes of the coefficients of the polynomials



. .






.

Rv



. .






.

Rv

.

T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0

T



. .






.

Rv

.

T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0

T

.

minimum value that nonnegative g takes over T ∩ Ball is



. .






.

Rv

.

T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0

T

.


Ball



. .






.

Rv

.

T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0

T

.


Ball

either 0 or > (2H + m)−dv



. .






.

Rv

.

T : x ∈ Rv |f1(x) > 0, · · · , fm(x) > 0

T

.


Ball

either 0 or > (2H + m)−dv

g = (x1x2 − 1)2 + x22


Multiple Regression Sketch. .



. .

Given :



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1

Let x(j) = arg minx∈Rk×1

‖A(j)x − b(j)‖,∀j ∈ [m]



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1


‖A(j)x − b(j)‖,∀j ∈ [m]

Choose : S to be a random Gaussian matrix



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1


‖A(j)x − b(j)‖,∀j ∈ [m]

Choose : S to be a random Gaussian matrixDenote y(j) = arg min

y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1


‖A(j)x − b(j)‖,∀j ∈ [m]


y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]

Gaurantee :



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1


‖A(j)x − b(j)‖,∀j ∈ [m]


y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]

Gaurantee : For all ε ∈ (0,1/2), one can set t = O(k/ε) such that



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1


‖A(j)x − b(j)‖,∀j ∈ [m]


y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]

Gaurantee : For all ε ∈ (0,1/2), one can set t = O(k/ε) such thatm∑

j=1‖A(j)y(j) − b(j)‖22 6 (1 + ε)

m∑j=1‖A(j)x(j) − b(j)‖22



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1


‖A(j)x − b(j)‖,∀j ∈ [m]


y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]


j=1‖A(j)y(j) − b(j)‖22 6 (1 + ε)

m∑j=1‖A(j)x(j) − b(j)‖22

with prob. 9/10



. .

Given : A(1),A(2), · · · ,A(m) ∈ Rn×k

b(1),b(2), · · · ,b(m) ∈ Rn×1


‖A(j)x − b(j)‖,∀j ∈ [m]


y∈Rk×1‖SA(j)y − Sb(j)‖, ∀j ∈ [m]


j=1‖A(j)y(j) − b(j)‖22 6 (1 + ε)

m∑j=1‖A(j)x(j) − b(j)‖22

with prob. 9/10


Warmup, inefficient WLRA Algorithm. .



. .




. .


.

Aij ∈ 0,±1,±2, · · · ,±∆



. .


..

W ij ∈ 0,1,2, · · · ,∆



. .


..

Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPT



. .


..

Output : rank-k A s.t. ‖W (A − A)‖2F 6 (1 + ε)OPTAlgorithm :



. .


..


1. create 2nk variables for U,V> ∈ Rn×k



. .


..



2. write polynomial g(x1, · · · , x2nk ) = ‖W (A − UV )‖2F



. .


..



2. write polynomial g(x1, · · · , x2nk ) = ‖W (A − UV )‖2F3. pick C ∈ [L−,L+], run polynomial verifier g(x) 6 C



. .


..



2. write polynomial g(x1, · · · , x2nk ) = ‖W (A − UV )‖2F3. pick C ∈ [L−,L+], run polynomial verifier g(x) 6 C4. optimize C by binary search over [L−,L+]



. .


..




Time : 2Ω(nk)



. .


..




Time : 2Ω(nk)

How can we do better?



. .


..



Time : 2Ω(nk)

How can we do better?



. .


..



Time : 2Ω(nk)

How can we do better?polynomial verifier runs in (# constraints · degree)O(# variables)



. .


..



Time : 2Ω(nk)


lower bound on cost (# constraints)−degreeO(# variables)



. .


..



Time : 2Ω(nk)



write a polynomial with few #variables, i.e. poly(kr/ε)



. .


..



Time : 2Ω(nk)



write a polynomial with few #variables, i.e. poly(kr/ε)without blowing up degree and #constraints too much


Main Idea

. .

. .

To reduce the number of variables to poly(kr/ε) :

1. Multiple regression sketch with O(k/ε) rows

2. Weight matrix W has rank at most r


Guess a Sketch. .


Guess a Sketch. .

. .



Guess a Sketch. .

. .


.

Aij ∈ 0,±1,±2, · · · ,±∆


Guess a Sketch. .

. .


..

W ij ∈ 0,1,2, · · · ,∆


Guess a Sketch. .

. .


..

Output : U,V> ∈ Rn×k s.t. ‖W (UV − A)‖2F 6 (1 + ε)OPT


Guess a Sketch. .

. .


..


Algorithm : W j be j th column of WDW j be diagonal matrix with vector W j

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6


Guess a Sketch. .

. .


..



.

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

DW 1 =

10

10

10


Guess a Sketch. .

. .


..



..

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

DW 2 =

11

10

00


Guess a Sketch. .

. .


..



...

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

DW 3 =

12

34

56


Guess a Sketch. .

. .


..



....

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

DW 4 =

21

20

10


Guess a Sketch. .

. .


..



.....

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

DW 5 =

22

44

66


Guess a Sketch. .

. .


..



......

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

DW 6 =

23

44

56


Guess a Sketch. .

. .


..


Algorithm :

......

.

U

V

W A )− (

2

F


Guess a Sketch. .

. .


..


Algorithm :

......

.

U

V

W A )− (

2

F


Guess a Sketch. .

. .


..


Algorithm :

......

.n∑j=1‖DW j UV j − DW j Aj‖22

U

V

W A )− (

2

F


Guess a Sketch. .

. .


..


Algorithm :

......


U

V

W A )− (

2

F


Guess a Sketch. .

. .


..


Algorithm :

......


n∑i=1‖U iVDW i − AiDW i‖22

U

V

W A )− (

2

F


Guess a Sketch. .

. .


..


Algorithm :

......



Sketch by Gaussian S,T> ∈ Rt×n

S T


Guess a Sketch. .

. .


..


Algorithm :

......



Sketch by Gaussian S,T> ∈ Rt×nn∑

j=1‖SDW j UV j − SDW j Aj‖22


Guess a Sketch. .

. .


..


Algorithm :

......





n∑i=1‖U iVDW i T − AiDW i T‖22


Guess a Sketch. .

. .


..


Algorithm :

......






Guess SDW j U ∈ Rt×k and VDW i T ∈ Rk×t


Guess a Sketch. .

. .


..


Algorithm :

......







.

SDW 1U = S

U

10

10

10


Guess a Sketch. .

. .


..


Algorithm :

......







..

SDW 2U = S

U

11

10

00


Guess a Sketch. .

. .


..


Algorithm :

......







...

SDW 3U = S

U

12

34

56


Guess a Sketch. .

. .


..


Algorithm :

......







....

SDW 4U = S

U

21

20

10


Guess a Sketch. .

. .


..


Algorithm :

......







.....

SDW 5U = S

U

22

44

66


Guess a Sketch. .

. .


..


Algorithm :

......







......

SDW 6U = S

U

23

44

56


Guess a Sketch. .

. .


..


Algorithm :

......







......

Create t × k variables for each of n SDW j U s?


Guess a Sketch. .

. .


..


Algorithm :

......







......


.

×t × kn


Guess a Sketch. .

. .


..


Algorithm :

......







......


.

×t × k

.

?


Guess a Sketch. .

. .


..


Algorithm :

......







......


.

×t × k

..

r


Guess a Sketch. .

. .



Algorithm : W j be j th column of W

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6


Guess a Sketch. .

. .



Algorithm : W j be j th column of W.

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

1 1 10 1 21 1 30 0 41 0 50 0 6

column span of W


Guess a Sketch. .

. .



Algorithm : W j be j th column of W..

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

101010

=

101010

W 1 = W 1


Guess a Sketch. .

. .



Algorithm : W j be j th column of W...

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

111000

=

111000

W 2 = W 2


Guess a Sketch. .

. .



Algorithm : W j be j th column of W....

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

123456

=

123456

W 3 = W 3


Guess a Sketch. .

. .



Algorithm : W j be j th column of W.....

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

212010

= +

101010

111000

W 4 = W 1 + W 2


Guess a Sketch. .

. .



Algorithm : W j be j th column of W......

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

224466

= +

101010

123456

W 5 = W 1 + W 3


Guess a Sketch. .

. .



Algorithm : W j be j th column of W.......

W =

1 1 1 2 2 20 1 2 1 2 31 1 3 2 4 40 0 4 0 4 41 0 5 1 6 50 0 6 0 6 6

234456

= +

111000

123456

W 6 = W 2 + W 3


Guess a Sketch. .

. .



Algorithm :....... n∑

j=1‖DW j UV j − DW j Aj‖22


Guess a Sketch. .

. .





sketchn∑


1 + ε


Guess a Sketch. .

. .





sketchn∑


1 + ε

create variables for SDW j U, ∀j ∈ [r ]


Guess a Sketch. .

. .





sketchn∑


1 + ε


.

S

U

10

10

10

create t × k variables for SDW 1U


Guess a Sketch. .

. .





sketchn∑


1 + ε


..

S

U

11

10

00



Guess a Sketch. .

. .





sketchn∑


1 + ε


...

S

U

12

34

56



Guess a Sketch. .

. .





sketchn∑


1 + ε


....

S

U

21

20

10

write SDW 4U as SDW 1U + SDW 2U


Guess a Sketch. .

. .





sketchn∑


1 + ε


.....

S

U

1 + 10 + 1

1 + 10 + 0

1 + 00 + 0



Guess a Sketch. .

. .





sketchn∑


1 + ε


......

S

U(

10

10

10

+

11

10

00

)



Guess a Sketch. .

. .





sketchn∑


1 + ε


.......

S

U

22

44

66



Guess a Sketch. .

. .





sketchn∑


1 + ε


........

S

U

1 + 10 + 2

1 + 30 + 4

1 + 50 + 6



Guess a Sketch. .

. .





sketchn∑


1 + ε


.........

S

U(

10

10

10

+

12

34

56

)



Guess a Sketch. .

. .





sketchn∑


1 + ε


.........

S

U

23

44

56



Guess a Sketch. .

. .





sketchn∑


1 + ε


.........

S

U

1 + 11 + 2

1 + 30 + 4

0 + 50 + 6



Guess a Sketch. .

. .





sketchn∑


1 + ε


.........

S

U(

11

10

00

+

12

34

56

)



Wrapping Up

Can use an explicit formula for the regression solution in terms ofthe variables created

The regression solution is a rational function, but can use tricks toclear the denominatorMultiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variablesTime : nO(rk2/ε)

(# constraints · degree)O(# variables) = (O(n) ·O(nk))O(krt)


Wrapping Up

Can use an explicit formula for the regression solution in terms ofthe variables createdThe regression solution is a rational function, but can use tricks toclear the denominator

Multiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variablesTime : nO(rk2/ε)



Wrapping Up

Can use an explicit formula for the regression solution in terms ofthe variables createdThe regression solution is a rational function, but can use tricks toclear the denominatorMultiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variables

Time : nO(rk2/ε)



Wrapping Up

Can use an explicit formula for the regression solution in terms ofthe variables createdThe regression solution is a rational function, but can use tricks toclear the denominatorMultiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variablesTime : nO(rk2/ε)



Wrapping Up

Can use an explicit formula for the regression solution in terms ofthe variables createdThe regression solution is a rational function, but can use tricks toclear the denominatorMultiple Regression Sketch + Bounded Rank Weight Matrix implya small number of variablesTime : nO(rk2/ε)



Open Problems

For a rank-r weight matrix W , the upper bound is nO(k2r/ε) but thelower bound is only 2Ω(r) - can we close this gap?

Can we prove a hardness result with respect to the parameter k ,e.g., a 2Ω(k) lower bound for WLRA problem?


Open Problems

For a rank-r weight matrix W , the upper bound is nO(k2r/ε) but thelower bound is only 2Ω(r) - can we close this gap?Can we prove a hardness result with respect to the parameter k ,e.g., a 2Ω(k) lower bound for WLRA problem?


. .

. .

`1 Low Rank Approximation


`1-Low Rank Approximation. .



. .

Given : A ∈ Rn×n, k ∈ N, α > 1



. .


Output :



. .



A A−

1



. .


Output : rank-k A ∈ Rn×n s.t.‖A − A‖1 6 α · min

rank−k A ′‖A ′ − A‖1

A A−

1



. .




.

∑i,j|A ′i,j − Ai,j |

A A−

1



. .




.

∑i,j|A ′i,j − Ai,j |

.OPT

A A−

1



. .


Output :

..

U ∈ Rn×k ,V ∈ Rk×n s.t.‖UV − A‖1 6 α · OPT

U

V

A−

1


Why is `1-low Rank Approximation Interesting?

The problem was shown to be NP-hard by Gillis-Vavasis’15 and anumber of heuristics have been proposed

Asked in several places if there are any approximation algorithms


Why is `1-low Rank Approximation Interesting?

The problem was shown to be NP-hard by Gillis-Vavasis’15 and anumber of heuristics have been proposedAsked in several places if there are any approximation algorithms


Thought Experiments. .



. .

Given : A set of points in 2-dimensions



. .


Goal : Find a line to fit those points



. .





. .



.



. .



.

Output : `2 minimizer line,



. .



.

Output : `2 minimizer line, `1 minimizer line





. .

Given : k = 1 and A =

4 0 0 00 1 1 10 1 1 10 1 1 1



. .


4 0 0 00 1 1 10 1 1 10 1 1 1

Output : rank-1 ‖‖F−solution A =

4 0 0 00 0 0 00 0 0 00 0 0 0



. .


4 0 0 00 1 1 10 1 1 10 1 1 1


4 0 0 00 0 0 00 0 0 00 0 0 0



. .


4 0 0 00 1 1 10 1 1 10 1 1 1


4 0 0 00 0 0 00 0 0 00 0 0 0

Output : rank-1 ‖‖1−solution A =

0 0 0 00 1 1 10 1 1 10 1 1 1



. .


4 0 0 00 1 1 10 1 1 10 1 1 1


4 0 0 00 0 0 00 0 0 00 0 0 0

Output : rank-1 ‖‖1−solution A =

0 0 0 00 1 1 10 1 1 10 1 1 1


Heuristics and Nearly `1 Setting

`1-setting

I Heuristic algorithms Ke-Kanade’05, Ding-Zhou-He-Zha’06,Kwak’08, Brooks-Dula-Boone’13

I Robust PCA Candes-Li-Ma-Wright’11,Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16

None of them can achieve better than poly(n) approximation ratio



`1-settingI Heuristic algorithms Ke-Kanade’05, Ding-Zhou-He-Zha’06,

Kwak’08, Brooks-Dula-Boone’13

I Robust PCA Candes-Li-Ma-Wright’11,Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16





Kwak’08, Brooks-Dula-Boone’13I Robust PCA Candes-Li-Ma-Wright’11,

Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16





Kwak’08, Brooks-Dula-Boone’13I Robust PCA Candes-Li-Ma-Wright’11,

Wright-Ganesh-Rao-Peng-Ma’09,Netrapalli-Niranjan-Sanghavi-Anandkumar-Jain’14,Yi-Park-Chen-Caramanis’16



Main Question, `1 Low Rank Approximation

QuestionGiven matrix A ∈ Rn×n, is there an algorithm that is able to output arank-k matrix A such that

‖A − A‖1 6 α · minrank−k A ′

‖A ′ − A‖1?

Or, are there any inapproximability results?


Main Results - Algorithms

Upper bound (Algorithm) for an arbitrary n × n matrix A

I poly(k) log n-approximation, poly(n) timeI O(k)-approximation, nO(k)2O(k2) time.I Bicriteria algorithm

F rank-2k O(1)-approximation, nO(k2) running time



Upper bound (Algorithm) for an arbitrary n × n matrix AI poly(k) log n-approximation, poly(n) time

I O(k)-approximation, nO(k)2O(k2) time.I Bicriteria algorithm




Upper bound (Algorithm) for an arbitrary n × n matrix AI poly(k) log n-approximation, poly(n) timeI O(k)-approximation, nO(k)2O(k2) time.

I Bicriteria algorithm




Upper bound (Algorithm) for an arbitrary n × n matrix AI poly(k) log n-approximation, poly(n) timeI O(k)-approximation, nO(k)2O(k2) time.I Bicriteria algorithm




Upper bound (Algorithm) for an arbitrary n × n matrix AI poly(k) log n-approximation, poly(n) timeI O(k)-approximation, nO(k)2O(k2) time.I Bicriteria algorithm






. .

`1-Low Rank Approximation Hardness



. .





. .



Output : U ∈ Rn×k , V ∈ Rk×n s.t. ‖UV − A‖1 6 α ·OPTwith prob. 9/10



. .




Assume : Exponential Time Hypothesis(ETH)[Impagliazzo-Paturi-Zane’98]



. .





for arbitrarily small constant γ > 0



. .





for arbitrarily small constant γ > 0for any algorithm running in nO(1) time



. .






Requires : α > (1 + 1log1+γ n

)



. .






Requires : α > (1 + 1log1+γ n

)

Implies a 2Ω(1/ε1−γ) Hardness


Open Problems

The hardness result is for (1 + 1/ poly(log n)) approximation, canwe obtain a hardness result for O(1) approximation ?

We have an nO(k)-time O(k)-approximation algorithm, can eitherthe approximation or the time be improved?We have a poly(n)-time O(log n) · poly(k)-approximationalgorithm. Can the approximation be improved?


Open Problems


We have an nO(k)-time O(k)-approximation algorithm, can eitherthe approximation or the time be improved?

We have a poly(n)-time O(log n) · poly(k)-approximationalgorithm. Can the approximation be improved?


Open Problems


We have an nO(k)-time O(k)-approximation algorithm, can eitherthe approximation or the time be improved?We have a poly(n)-time O(log n) · poly(k)-approximationalgorithm. Can the approximation be improved?


Conclusions

Studied intractable matrix factorization problems through the lensof parameterized complexity

I nonnegative matrix factorizationI weighted low rank approximationI `1-low rank approximation

Algorithm techniques

I polynomial system verifierI random matrices for dimensionality reduction to reduce the number

of variables

Hardness techniques

I random 4-SAT, ETH, etc.

Parameterized Complexity gives a way of coping with intractabilityfor emerging machine learning problems


Conclusions


I nonnegative matrix factorization

I weighted low rank approximationI `1-low rank approximation



of variables

Hardness techniques




Conclusions


I nonnegative matrix factorizationI weighted low rank approximation

I `1-low rank approximationAlgorithm techniques


of variables

Hardness techniques




Conclusions





of variables

Hardness techniques




Conclusions





of variablesHardness techniques




Conclusions



Algorithm techniquesI polynomial system verifier

I random matrices for dimensionality reduction to reduce the numberof variables

Hardness techniques




Conclusions



Algorithm techniquesI polynomial system verifierI random matrices for dimensionality reduction to reduce the number

of variables

Hardness techniques




Conclusions








Conclusions








Conclusions








Thank you!. .

. .

Questions?


Cauchy Distribution. .

. .

Cauchy distribution C(µ,γ), pdf f (x) = 1πγ

γ2

(x−µ)2+γ

µ

1π

For any three independent random variablesX ,Y ,Z drawn from Cauchy distribution.For any a,b ∈ R, aX + bY ∼ (|a|+ |b|)Z
