Parameterized Complexity of Matrix Factorization - IBM ?· Parameterized Complexity of Matrix Factorization…

  • Published on
    18-Aug-2018

  • View
    212

  • Download
    0

Embed Size (px)

Transcript

<ul><li><p>Parameterized Complexity of Matrix Factorization</p><p>David P. Woodruff</p><p>IBM Almaden</p><p>. .</p><p>. .</p><p>Based on joint works with Ilya Razenshteyn, Zhao Song, Peilin Zhong</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 1 / 50</p></li><li><p>Outline</p><p>Frobenius norm factorization can be done in polynomial time</p><p>Intractable matrix problems</p><p>I Nonnegative matrix factorization(NMF)</p><p>F Arora-Ge-Kannan-Moitra12, Moitra13</p><p>I Weighted low rank approximation(WLRA)</p><p>F Razenshteyn-Song-W16</p><p>I `1 low rank approximation(`1LRA)</p><p>F Song-W-Zhong16</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50</p></li><li><p>Outline</p><p>Frobenius norm factorization can be done in polynomial timeIntractable matrix problems</p><p>I Nonnegative matrix factorization(NMF)</p><p>F Arora-Ge-Kannan-Moitra12, Moitra13</p><p>I Weighted low rank approximation(WLRA)</p><p>F Razenshteyn-Song-W16</p><p>I `1 low rank approximation(`1LRA)</p><p>F Song-W-Zhong16</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50</p></li><li><p>Outline</p><p>Frobenius norm factorization can be done in polynomial timeIntractable matrix problems</p><p>I Nonnegative matrix factorization(NMF)</p><p>F Arora-Ge-Kannan-Moitra12, Moitra13I Weighted low rank approximation(WLRA)</p><p>F Razenshteyn-Song-W16</p><p>I `1 low rank approximation(`1LRA)</p><p>F Song-W-Zhong16</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50</p></li><li><p>Outline</p><p>Frobenius norm factorization can be done in polynomial timeIntractable matrix problems</p><p>I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra12, Moitra13</p><p>I Weighted low rank approximation(WLRA)</p><p>F Razenshteyn-Song-W16</p><p>I `1 low rank approximation(`1LRA)</p><p>F Song-W-Zhong16</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50</p></li><li><p>Outline</p><p>Frobenius norm factorization can be done in polynomial timeIntractable matrix problems</p><p>I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra12, Moitra13</p><p>I Weighted low rank approximation(WLRA)</p><p>F Razenshteyn-Song-W16I `1 low rank approximation(`1LRA)</p><p>F Song-W-Zhong16</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50</p></li><li><p>Outline</p><p>Frobenius norm factorization can be done in polynomial timeIntractable matrix problems</p><p>I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra12, Moitra13</p><p>I Weighted low rank approximation(WLRA)F Razenshteyn-Song-W16</p><p>I `1 low rank approximation(`1LRA)</p><p>F Song-W-Zhong16</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50</p></li><li><p>Outline</p><p>Frobenius norm factorization can be done in polynomial timeIntractable matrix problems</p><p>I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra12, Moitra13</p><p>I Weighted low rank approximation(WLRA)F Razenshteyn-Song-W16</p><p>I `1 low rank approximation(`1LRA)</p><p>F Song-W-Zhong16</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50</p></li><li><p>Outline</p><p>Frobenius norm factorization can be done in polynomial timeIntractable matrix problems</p><p>I Nonnegative matrix factorization(NMF)F Arora-Ge-Kannan-Moitra12, Moitra13</p><p>I Weighted low rank approximation(WLRA)F Razenshteyn-Song-W16</p><p>I `1 low rank approximation(`1LRA)F Song-W-Zhong16</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 2 / 50</p></li><li><p>Frobenius Norm Matrix Factorization</p><p>QuestionGiven a matrix A Rnn and k &gt; 1, the goal is to output two matricesU Rnk ,V Rkn such that</p><p>UV A2F 6 (1 + ) minrankk A A A2F ,</p><p>where A2F = (n</p><p>i=1n</p><p>j=1 A2i,j)</p><p>2, [0,1)</p><p>Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50</p></li><li><p>Frobenius Norm Matrix Factorization</p><p>QuestionGiven a matrix A Rnn and k &gt; 1, the goal is to output two matricesU Rnk ,V Rkn such that</p><p>UV A2F 6 (1 + ) minrankk A A A2F ,</p><p>where A2F = (n</p><p>i=1n</p><p>j=1 A2i,j)</p><p>2, [0,1)</p><p>Useful for data compression, easier to store factorizations U,V</p><p>Can be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50</p></li><li><p>Frobenius Norm Matrix Factorization</p><p>QuestionGiven a matrix A Rnn and k &gt; 1, the goal is to output two matricesU Rnk ,V Rkn such that</p><p>UV A2F 6 (1 + ) minrankk A A A2F ,</p><p>where A2F = (n</p><p>i=1n</p><p>j=1 A2i,j)</p><p>2, [0,1)</p><p>Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... time</p><p>Fails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50</p></li><li><p>Frobenius Norm Matrix Factorization</p><p>QuestionGiven a matrix A Rnn and k &gt; 1, the goal is to output two matricesU Rnk ,V Rkn such that</p><p>UV A2F 6 (1 + ) minrankk A A A2F ,</p><p>where A2F = (n</p><p>i=1n</p><p>j=1 A2i,j)</p><p>2, [0,1)</p><p>Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorization</p><p>Not robust to noise, outliers, or missing entries</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50</p></li><li><p>Frobenius Norm Matrix Factorization</p><p>QuestionGiven a matrix A Rnn and k &gt; 1, the goal is to output two matricesU Rnk ,V Rkn such that</p><p>UV A2F 6 (1 + ) minrankk A A A2F ,</p><p>where A2F = (n</p><p>i=1n</p><p>j=1 A2i,j)</p><p>2, [0,1)</p><p>Useful for data compression, easier to store factorizations U,VCan be solved using the singular value decomposition (SVD) inmatrix multiplication n2.376... timeFails if you want a nonnegative factorizationNot robust to noise, outliers, or missing entries</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 3 / 50</p></li><li><p>. .</p><p>. .</p><p>Nonnegative Matrix Factorization</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 4 / 50</p></li><li><p>Example of NMF. .</p><p>. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50</p></li><li><p>Example of NMF. .</p><p>. .</p><p>Given : A Rnn, n = 4, k = 2</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50</p></li><li><p>Example of NMF. .</p><p>. .</p><p>Given : A Rnn, n = 4, k = 2</p><p>A =</p><p>0 1 1 11 2 1 12 4 2 23 5 2 2</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50</p></li><li><p>Example of NMF. .</p><p>. .</p><p>Given : A Rnn, n = 4, k = 2</p><p>A =</p><p>0 1 1 11 2 1 12 4 2 23 5 2 2</p><p>Question : Are there two matrices U,V&gt; Rnk , such thatUV = A,U &gt; 0,V &gt; 0</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50</p></li><li><p>Example of NMF. .</p><p>. .</p><p>Given : A Rnn, n = 4, k = 2</p><p>A =</p><p>0 1 1 11 2 1 12 4 2 23 5 2 2</p><p>Question : Are there two matrices U,V&gt; Rnk , such thatUV = A,U &gt; 0,V &gt; 0</p><p>=</p><p>0 11 12 23 2</p><p> [1 1 0 00 1 1 1]</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50</p></li><li><p>Example of NMF. .</p><p>. .</p><p>Given : A Rnn, n = 4, k = 2</p><p>A =</p><p>0 1 1 11 2 1 12 4 2 23 5 2 2</p><p>Question : Are there two matrices U,V&gt; Rnk , such thatUV = A,U &gt; 0,V &gt; 0</p><p>=</p><p>0 11 12 23 2</p><p> [1 1 0 00 1 1 1]</p><p>= UV</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 5 / 50</p></li><li><p>Main Question, Nonnegative Matrix Factorization</p><p>QuestionGiven matrix A Rnn and k &gt; 1, is there an algorithm that candetermine if there exist two matrices U,V&gt; Rnk ,</p><p>UV = A,U &gt; 0,V &gt; 0.</p><p>Or, are there any hardness results?</p><p>Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learningApplications</p><p>I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50</p></li><li><p>Main Question, Nonnegative Matrix Factorization</p><p>QuestionGiven matrix A Rnn and k &gt; 1, is there an algorithm that candetermine if there exist two matrices U,V&gt; Rnk ,</p><p>UV = A,U &gt; 0,V &gt; 0.</p><p>Or, are there any hardness results?</p><p>Equivalent to computing the nonnegative rank of A, rank+(A)</p><p>Fundamental question in machine learningApplications</p><p>I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50</p></li><li><p>Main Question, Nonnegative Matrix Factorization</p><p>QuestionGiven matrix A Rnn and k &gt; 1, is there an algorithm that candetermine if there exist two matrices U,V&gt; Rnk ,</p><p>UV = A,U &gt; 0,V &gt; 0.</p><p>Or, are there any hardness results?</p><p>Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learning</p><p>Applications</p><p>I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50</p></li><li><p>Main Question, Nonnegative Matrix Factorization</p><p>QuestionGiven matrix A Rnn and k &gt; 1, is there an algorithm that candetermine if there exist two matrices U,V&gt; Rnk ,</p><p>UV = A,U &gt; 0,V &gt; 0.</p><p>Or, are there any hardness results?</p><p>Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learningApplications</p><p>I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50</p></li><li><p>Main Question, Nonnegative Matrix Factorization</p><p>QuestionGiven matrix A Rnn and k &gt; 1, is there an algorithm that candetermine if there exist two matrices U,V&gt; Rnk ,</p><p>UV = A,U &gt; 0,V &gt; 0.</p><p>Or, are there any hardness results?</p><p>Equivalent to computing the nonnegative rank of A, rank+(A)Fundamental question in machine learningApplications</p><p>I Text mining, Spectral data analysis, Scalable Internet distanceprediction, Non-stationary speech denoising, Bioinformatics,Nuclear imaging, etc.</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 6 / 50</p></li><li><p>Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank</p><p>I rank(A) 6 rank+(A)I Vavasis09, determining whether rank+(A) = rank(A) is NP-hard.</p><p>.</p><p>.</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50</p></li><li><p>Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank</p><p>I rank(A) 6 rank+(A)</p><p>I Vavasis09, determining whether rank+(A) = rank(A) is NP-hard.</p><p>.</p><p>.</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50</p></li><li><p>Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank</p><p>I rank(A) 6 rank+(A)I Vavasis09, determining whether rank+(A) = rank(A) is NP-hard.</p><p>.</p><p>.</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50</p></li><li><p>Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank</p><p>I rank(A) 6 rank+(A)I Vavasis09, determining whether rank+(A) = rank(A) is NP-hard.</p><p>.</p><p>.</p><p>A =</p><p>1 1 0 01 0 1 00 1 0 10 0 1 1</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50</p></li><li><p>Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank</p><p>I rank(A) 6 rank+(A)I Vavasis09, determining whether rank+(A) = rank(A) is NP-hard.</p><p>.</p><p>.</p><p>A =</p><p>1 1 0 01 0 1 00 1 0 10 0 1 1</p><p>=</p><p>1 1 01 0 10 1 00 0 1</p><p> 1 0 0 10 1 0 1</p><p>0 0 1 1</p><p>rank(A) = 3,</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50</p></li><li><p>Rank vs. Nonnegative Rankrank(A) is the rank and rank+(A) is the nonnegative rank</p><p>I rank(A) 6 rank+(A)I Vavasis09, determining whether rank+(A) = rank(A) is NP-hard.</p><p>.</p><p>.</p><p>A =</p><p>1 1 0 01 0 1 00 1 0 10 0 1 1</p><p>=</p><p>1 1 01 0 10 1 00 0 1</p><p> 1 0 0 10 1 0 1</p><p>0 0 1 1</p><p>rank(A) = 3, but rank+(A) = 4</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 7 / 50</p></li><li><p>Polynomial System Verifier</p><p>. .</p><p>. .</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50</p></li><li><p>Polynomial System Verifier</p><p>. .</p><p>. .[Renegar92, Basu-Pollack-Roy96]</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50</p></li><li><p>Polynomial System Verifier</p><p>. .</p><p>. .[Renegar92, Basu-Pollack-Roy96]</p><p>Given : a polynomial system P(x) over the real numbers</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50</p></li><li><p>Polynomial System Verifier</p><p>. .</p><p>. .[Renegar92, Basu-Pollack-Roy96]</p><p>Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, , xv )</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50</p></li><li><p>Polynomial System Verifier</p><p>. .</p><p>. .[Renegar92, Basu-Pollack-Roy96]</p><p>Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, , xv )</p><p>m : #polynomial constraints fi(x) &gt; 0, i [m]</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50</p></li><li><p>Polynomial System Verifier</p><p>. .</p><p>. .[Renegar92, Basu-Pollack-Roy96]</p><p>Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, , xv )</p><p>m : #polynomial constraints fi(x) &gt; 0, i [m]d : maximum degree of all polynomial constraints</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50</p></li><li><p>Polynomial System Verifier</p><p>. .</p><p>. .[Renegar92, Basu-Pollack-Roy96]</p><p>Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, , xv )</p><p>m : #polynomial constraints fi(x) &gt; 0, i [m]d : maximum degree of all polynomial constraintsH : the bitsizes of the coefficients of the polynomials</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50</p></li><li><p>Polynomial System Verifier</p><p>. .</p><p>. .[Renegar92, Basu-Pollack-Roy96]</p><p>Given : a polynomial system P(x) over the real numbersv : #variables, x = (x1, x2, , xv )</p><p>m : #polynomial constraints fi(x) &gt; 0, i [m]d : maximum degree of all polynomial constraintsH : the bitsizes of the coefficients of the polynomials</p><p>In (md)O(v) poly(H) time, candecide if there exists a solution to polynomial system P</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 8 / 50</p></li><li><p>Main Idea</p><p>. .</p><p>. .</p><p>1. Write minU,V&gt;Rnk ,U,V&gt;0</p><p>UV A2F as a polynomial system</p><p>that has poly(k) variables and poly(n) constraints and degree</p><p>2. Use polynomial system verifier to solve it</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 9 / 50</p></li><li><p>Algorithm. .</p><p>. .Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50</p></li><li><p>Algorithm. .</p><p>. .</p><p>Given : A Rnn, k N</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50</p></li><li><p>Algorithm. .</p><p>. .</p><p>Given : A Rnn, k N</p><p>Question :</p><p>Razenshteyn-Song-Woodruff-Zhong Parameterized Complexity of Matrix Factorization 10 / 50</p></li><li><p>Algorithm. .</p><p>. .</p><p>Given : A Rnn, k N</p><p>Question : Are there matrices U,V&gt; Rnk such th...</p></li></ul>