1

Nonnegative Matrix Factorization Nonnegative Matrix Factorization via Rank-one Downdatevia Rank-one Downdate

Ali GhodsiDepartment of Statistics and Actuarial Science

David R. Cheriton School of Computer Science

University of Waterloo

Joint work with Stephen Vavasis and Michael Biggs

University of Waterloo

2

Nonnegative Matrix FactorizationNonnegative Matrix Factorization

3

4

560 by 1965

560 by 2

2 by 1965

20 by 28 20 by 28

-2.19

-0.02

-3.19

1.02

2 by 12 by 1

5

Singular Value Decomposition (SVD)Singular Value Decomposition (SVD)

6

HistoryHistory

7

HistoryHistory

8

HistoryHistory

9

HistoryHistory

10

History (Algorithms)History (Algorithms)

11

History (Algorithms)History (Algorithms)

12

First observationFirst observation

13

Power methodPower method

Computes the leading singular vectors/value (or eigenvector/value) of a matrix :)d(powermetho,, Avu

1

2 while not converged

uA

uA

uAv

AvAv

u

T

T

T

ones of vectorv

3

4

5

6 end

14

1

2

3

4

5 for all set

6 end for

Naive approach to NMF using this Naive approach to NMF using this observationobservation

Without step 5, this will simply compute the SVD (Jordan's algorithm, Camille Jordan 1874. )

:)nmf(, AHW

ki : for 1)d(powermetho],,[ Avu

TTii vHuW ,

TvuAA

0jiA ,0jiA ,

15

Rank-one Downdata (R1D)Rank-one Downdata (R1D)

16

Objective functionObjective function

17

ApproxRankOneSubmatrix(A)ApproxRankOneSubmatrix(A)

18

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

A =

19

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

0.14 0.07 0.64 0.41 0.55v:

20

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

0.0 0.0 0.64 0.41 0.55v:

21

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

v: 0.0 0.0 0.64 0.41 0.55

22

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

0.160.210.220.440.740.20

v: 0.0 0.0 0.64 0.41 0.55

23

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v: 0.0 0.0 0.64 0.41 0.55

0.00.00.0

0.440.740.20

24

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v: 0.0 0.0 0.64 0.41 0.55

0.00.00.0

0.440.740.20

25

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v: 0.0 0.0 0.60 0.28 0.59

0.00.00.0

0.440.740.20

26

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v:

0.00.00.0

0.440.740.20

0.0 0.0 0.60 0.28 0.59

27

Modified power iteration: DemoModified power iteration: Demo

Rank-1submatrix

Rank-1submatrix

u:

v:

0.00.00.0

0.440.740.20

0.0 0.0 0.60 0.28 0.59

Zero-out!Zero-out!

28

Modified power iteration: DemoModified power iteration: Demo

Anew =

Rank-1submatrix

29

Rank-one Downdata (R1D)Rank-one Downdata (R1D)

30

A simple model for textA simple model for text

31

Generating a corpus in the modelGenerating a corpus in the model

32

Theorem about textTheorem about text

33

LSILSI

34

R1DR1D

35

Theorem about imagesTheorem about images

36

Experimental resultsExperimental results

37

LSILSI

38

NMF-DIVNMF-DIV

39

R1DR1D

40

LSILSI

41

NMF_DIVNMF_DIV

42

R1DR1D

43

Thank you!Thank you!