Non-negative Matrix Factorization with Sparseness Constraints

NON-NEGATIVE MATRIX FACTORIZATION WITH SPARSENESS CONSTRAINTS

JOURNAL OF MACHINE LEARNING RESEARCH,2004Patrik O. Hoyer

Jain-De,Lee

2

OUTLINE Introduction

Adding Sparseness Constraints to NMF

Experiments with Sparseness Constraints

Conclusions

3

INTRODUCTION• Non-negative matrix factorization (NMF)• A useful representation typically makes latent

structure in the data explicit• Reduces the dimensionality of the data

• The non-negativity constraints make the representation purely additive

4

INTRODUCTION• Sparse representation• Representation encodes much of the data using few

‘active’ components

• The sparseness given by NMF is somewhat of a side-effect rather than a goal

• Include the option to control sparseness explicitly

5

ADDING SPARSENESS CONSTRAINTS TO NMF• The concept of sparse coding• Only a few units are effectively used to represent

typical data vectors

Illustration of various degrees of sparseness

6

ADDING SPARSENESS CONSTRAINTS TO NMF• Sparseness measure

• Based on the relationship between the L1 norm and the L2 norm

1

/)()(

2

n

xxinxsparseness i

where n is the dimensionality of x

7

ADDING SPARSENESS CONSTRAINTS TO NMF• To constrain NMF to find solutions with desired

degrees of sparseness• What exactly should be sparse?• Under optional constraints minimized

iSwsparseness wi ,)(

iShsparseness hi ,)(

Where wi is the ith column of W and hi is the ith row of H

Sw and Sh are the desired sparsenesses of W and H (respectively)

8

ADDING SPARSENESS CONSTRAINTS TO NMF• Projected gradient descent algorithm for NMF

with sparseness constraints

Initialize

Project

Iterate

9

ADDING SPARSENESS CONSTRAINTS TO NMF

Project

If sparseness constraints on W:

Project each column of W to be non-negative

Have unchanged L2 norm, L1 norm set to achieve desired sparseness

If sparseness constraints on H:

Project each row of H to be non-negative

Have unchanged L2 norm, L1 norm set to achieve desired sparseness

10

ADDING SPARSENESS CONSTRAINTS TO NMF• Iterate• If sparseness constraints on W(or H) apply

• Set or

• Project• else take standard multiplicative step

TW HVWHWW )(

)( VWHWHH TH

Where μW and μH are small positive constants

T

T

HWHVHWW

)( or

)(WHWVWHH T

T

11

ADDING SPARSENESS CONSTRAINTS TO NMF

• Projection operator• Problem

Given any vector x, find the closest non-negative vector s with a given L1 norm and a given L2 norm

12

ADDING SPARSENESS CONSTRAINTS TO NMF• Algorithm• Set• Set Z={}• Iterate

1. Set2. Set ,where α ≥ 03. If all components of s are non-negative, return

s, end 4. Set Z=Z {i ; s∪ i<0}5. Set si=0 , 6. Calculate 7. Set si= si –c ,8. Go to 1

ixxLxs iii ),dim(/)( 1

0

))()/(dim(1 ZsizexLmi

)( msms

Zi

Zi))()/(dim()( 1 ZsizexLsc i

ZiifZiif

13

EXPERIMENTS WITH SPARSENESS CONSTRAINTS

NMF applied to various image data sets(a) Basis images given by NMF applied to face image data from the CBCL database(b) Basis images derived from the ORL face image database

(c) Basis vectors from NMF applied to ON/OFF-contrast filtered natural image data

14

EXPERIMENTS WITH SPARSENESS CONSTRAINTS

Features learned from the CBCL face image database using NMF with sparseness constraints

15

EXPERIMENTS WITH SPARSENESS CONSTRAINTS

Features learned from the ORL face image database using NMF with three levels sparseness constraints (a) 0.5 (b) 0.6 (c) 0.75

16

EXPERIMENTS WITH SPARSENESS CONSTRAINTS

Standard NMF (Figure 1c)

The sparseness of the coefficients was fixed at 0.85

17

EXPERIMENTS WITH SPARSENESS CONSTRAINTS

Number of iterations required for the projectionalgorithm to converge

18

CONCLUSIONS• Useful to control the degree of sparseness explicitly

• Describe a projection operator capable of simultaneously enforcing both L1 and L2 norms

• To show its use in the NMF framework for learning representations that could not be obtained by regular NMF