1/ 9

Fast greedy algorithms for dictionary selectionwith generalized sparsity constraintsKaito Fujii & Tasuku Soma (UTokyo)

Neural Information Processing Systems 2018, spotlight presentationDec. 7, 2018

2/ 9

DictionaryIf real-world signals consist of a few patterns,a “good” dictionary gives sparse representations of each signal

patch dictionarysparse representation

≈

≈

≈

0.4 +0.1 +0.90.2 +0.2 +0.30.5 +0.5 +0.1

2/ 9

DictionaryIf real-world signals consist of a few patterns,a “good” dictionary gives sparse representations of each signal

patch

dictionarysparse representation

≈

≈

≈

0.4 +0.1 +0.90.2 +0.2 +0.30.5 +0.5 +0.1

2/ 9

DictionaryIf real-world signals consist of a few patterns,a “good” dictionary gives sparse representations of each signal

patch dictionary

sparse representation

≈

≈

≈

0.4 +0.1 +0.90.2 +0.2 +0.30.5 +0.5 +0.1

2/ 9

DictionaryIf real-world signals consist of a few patterns,a “good” dictionary gives sparse representations of each signal

patch dictionarysparse representation

≈

≈

≈

0.4 +0.1 +0.90.2 +0.2 +0.30.5 +0.5 +0.1

3/ 9

Dictionary selection [Krause–Cevher’10]

Union of existing dictionaries

DCT basis Haar basis Db4 basis Coiflet basisSelected atoms as a dictionary

≈ w1 +w2 +w3 ≈ w1 +w2 +w3

Atoms for each patch yt (∀t ∈ [T ])

XZ1 Z2

3/ 9

Dictionary selection [Krause–Cevher’10]

Union of existing dictionaries

DCT basis Haar basis Db4 basis Coiflet basisSelected atoms as a dictionary

≈ w1 +w2 +w3 ≈ w1 +w2 +w3

Atoms for each patch yt (∀t ∈ [T ])

XZ1 Z2

3/ 9

Dictionary selection [Krause–Cevher’10]

Union of existing dictionaries

DCT basis Haar basis Db4 basis Coiflet basisSelected atoms as a dictionary

≈ w1 +w2 +w3 ≈ w1 +w2 +w3Atoms for each patch yt (∀t ∈ [T ])

XZ1 Z2

3/ 9

Dictionary selection [Krause–Cevher’10]

Union of existing dictionaries

DCT basis Haar basis Db4 basis Coiflet basisSelected atoms as a dictionary

≈ w1 +w2 +w3 ≈ w1 +w2 +w3Atoms for each patch yt (∀t ∈ [T ])

XZ1 Z2

4/ 9

Dictionary selection with sparsity constraintsMaximizeX⊆V max

(Z1,··· ,ZT )∈I : Zt⊆XT∑t=1

ft(Zt) subject to |X | ≤ k

1st maximization:selecting a set X of atomsas a dictionary

Our contributionsOur contributions1 Replacement OMP:

A fast greedy algorithm with approximation ratio guarantees2 p-Replacement sparsity families:

A novel class of sparsity constraints generalizing existing ones

4/ 9

Dictionary selection with sparsity constraintsMaximizeX⊆V max

(Z1,··· ,ZT )∈I : Zt⊆XT∑t=1

ft(Zt) subject to |X | ≤ k

2nd maximization:selecting a set Zt ⊆ X of atomsfor a sparse representation of each patchunder sparsity constraint I

Our contributionsOur contributions1 Replacement OMP:

A fast greedy algorithm with approximation ratio guarantees2 p-Replacement sparsity families:

A novel class of sparsity constraints generalizing existing ones

4/ 9

Dictionary selection with sparsity constraintsMaximizeX⊆V max

(Z1,··· ,ZT )∈I : Zt⊆XT∑t=1

ft(Zt) subject to |X | ≤ kset function representingthe quality of Zt for patch ytsparsity constraint

Our contributionsOur contributions1 Replacement OMP:

A fast greedy algorithm with approximation ratio guarantees2 p-Replacement sparsity families:

A novel class of sparsity constraints generalizing existing ones

4/ 9

Dictionary selection with sparsity constraintsMaximizeX⊆V max

(Z1,··· ,ZT )∈I : Zt⊆XT∑t=1

ft(Zt) subject to |X | ≤ kset function representingthe quality of Zt for patch ytsparsity constraint

Our contributionsOur contributions1 Replacement OMP:

A fast greedy algorithm with approximation ratio guarantees

2 p-Replacement sparsity families:A novel class of sparsity constraints generalizing existing ones

4/ 9

Dictionary selection with sparsity constraintsMaximizeX⊆V max

(Z1,··· ,ZT )∈I : Zt⊆XT∑t=1

ft(Zt) subject to |X | ≤ kset function representingthe quality of Zt for patch ytsparsity constraint

Our contributionsOur contributions1 Replacement OMP:

A fast greedy algorithm with approximation ratio guarantees2 p-Replacement sparsity families:

A novel class of sparsity constraints generalizing existing ones

5/ 9

1 Replacement OMPReplacement Greedy for two-stage submodular maximization [Stan+’17]

Replacement Greedy O(s2dknT) running time1st result application to dictionary selection

Replacement OMP O((n+ ds)kT) running time2nd result O(s2d) acceleration with the concept of OMP

5/ 9

1 Replacement OMPReplacement Greedy for two-stage submodular maximization [Stan+’17]

Replacement Greedy O(s2dknT) running time1st result application to dictionary selection

Replacement OMP O((n+ ds)kT) running time2nd result O(s2d) acceleration with the concept of OMP

5/ 9

1 Replacement OMPReplacement Greedy for two-stage submodular maximization [Stan+’17]

Replacement Greedy O(s2dknT) running time1st result application to dictionary selection

Replacement OMP O((n+ ds)kT) running time2nd result O(s2d) acceleration with the concept of OMP

6/ 9

1 Replacement OMP

algorithm approximationratio

runningtime

empiricalperformance

SDSMA [Krause–Cevher’10]

SDSOMP [Krause–Cevher’10]

Replacement GreedyReplacement OMP

7/ 9

2 p-Replacement sparsity families

individual sparsity[Krause–Cevher’10]

individual matroids[Stan+’17]

block sparsity[Krause–Cevher’10]

average sparsityw/o individual sparsity

average sparsity[Cevher–Krause’11]

⊆⊆

⊆⊆

k-replacement sparse

(2k − 1)-replacement sparse

(3k − 1)-replacement sparse

7/ 9

2 p-Replacement sparsity families

individual sparsity[Krause–Cevher’10]

individual matroids[Stan+’17]

block sparsity[Krause–Cevher’10]

average sparsityw/o individual sparsity

average sparsity[Cevher–Krause’11]

⊆⊆

⊆⊆

k-replacement sparse

(2k − 1)-replacement sparse

(3k − 1)-replacement sparse

8/ 9

2 p-Replacement sparsity familiesWe extend Replacement OMP to p-replacement sparsity familiesTheorem

Replacement OMP achieves m22sM2s,21− exp − kp Ms,2m2s

-approximationif I is p-replacement sparse

Assumptionft(Zt) = maxwt : supp(wt)⊆Zt ut(wt)where ut ism2s-strongly concave on Ω2s = (x,y): ∥x− y∥0 ≤ 2sand Ms,2-smooth on Ωs,2 = (x,y): ∥x∥0 ≤ s, ∥y∥0 ≤ s, ∥x− y∥0 ≤ 2

9/ 9

Overview1 Replacement OMP: A fast algorithm for dictionary selection2 p-Replacement sparsity families: A class of sparsity constraints

Other contributionsOther contributionsEmpirical comparison with dictionary learning methodsExtensions to online dictionary selection

Poster #78 at Room 210 & 230 AB, Thu 10:45–12:45