Upload
tabitha-copeland
View
227
Download
5
Tags:
Embed Size (px)
Citation preview
Learning sparse representations to restore, classify, and sense images and videos
Guillermo Sapiro
University of Minnesota
Supported by NSF, NGA, NIH, ONR, DARPA, ARO, McKnight FoundationSupported by NSF, NGA, NIH, ONR, DARPA, ARO, McKnight Foundation
Learning Sparsity 2
Martin Duarte
Rodriguez
Ramirez
Lecumberry
Learning Sparsity 3
Overview
• Introduction– Denoising, Demosaicing, Inpainting– Mairal, Elad, Sapiro, IEEE-TIP, January 2008
• Learn multiscale dictionaries – Mairal, Elad, Sapiro, SIAM-MMS, April 2008
• Sparsity + Self-similarity– Mairal, Bach, Ponce, Sapiro, Zisserman, pre-print.
• Incoherent dictionaries and universal coding– Ramirez, Lecumberry, Sapiro, June 2009, pre-print
• Learning to classify– Mairal, Bach, Ponce, Sapiro, Zisserman, CVPR 2008, NIPS 2008– Rodriguez and Sapiro, pre-print, 2008.
• Learning to sense sparse signals– Duarte and Sapiro, pre-print, May 2008, IEEE-TIP to appear
Learning Sparsity 4
Introduction I: Sparse and
Redundant Representations
Webster Dictionary: Of few and scattered elements
Learning Sparsity 5
Relation to measurements
Restoration by Energy Minimization
Thomas Bayes 1702 -
1761
Prior or regularizationy : Given measurements
x : Unknown to be recovered
xPryx21
xf22
Restoration/representation algorithms are often related to the minimization of an energy function of the form
Bayesian type of approach
What is the prior? What is the image model?
Learning Sparsity 6
The Sparseland Model for Images
MK
N
DA fixed Dictionary
Every column in D (dictionary) is a prototype signal (Atom).
The vector contains very few (say L) non-zeros.A sparse
& random vector
αx
N
Learning Sparsity 7
What Should the Dictionary D Be?
ˆx̂L.t.sy21
minargˆ00
22
DD
D should be chosen such that it sparsifies the representations (for a given task!)
Learn D :
Multiscale Learning
Color Image Examples
Task / sensing adapted
Internal structure
One approach to choose D is from a known set of transforms
(Steerable wavelet, Curvelet, Contourlets, Bandlets, …)
Learning Sparsity 8
Introduction II: Dictionary
Learning
Learning Sparsity 9
Each example is a linear combination of atoms from
D
Measure of Quality for D
DX A
Each example has a sparse representation with no more than L
atoms
L,j.t.sxMin0
0j
P
1j
2
2jj,
DAD Field & Olshausen (‘96)
Engan et. al. (‘99)Lewicki & Sejnowski (‘00)
Cotter et. al. (‘03)Gribonval et. al. (‘04)
Aharon, Elad, & Bruckstein (‘04) Aharon, Elad, & Bruckstein (‘05)
Ng et al. (‘07)Mairal, Sapiro, Elad (‘08)
Learning Sparsity 10
The K–SVD Algorithm – General
DInitialize D
Sparse Coding
Orthogonal Matching Pursuit (or L1)
Dictionary Update
Column-by-Column by SVD computation over the relevant
examples
Aharon, Elad, & Bruckstein (`04)
XT
Learning Sparsity 11
Show me the pictures
Learning Sparsity 12
Change the Metric in the OMP
Learning Sparsity 13
Non-uniform noise
Learning Sparsity 14
Example: Non-uniform noise
Learning Sparsity 15
Example: Inpainting
Learning Sparsity 16
Example: Demoisaic
Learning Sparsity 17
Example: Inpainting
Learning Sparsity 18
Not enough fun yet?:
Multiscale Dictionaries
Learning Sparsity 19
Learned multiscale dictionary
Learning Sparsity 20
Learning Sparsity 21
Color multiscale dictionaries
Learning Sparsity 22
Example
Learning Sparsity 23
Video inpainting
Extending the Models
Learning Sparsity 24
Universal Coding and Incoherent Dictionaries
• Consistent• Improved generalization properties• Improved active set computation• Improved coding speed• Improved reconstruction• See poster by Ramirez and Lecumberry…
Learning Sparsity 25
Sparsity + Self-similarity=Group Sparsity
• Combine the two of the most successful models for images
• Mairal, Bach. Ponce, Sapiro, Zisserman, pre-print, 2009
Learning Sparsity 26
Learning to Classify
Learning Sparsity 28
Global Dictionary
Learning Sparsity 29
Barbara
Learning Sparsity 30
Boat
Learning Sparsity 31
Digits
Which dictionary? How to learn them?
• Multiple reconstructive dictionary? (Payre)
• Single reconstructive dictionary? (Ng et al, LeCunn et al.)
• Dictionaries for classification!
• See also Winn et al., Holub et al., Lasserre et al., Hinton et al. for joint discriminative/generative probabilistic approaches
Learning Sparsity 32
Learning Sparsity 33
Learning multiple reconstructive and discriminative dictionaries
With J. Mairal, F. Bach, J. Ponce, and A. Zisserman, CVPR ’08, NIPS ‘08
Learning Sparsity 34
Texture classification
Semi-supervised detection learning
MIT -- Learning Sparsity 35
Learning Sparsity 36
Learning a Single Discriminative and Reconstructive Dictionary
• Exploit the representation coefficients for classification– Include this in the optimization
– Class supervised simultaneous OMP
With F. Rodriguez
Learning Sparsity 37
Digits images: Robust to noise and occlusions
Learning Sparsity 38
Supervised Dictionary Learning
With J. Mairal, F. Bach, J. Ponce, and A. Zisserman, NIPS ‘08
Learning to Sense Sparse Images
Motivation
• Compressed sensing (Candes &Tao, Donoho, et al.)– Sparsity
– Random sampling• Universality
• Stability
• Shall the sensing be adapted to the data type?– Yes! (Elad, Peyre, Weiss et al., Applebaum et al, this talk).
• Shall the sensing and dictionary be learned simultaneously?
Learning Sparsity 40
Some formulas….
Learning Sparsity 41
+ “RIP (Identity Gramm Matrix)”
Design the dictionary and sensing together
Learning Sparsity 42
Just Believe the Pictures
Learning Sparsity 43
Just Believe the Pictures
Learning Sparsity 44
Just Believe the Pictures
Learning Sparsity 45
Learning Sparsity 46
Conclusions• State-of-the-art denoising results for still (shared with Dabov et
al.) and video• General• Vectorial and multiscale learned dictionaries• Dictionaries with internal structure• Dictionary learning for classification
– See also Szlam and Sapiro, ICML 2009– See also Carin et al, ICML 2009
• Dictionary learning for sensing
• A lot of work still to be done!
Please do not use the wrong dictionaries…
• 12 M pixel image
• 7 million patches
• LARS+online learning:
~8 minutes
• Mairal, Bach, Ponce, Sapiro, ICML 2009
Learning Sparsity 47
Learning Sparsity 48