Upload
zukun
View
221
Download
1
Embed Size (px)
Citation preview
Tutorial
Advanced Information Theory in CVPR “in a Nutshell”
CVPRJune 13-18 2010
San Francisco,CAIntroduction to ITinCVPR:The 4 Axes
Francisco Escolano
Why ITinCVPR?
Information theory (IT) has been growing in interest within theComputer Vision and Pattern Recognition (CVPR) community.In confluence with Bayesian theory, energy minimization, and othermethodologies, the role of IT is key to face the complex task ofdeveloping reliable and efficient CVPR algorithms.However, such confluence must evolve from exploiting IT for solvingspecific tasks towards an identification of common IT elements to alltasks and, mostly important, their interrelation.Here we consider four dimensions (axes) for describing theseinterrelations: measures, principles, theories, and entropy estimators.
2/24
The four Axes
Figure: ITinCVPR algorithms lay in a 4D conceptual manifold
3/24
IT Measures
Definition
IT measures are probability-based criteria designed for quantifying:information content, information gain and loss, information bounds,high-order statistical dependencies,....
Roles
The main roles of IT measures are:
I Definition of error bounds.
I Definition of discrimination criteria.
I Definition of optimization criteria for generative processes.
I Definition of convergence criteria.
4/24
IT Measures (2)
Error-bound Discriminate Generate Converge
Entropy√ √
Kullback-Leibler√ √ √
Jensen-Shannon√ √
Bregman√
Mutual Info√ √
Chernoff Info√ √
Channel Capacity√ √
Fisher-Rao Tensor√
Table: The roles of some IT measures
5/24
IT Measures (3)
MD Feature Selection
Number of Selected Gene
Cla
ss (
dise
ase)
MELANOMAMELANOMAMELANOMAMELANOMAMELANOMAMELANOMA
BREASTBREAST
MELANOMANSCLCNSCLCNSCLC
BREASTMCF7D−repro
BREASTMCF7A−repro
COLONCOLONCOLONCOLONCOLONCOLONCOLON
LEUKEMIALEUKEMIALEUKEMIALEUKEMIALEUKEMIA
K562A−reproK562B−reproLEUKEMIA
NSCLCNSCLCNSCLC
PROSTATEOVARIANOVARIANOVARIANOVARIANOVARIAN
PROSTATEMELANOMA
OVARIANUNKNOWN
RENALNSCLC
BREASTRENALRENALRENALRENALRENALRENALRENALNSCLCNSCLC
BREASTCNSCNS
BREASTRENAL
CNSCNSCNS
19→
135
246
663
766
982
1177
1470
1671
→ 2
080
3227
3400
3964
4057
4063
4110
4289
4357
4441
4663
4813
5226
5481
5494
5495
5508
5790
5892
6013
6019
6032
6045
6087
→ 6
145
6184
6643
mRMR Feature Selection
Number of Selected Gene
MELANOMAMELANOMAMELANOMAMELANOMAMELANOMAMELANOMA
BREASTBREAST
MELANOMANSCLCNSCLCNSCLC
BREASTMCF7D−repro
BREASTMCF7A−repro
COLONCOLONCOLONCOLONCOLONCOLONCOLON
LEUKEMIALEUKEMIALEUKEMIALEUKEMIALEUKEMIA
K562A−reproK562B−reproLEUKEMIA
NSCLCNSCLCNSCLC
PROSTATEOVARIANOVARIANOVARIANOVARIANOVARIAN
PROSTATEMELANOMA
OVARIANUNKNOWN
RENALNSCLC
BREASTRENALRENALRENALRENALRENALRENALRENALNSCLCNSCLC
BREASTCNSCNS
BREASTRENAL
CNSCNSCNS
133
134
→ 1
3523
325
938
156
113
7813
8214
0918
41→
208
020
8120
8320
8632
5333
7133
7243
8344
5945
2754
3555
0455
3856
9658
1258
8759
3460
7261
15→
614
563
0563
9964
2964
3065
66
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Figure: Microarray Gene Selection with Multi-dimensional Mutual Info
6/24
IT Measures (and 4)
Figure: Mutliple Point-Set Registration though Havrda-Chavrat Divergence
7/24
IT Principles
Definition
Formal requirements casting the optimal solutions to the CVPRproblem at hand and thus the design of a proper algorithm.
Roles
The main roles of IT principles are the definition of:
I Model-order selection criteria (] regions/clusters).
I Feature selection and transformation (coding) criteria.
I Optimization criteria for generating models for the data.
I Classification-design criteria.
8/24
IT Principles (2)
Model order Features Generate Classify Coding
MDL√
MML√ √ √
MaxEnt√ √
MiniMax√ √ √
Infomax√ √
Non-Gaussianity√ √
MED√
Min Correlation√
Inf.Bottleneck√
Table: The roles of some IT principles
9/24
IT Principles (and 3)
Figure: MDL Model Order Selection in Gaussian Mixtures
10/24
IT Theories
Definition
Mathematical developments leading to different formulations of thesame problem and, thus contributing with new formal perspectives.
Main Roles
The main roles of IT theories are the definition of:
I Generative/discriminative formulations.
I Matching formulations.
I Clustering formulations.
I Feature and classification design formulations.
11/24
IT Theories (2)
Discr. Generate Segment Cluster Feature/Class.
Method of Types√ √ √
Rate-Distortion√ √ √
Info geometry√ √ √ √
Proj. pursuit√ √ √
Bregman divs.√ √ √
Table: The roles of some IT theories
12/24
IT Theories (3)
Figure: Saliency Filtering through Method of Types
13/24
IT Theories (and 4)
Figure: Minimum Enclosing Bregman Balls
14/24
Entropy Estimation
Definition
Asymptotically unbiased non-parametric estimator of entropy orrelated IT measure (KL-divergence, Mutual Info) considering alsodifferent definitions of entropy and their formal interrelations.
Definitions of entropy
I Shannon: H(p) = −∑
i pi log pi .
I Renyi: Rα(p) = 11−α log (
∑i pα
i ).
I Tsallis: Sα(p) = 1α−1 (1−
∑i pα
i ).
I Burg: B(p) =∑
i log pi .
15/24
Entropy Estimation (2)
Problems in EE
Typically a small number of samples N and high number ofdimensions d : curse of dimensionality.
Types of EEs
I Plug-in: Necessarily estimate the underlying pdf (binning,Parzen Windows, Voronoi-based...). Useful for low d .
I Bypass: Exploit only the available data avoiding pdf estimation.Typically nearest neighbors-based methods (Entropic Graphs,Leonenko) or KD-partitions. Useful for intermediate and evenhigh d .
16/24
Entropy Estimation (3)
−20 −15 −10 −5 0 5 10 15 20−20
−15
−10
−5
0
5
10
15
20
25
0 2 4 6 8 10 12 14 16 18 200
2
4
6
8
10
12
14
16
18
20
Figure: Entropic Graph Entropy Estimation (Gaussian vs Uniform)
17/24
Entropy Estimation (and 4)
Figure: Joint PDFs for Image Alignmnet: without and with interpolation
18/24
The ITinCVPR Tube
Figure: ITinCVPR tube lines, stations and quarters. 19/24
ITinCVPR Tube
Quarters ≡ Tasks
Our tube links IT elements for solving 6 CVPR tasks: featureextraction and grouping, segmentation, registration/matching andrecognition, feature selection and transformation, image and patternclustering and classifier design
Lines and transfer stationsI Measures, principles and theories lines interact through transfer
stations. Transferring can imply carrying on previous ITelements.
I Entropy estimation (fourth dimension) is marked at somepoints where such estimation is either critical or challenging.
20/24
The Measures Lines
Figure: ITinCVPR tube lines, stations and quarters. 21/24
The Principles Lines
Figure: ITinCVPR tube lines, stations and quarters. 22/24
The Theories Lines
Figure: ITinCVPR tube lines, stations and quarters. 23/24
Tutorial Schedule
Hour Topic Instructor
2-2:15pm Introduction: The 4 axes Escolano
2:15-2:45pm Interest Points & Method of Types Escolano
2:45-3:30pm High-Dimensional Feature Selection Escolano
3:30-4pm COFFEE BREAK4-4:30pm Isocontours and Image Registration Rangarajan
4:30-5:15pm Shape Matching with I-Divergences Rangarajan
5:15-6pm Gaussian Mixtures: MDL and Variational Both
Table: Tutorial Schedule
24/24