Local Invariant Feature Descriptors

Bin Fan

National Laboratory of Pattern Recognition (NLPR)Institute of Automation, Chinese Academy of

Sciences

局部图像特征描述 —— 应用

• Wide-Baseline Image Matching– Structure from Motion, Image-based Localization,

Image Stitch

• Object/Instance/Scene Recognition• Object Detection• Image Retrieval

Structure From Motion

Object Recognition

Database

…

Design method:• Handcrafted Descriptors• Data-driven Descriptors

Categories of Descriptors

Developments: Handcrafted Descriptors

• 1999, SIFT [Citation: 23819]• 2003, Shape Context• 2006, SURF [Citation: 4093]• 2008, SMD, DAISY• 2009, OSID, CS-LBP• 2010, BRIEF, HRI-CSLTP, BiCE• 2011, ORB, BRISK, LIOP, MROGH• 2012, FREAK, KAZE, SYM• 2013, Line Context

• 2004, PCA-SIFT• 2007, LDE, Learning descriptor[Brown et al.]• 2009, Best DAISY• 2012, D-BRIEF, Learning descriptor by convex

optimization[Simonyan et al.], BGM/LBGM, LDAHash

• 2013, BinBoost, SQ-SIFT/DAISY

Developments: Data-driven Descriptors

Design method:• Handcrafted Descriptors• Data-driven Descriptors

Encode information:• Gradient-based Descriptors• Intensity-based Descriptors• Descriptor-based Descriptors

Categories of Descriptors

• Gradient-Based– SIFT、 DAISY、 BiCE、MROGH、 BGM、

LBGM、 BinBoost、 Learning Descriptor[Brown et al., Simonyan et al.]

• Intensity-Based– CS-

LBP、 OSID、 BRIEF、 ORB、 BRISK、 FREAK、 LDE、 D-BREIF、 LIOP

• Descriptor-Based– LDAHash, LDP[Cai et al.,PAMI’11]

Design method:• Handcrafted Descriptors• Data-driven Descriptors

Data type:• Floating-point Descriptors• Binary Descriptors

Encode information:• Gradient-based Descriptors• Intensity-based Descriptors• Descriptor-based Descriptors

Categories of Descriptors

• Floating-point Descriptors– SIFT、 SURF、 DAISY、 CS-LBP、 OSID、

MROGH、 LIOP、 LBGM、 LDE…

• Binary Descriptors– BiCE、 BRIEF、 ORB、 FREAK、 BRISK、

BGM、 BinBoost、 LDAHash、 D-BRIEF…

Name Mem. Com. Mat. Dist. Rob.

Floating point descriptors

SIFT ●●● ●●● ●●● ●●● ●●●

SURF ●○ ●● ●○ ●●○ ●●○

DAISY ●●●●○ ●●●● ●●●●○ ●●●○ ●●●○

LIOP ●●●○ ●●● ●●●○ ●●●● ●●●●

MROGH ●●●● ●●●●○ ●●●● ●●●●○ ●●●●○

Binary descriptors

BRIEF ●○ ○ ● ●●● ●●

ORB ●○ ● ● ●●● ●●○

FREAK ●○ ● ● ●●● ●●○

D-BRIEF ○ ○ ○ ●● ●●

BinBoost ● ●● ○ ●●● ●●●

Handcrafted Descriptors - SIFT

• Binning of Spatial Coordinates and Gradient Orientations• Soft Assignment of Binning• 4x4 spatial grids, 8 gradient orientations, 128 dim SIFT• Normalization

SIFT Descriptor [Lowe’99]

Handcrafted Descriptors - DAISY

• Log-polar grid arrangement• Gaussian pooling of histograms of gradient orientations• Efficient for dense computation, but not for sparse keypoints!

DAISY Descriptor [Tola et al.’08]

Descriptor Learning – Data Driven Methods

Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12]

Normalized Patch

SmoothLow-level

feature extraction

Spatial pooling

Post process

Projection

Descriptor

Learning

Descriptor Learning – Data Driven Methods

Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12]

• Pre-defined low level features: gradient-based, filter bank based• Pre-defined spatial poolings: SIFT-like, DAISY-like, GLOH-like• Optimized combination of low level feature + spatial pooling• Projection: PCA, LDE …

1st: DAISY-like spatial pooling + filter bank [high Dim]2nd: DAISY-like spatial pooling + gradient [moderate Dim]

PCA is better than LDE for projecting descriptor

Descriptor Learning – Data Driven Methods

Simonyan et al.’s method [ECCV’12]

Normalized Patch

SmoothGradient

map calculation

Spatial pooling

Projection

Descriptor

Learning

• Spatial pooling is constrained to rings• Using L1 regularization to select pooling rings from a large pool• Max-Margin based objective function [convex]• Best reported results in the Brown et al.’s dataset

Handcrafted Binary Descriptors

Pioneering work: LBP

Handcrafted Binary Descriptors

1 1( ) ( ; , ), , ( ; , ) {0,1}nn n nf P P x y P x y

Construct descriptor by binary tests:

Binary tests:

1, ( ) ( )( ; , )

0, ( ) ( )

P x P yP x y

P x P y

Pre-defined positions for binary tests:2 2

(0, ), (0, )25 25

S Sx G y G

BRIEF [ECCV’10, PAMI’12]

Handcrafted Binary Descriptors - BRIEF

Low memory, Fast to compute and match

Limited performance

Handcrafted Binary Descriptors

FREAK [CVPR’12]

Organizing sampling points analogous to retina structure

Learning Binary Descriptors

D-BRIEF [ECCV’12]

• Linear representation of projection matrix by Box/Gaussian/Rect filters• Approximate projection by filter responses• Efficient computation of Box/Gaussian/Rect filter responses• Binarization after discriminative projection• Extremely compact [only 32bits = 4 bytes]

Learning Binary Descriptors

BGM [NIPS’12]

• Explore gradient orientation maps as weak learners• Each bit is construct by one weak learner• Select discriminative gradient orientation maps by boosting

(P1(1), P2

(1),c(1)) (P1(2), P2

(2),c(2)) (P1(n), P2

(n),c(n))

…

Learning Binary Descriptors

BinBoost [CVPR’13]

• Each bit as a linear combination of many gradient orientation maps• Optimization based on boosting• Very compact [64 bits = 8 bytes]

Dataset and Evaluation

• Different contexts Image Matching Object/Instance Recognition Image Retrieval

Dataset and Evaluation: Matching

…

Oxford dataset [2D scenes]: popular benchmarkhttp://www.robots.ox.ac.uk/~vgg/research/affine/index.htmlK. Mikolajczyk, C. Schmid,  A performance evaluation of local descriptors. PAMI’05

Dataset and Evaluation: Matching

Oxford dataset [2D scenes]: popular benchmark

Evaluation protocol: recall vs. 1-precision

#

#

correct matchesrecall

correspondences

#

#

correct matchesprecision

matches

Dataset and Evaluation: Matching

Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptorshttp://www.cs.ubc.ca/~mbrown/patchdata/patchdata.htmlM. Brown, G. Hua and S. Winder,  Discriminant Learning of Local Image Descriptors. PAMI’12

Three different subsets, each of which has more than 400k patch pairs

Liberty Notre Dame Yosemite

Dataset and Evaluation: Matching

Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors

Evaluation protocol: False Positive Rate(FPR) vs. Recall

# False PositivesFPR

# Negatives

#True PositivesRecall

# Positives

Dataset and Evaluation: Recognition

• Dataset: Ukbench, ZuBuD, …• Evaluation Protocol: Recognition rate, recall

Dataset and Evaluation: Retrieval

• Dataset: Oxford/Paris Building, Holidays• Evaluation Protocol: mAP, Precision vs. Recall

AP(Average Precision): Precision across all recallsmAP: mean AP of all queries

Resources

• OpenCV: http://opencv.org/– SIFT, SURF, BRISK, BRIEF, ORB, FREAK

• VLFeat: http://www.vlfeat.org/– SIFT, LIOP, Covariant Feature Detectors

• Oxford VGG: http://www.robots.ox.ac.uk/~vgg/research/affine/index.html

• Authors’ pages…

Published Evaluations: Matching

1. K. Mikolajczyk and C. Schmid,  A Performance Evaluation of Local Descriptors. PAMI’05

2. P. Moreels and P. Perona,   Evaluation of Features Detectors and Descriptors based on 3D objects. IJCV’07

3. Anders Lindbjerg Dahl et al., Finding the Best Feature Detector-Descriptor Combination. 3DIMPVT’11

4. O.Miksik and K. Mikolajczyk, Evaluation of Local Detectors and Descriptors for Fast Feature Matching, ICPR’12

5. J. Heinly et al., Comparative Evaluation of Binary Features, ECCV’12

Published Evaluations: Classification/Recognition

1. K. Mikolajczyk et al.,  Local Features for Object Class Recognition. ICCV’05

2. E. Seemann et al.,   An Evaluation of Local Shape-Based Features for Pedestrian Detection. BMVC’05

3. M. Stark and B. Schiele, How Good are Local Features for Classes of Geometric Objects. ICCV’07

4. J. Zhang et al., Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, IJCV’07

5. K. E. A. Van de Sande et al., Evaluation of Color Descriptors for Object and Scene Recognition, PAMI’10

Joint work with Zhenhua Wang

Our Work

1. Feature Description by Intensity Order Pooling2. Local Intensity Order Pattern

Feature Description by Intensity Order Pooling

With a reference orientation:SIFT, SURF, DAISY, CS-LBP …

Category of handcrafted descriptors

With a reference orientation:SIFT, SURF, DAISY, CS-LBP …

Category of handcrafted descriptors

+: encode spatial information, high discriminability-: sensitive to orientation estimation error

36%

64%

Match vs. Orientation error

With a reference orientation:SIFT, SURF, DAISY, CS-LBP …

Category of handcrafted descriptors

RobustnessDistinctiveness

+: encode spatial information, high discriminability-: sensitive to orientation estimation error

Without a reference orientation:RIFT, Spin image

Category of handcrafted descriptors

+: inherently rotation invariance, robust to orientation estimation error-: discard some spatial information, limited discriminability

0 255/2π

0

r

Without a reference orientation:RIFT, Spin image

Category of handcrafted descriptors

RobustnessDistinctiveness

+: inherently rotation invariance, robust to orientation estimation error-: discard some spatial information, limited discriminablity

With a reference orientation:SIFT, SURF, DAISY, CS-LBP …

Without a reference orientation:RIFT, Spin image

Category of handcrafted descriptors

RobustnessDistinctiveness

+: encode spatial information, high discriminability-: sensitive to orientation estimation error

+: inherently rotation invariance, robust to orientation estimation error-: discard some spatial information, limited discriminablity

With a reference orientation:SIFT, SURF, DAISY, CS-LBP …

Without a reference orientation:RIFT, Spin image

Category of handcrafted descriptors

RobustnessDistinctiveness

+: encode spatial information, high discriminability-: sensitive to orientation estimation error

+: inherently rotation invariance, robust to orientation estimation error-: discard some spatial information, limited discriminablity

Robustness

Distinctiveness

Our Solution

Gradient orientation maps [SIFT]

Center-symmetrical binary pattern [CS-LBP]

Construct a local coordinate for low-level feature computation

……

……

…

……

Our Solution

Pool low-level features by intensity orders

Using multiple support regions

Our Solution

Using multiple support regions

Gradient orientation maps -> MROGH

Center-symmetrical binary pattern -> MRRID

Code: http://www.openpr.org.cn/index.php/89-MROGH-v1.1/View-details.html

Multiple Support Regions vs. Single Support Region

Experiments

MROGH MRRID SIFT

SR-i: Results of using the i-th support regionMR: Results of using multiple support region

Averaged results over 140 image pairs

Experiments

Image Matching – Oxford Dataset

Hessian-Affine, Viewpoint change

Experiments

Image Matching – Oxford Dataset

Harris-Affine, Image Blur

Experiments

Object Recognition:

Datasets: 53 Objects, ZuBuD, Ukbench

265 images of 53 objectsEach object has 5 images of different viewpoints

Experiments

Object Recognition:

Datasets: 53 Objects, ZuBuD, Ukbench

1005 images of 201 buildings in the Zurich cityEach building has 5 images of different viewpoints, across seasons

Experiments

Object Recognition:

Datasets: 53 Objects, ZuBuD, Ukbench

10200 images of 2550 objects [first 4000 images used here]Each object has 4 images of different viewpoints

Experiments

Object Recognition:

53 ObjectsRIFT SIFT DAISY MROGH MRRID

37.0% 52.2% 61.2% 72.5% 57.4%

ZuBuDRIFT SIFT DAISY MROGH MRRID

66.8% 75.5% 83.1% 88.1% 78.6%

UkbenchRIFT SIFT DAISY MROGH MRRID

34.0% 48.2% 58.3% 74.0% 57.5%

Experiments

Recognition examples: 53 Objects

input images

Experiments

Recognition examples: ZuBuD

input images

Experiments

Recognition examples: Ukbench

input images

Local Intensity Order Pattern

Local Intensity Order Pattern• Explore the relative intensity relationship among

neighboring points• Rotationally invariant computation of neighboring points’

intensities• Intensity order based pooling

Code: http://vision.ia.ac.cn/Students/wzh/publication/liop/index.html http://www.vlfeat.org/api/liop.html

Experiments

Image Matching: Oxford dataset

Experiments

Image Matching: Oxford dataset

Experiments

Image Matching: Complex Brightness Change

Questions?

Thank you