Distinctive Image Featuresfrom Scale-Invariant Keypoints

David Lowe

object instance recognition (matching)

Photosynth

Challenges

• Scale change

• Rotation

• Occlusion

• Illumination

……

Strategy

• Matching by stable, robust and distinctive local features.

• SIFT: Scale Invariant Feature Transform; transform image data into scale-invariant coordinates relative to local features

SIFT

• Scale-space extrema detection

• Keypoint localization

• Orientation assignment

• Keypoint descriptor

Scale-space extrema detection

• Find the points, whose surrounding patches (with some scale) are distinctive

• An approximation to the scale-normalized Laplacian of Gaussian

Maxima and minima in a 3*3*3 neighborhood

Keypoint localization

• There are still a lot of points, some of them are not good enough.

• The locations of keypoints may be not accurate.

• Eliminating edge points.

(1)

(2)

(3)

Eliminating edge points

• Such a point has large principal curvature across the edge but a small one in the perpendicular direction

• The principal curvatures can be calculated from a Hessian function

• The eigenvalues of H are proportional to the principal curvatures, so two eigenvalues shouldn’t diff too much

Orientation assignment

• Assign an orientation to each keypoint, the keypoint descriptor can be represented relative to this orientation and therefore achieve invariance to image rotation

• Compute magnitude and orientation on the Gaussian smoothed images

Orientation assignment

• A histogram is formed by quantizing the orientations into 36 bins;

• Peaks in the histogram correspond to the orientations of the patch;

• For the same scale and location, there could be multiple keypoints with different orientations;

Feature descriptor

Feature descriptor

• Based on 16*16 patches

• 4*4 subregions

• 8 bins in each subregion

• 4*4*8=128 dimensions in total

Application: object recognition

• The SIFT features of training images are extracted and stored

• For a query image

1. Extract SIFT feature

2. Efficient nearest neighbor indexing

3. 3 keypoints, Geometry verification

Extensions

• PCA-SIFT

1. Working on 41*41 patches

2. 2*39*39 dimensions

3. Using PCA to project it to 20 dimensions

Surf

• Approximate SIFT

• Works almost equally well

• Very fast

Conclusions

• The most successful feature (probably the most successful paper in computer vision)

• A lot of heuristics, the parameters are optimized based on a small and specific dataset. Different tasks should have different parameter settings.

• Learning local image descriptors (Winder et al 2007): tuning parameters given their dataset.

• We need a universal objective function.

comments

• Ian: “For object detection, the keypoint localization process can indicate which locations and scales to consider when searching for objects”.

• Mert: “uniform regions may be quite informative when detecting

some types of ojbects , but SIFT ignore them”

• Mani: “region detectors comparison”• Eamon:” whether one could go directly to a surface

representation of a scene based on SIFT features “