Lecture 5&6 –Finding Features, Affine Invariance,...

CAP5415-ComputerVisionLecture5and6-FindingFeatures,

AffineInvariance,SIFT

UlasBagcibagci@ucf.edu

Lecture5&6–FindingFeatures,AffineInvariance,SIFT

Outline

• ConceptofScale– Pyramids– Scale-spaceapproachesbriefly

• Scaleinvariantregionselection• SIFT:animageregiondescriptor

• DavidLowe,IJCV2004paper[Pleasereadit!]

Reminder:Motivation

• ImageMatching– Fundamentalaspectofmanyproblems

• ObjectRecognition• 3DStructures• StereoCorrespondence• MotionTracking• ….

Reminder:Motivation

• ImageMatching– Fundamentalaspectofmanyproblems

• ObjectRecognition• 3DStructures• StereoCorrespondence• MotionTracking• ….

Whatarethedesiredfeaturestoconductthesetasks?

Review oftheCornerDetectionLecture5&6–FindingFeatures,AffineInvariance,SIFT

rotation

translation

scaling

Cornersareinvariantto

• Theextrema inasignalanditsfirstafewderivatives provideausefulgeneralpurposedescriptionformanykindsofsignals– Edges– Corners

• Inphysics,objectsliveonarangeofscales.

• Inphysics,objectsliveonarangeofscales.• Neighborhoodoperationscanonlyextractlocalfeatures(atascaleofatmostafewpixels),however,imagescontaininformationatlargerscales

Scale-Space

• Anygivenimagecancontainobjectsthatexistatscalesdifferentfromotherobjectsinthesameimage

Scale-Space

• Or,thatevenexistatmultiplescalessimultaneously

Scale-Space

• Or,thatevenexistatmultiplescalessimultaneously

Multi-scaleRepresentationLecture5&6–FindingFeatures,AffineInvariance,SIFT

SmallScaleFeatures

SmallerFilterMasks

LargeScaleFeatures

LargerFilterMasks

Multi-scaleRepresentationLecture5&6–FindingFeatures,AffineInvariance,SIFT

SmallScaleFeatures

SmallerFilterMasks

LargeScaleFeatures

LargerFilterMasks

ComputationalCostincreases!

Doublingthescaleleadstofour-foldincreaseinthenumberofoperationsin2D.

Multi-ScaleRepresentation:Pyramid

• Pyramid isonewaytorepresentimagesinmulti-scale– Pyramidisbuiltbyusingmultiplecopiesofimage.– Eachlevelinthepyramidis1/4ofthesizeofpreviouslevel.

– Thelowestlevelisofthehighestresolution.– Thehighestlevelisofthelowestresolution.

Multi-ScaleRepresentation:PyramidLecture5&6–FindingFeatures,AffineInvariance,SIFT

Pyramidcancaptureglobalandlocalfeatures

Gaussian PyramidsLecture5&6–FindingFeatures,AffineInvariance,SIFT

17Source: Forsyth

Laplacian PyramidsLecture5&6–FindingFeatures,AffineInvariance,SIFT

18Source: Forsyth

-2 1 1-2

Laplacian ofGaussian(LoG):GaussianfirsttosmoothimagesthenperformLaplacian operation

52(G� ⇤ I) = I ⇤ 52G�

(x, y) = � x

2⇡�4e

�(x2+y

2)/(2�2)

(x, y) = � x

2⇡�4e

�(x2+y

2)/(2�2)

(x, y) =1

2⇡�4

(x2 + y

2 � 2�2)

�(x2+y

2)/(2�2)

Repeatsamederivativeforycomponent,andthentakesecondderivatives.

Laplacian PyramidConstructionLecture5&6–FindingFeatures,AffineInvariance,SIFT

Decimation andInterpolationLecture5&6–FindingFeatures,AffineInvariance,SIFT

LowPass filter(i.e.,Gaussian)

y(n) = x(n) ⇤ h(n) =X

h(k)x(n� k)

y(n) = x(n) ⇤ h(n) =X

h(k)x(n� k)

z(n) = y(2n)

y(n) = x(n) ⇤ h(n) =X

h(k)x(n� k)

z(n) = y(2n)

z(n) =X

h(k)x(2n� k)

DifferenceofGaussian(DoG)

• ItisacommonapproximationofLoG (betterruntime).

D�,↵(x, y) = L(x, y,↵)� L(x, y,↵�)

• ItisthedifferencebetweenablurredcopyofimageIandanevenmoreblurredcopyofI.

D�,↵(x, y) = L(x, y,↵)� L(x, y,↵�)

• ItisthedifferencebetweenablurredcopyofimageIandanevenmoreblurredcopyofI.

D�,↵(x, y) = L(x, y,↵)� L(x, y,↵�)

5G�(x, y) ⇡G↵�(x, y)�G�(x, y)

(↵� 1)�2

ScaleSelection-AutomatedLecture5&6–FindingFeatures,AffineInvariance,SIFT

• Find scale that gives local maxima of some function f in both position and scale.

region size

Image 1f

region size

Image 2

s1 s2K. Grauman,

GoodFunction?Lecture5&6–FindingFeatures,AffineInvariance,SIFT

• A “good” function for scale detection:has one stable sharp peak

region size

Good !

• Forusualimages:agoodfunctionwouldbeaonewhichrespondstocontrast(sharplocalintensitychange)

PatchSizeCorrespondingtoScaleLecture5&6–FindingFeatures,AffineInvariance,SIFT

)),(( )),((11

ss ¢¢= xIfxIfmm iiii !!

How to find corresponding patch sizes?

36K.Grauman,B.Leibe)),((

mii ! )),((1

sxIfmii ¢!

mii ! )),((1

sxIfmii ¢!

mii ! )),((1

sxIfmii ¢!

mii ! )),((1

sxIfmii ¢!

mii ! )),((1

sxIfmii ¢!

• Functionresponsesforincreasingscale(scalesignature)

K.Grauman,B.Leibe)),((

mii ! )),((1

s ¢¢xIfmii !

NormalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

ScaleInvariantFeatureTransform(SIFT)

• Lowe.,D.2004,IJCV

43cited>43K

• Showsexistence,importance,andvalueofinvarianttypesofdetector

• Demonstratestherichnessthatfeaturedescriptorscanbringtofeaturematching

InsteadofusingLoG (Laplacian ofGaussian) likeinHarrisandHessian-basedoperators,SIFTuses

DoG (DifferenceofGaussian).

• Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters

OverallProcedureataHighLevelLecture5&6–FindingFeatures,AffineInvariance,SIFT

Scale-SpaceExtremaDetection

Searchovermultiplescalesandimagelocations

KeyPointLocalization

Fitamodeltodeterminelocationandscale.SelectKeyPoints basedona

measureofstability.

OrientationAssignment

Computebestorientation(s)foreachkeyPoint region.

KeyPointDescription

Uselocalimagegradientsatselectedscaleandrotationtodescribeeach

keyPoint region.

SIFT DescriptorLecture5&6–FindingFeatures,AffineInvariance,SIFT

Basic idea:• Take n x n (i.e., n=16) square window (around a feature/interest point)• Divide them into m x m (i.e., m=4) cells• Compute gradient orientation for each cell• Create histogram over edge orientations weighted by magnitude

0 2anglehistogram

16histogramsx8orientations=128features

Full version• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)• Compute an orientation histogram for each cell• 16 cells * 8 orientations = 128 dimensional descriptor• Threshold normalize the descriptor:

suchthat:

PropertiesofSIFTLecture5&6–FindingFeatures,AffineInvariance,SIFT

Extraordinarilyrobustmatchingtechnique

– Canhandlechangesinviewpoint– Canhandlesignificantchangesinillumination– Efficient(realtime)– Lotsofcodeavailable

• http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT

Revisit SIFTStepsLecture5&6–FindingFeatures,AffineInvariance,SIFT

(1) Scale-space extrema detection– Extract scale and rotation invariant interest points (i.e.,

keypoints). (2) KeyPoint localization

– Determine location and scale for each interest point.– Eliminate “weak” keypoints

(3) Orientation assignment– Assign one or more orientations to each keypoint.

(4) KeyPoint descriptor– Use local image gradients at the selected scale.

(1)Scale-SpaceExtrema DetectionLecture5&6–FindingFeatures,AffineInvariance,SIFT

Wewanttofindpointsthatgiveusinformationabouttheobjectsintheimage.

Theinformationabouttheobjectsisintheobject’sedges.

Wewillrepresenttheimageinawaythatgivesustheseedgesasthisrepresentationsextrema points.

• Harris-Laplace

Find local maxima of:– Harris detector in space – LoG in scale

• SIFT

Find local maxima of:– Hessian in space – DoG in scale

Hessian

¬ Harris ®

• Extractlocalextrema(i.e.,minimaormaxima)inDoG pyramid.

-Compareeachpointtoits8neighborsatthesamelevel,9neighborsinthelevelabove,and9neighborsinthelevelbelow(i.e.,26total).

( )skD

( )s2kD

• Laplacian of Gaussian kernel– Scale normalized (x by scale2)– Proposed by Lindeberg

• Scale-space detection– Find local maxima across scale/space– A good “blob” detector

(2)KeyPoint Localization

• There are still a lot of points, some of them are not good enough.

• The locations of keypoints may be not accurate.

• Eliminating edge points.

(2)KeyPoint LocalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

� Reject (1) pointswithlowcontrast(flat)(2)poorlylocalizedalonganedge(edge)

(2)KeyPoint LocalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

� Reject (1) pointswithlowcontrast(flat)(2)poorlylocalizedalonganedge(edge)

Reject

WhereandpracticallySIFTusesr=10.

|H| <(r + 1)2

r =�1

InaccurateKeyPoint LocalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

xSampling

DetectedExtrema

TrueExtrema

Poorcontrast

InaccurateKeyPoint LocalizationLecture5&6–FindingFeatures,AffineInvariance,SIFT

TheSolution:

( ) xxDxx

Taylorexpansion:

Minimizetofindaccurateextrema:

xDx !!

Brown&Lowe2002

Ifoffsetfromsamplingpointislargerthan0.5-Keypoint shouldbeinadifferentsamplingpoint.

Localextremas

Localextremas Removelowcontrastfeatures

Localextremas Removelowedges

SIFTDescriptorLecture5&6–FindingFeatures,AffineInvariance,SIFT

(3)OrientationAssignmentLecture5&6–FindingFeatures,AffineInvariance,SIFT

m(x, y) = (L(x +1, y)− L(x −1, y))2 + (L(x, y +1)− L(x, y −1))2

θ(x, y) = a tan2((L(x, y +1)− L(x, y −1)) / (L(x +1, y)− L(x −1, y)))

36 bins (i.e., 10o per bin)

• Histogramentriesare weighted by(i)gradientmagnitudeand(ii)aGaussianfunctionwithσ equalto1.5timesthescaleofthekeypoint.

L(x, y,σ ) = G(x, y,σ )* I(x, y)

Create histogram of gradient directions, within a region around the keyPoint, at selected scale:

(3)OrientationAssignment

(4)KeyPoint DescriptorLecture5&6–FindingFeatures,AffineInvariance,SIFT

• Partial Voting: distribute histogram entries into adjacent bins (i.e., additional robustness to shifts)– Each entry is added to all bins, multiplied by a weight of 1-d,

where d is the distance from the bin it belongs.

(4)KeyPoint DescriptorLecture5&6–FindingFeatures,AffineInvariance,SIFT

• Descriptordependsontwomainparameters:– (1)numberoforientations– nxnarrayoforientationhistograms

EvaluatingResultsLecture5&6–FindingFeatures,AffineInvariance,SIFT

Howcanwemeasuretheperformanceofafeaturematcher?

false positive rate

truepositive

rate# true positives matched

# true positives

0.1# false positives matched

# true negatives

features that really do have a match

the matcher correctlyfound a match

the matcher said yes when the right answer was no

EvaluatingResultsLecture5&6–FindingFeatures,AffineInvariance,SIFT

Howcanwemeasuretheperformanceofafeaturematcher?

false positive rate

truepositive

rate# true positives matched

# true positives

0.1# false positives matched

# true negatives

features that really do have a match

the matcher correctlyfound a match

the matcher said yes when the right answer was no

ROC curve (“Receiver Operator Characteristic”)

ChangeofIlluminationLecture5&6–FindingFeatures,AffineInvariance,SIFT

Change of brightness => doesn’t effect gradients (difference of pixels value).

Change of contrast => doesn’t effect gradients (up to normalization).Saturation (non-linear change of illumination) =>affects magnitudes much more than orientation.=> Threshold gradient magnitudes to 0.2 and

renormalize.

Illuminationinvariance?Lecture5&6–FindingFeatures,AffineInvariance,SIFT

1. Beforeyoustart,youcannormalizetheimages.2. Differencebasedmetricscanbeused(Haar,SIFT,…)

• Extractfeatures

• Extractfeatures• Computeputativematches

• Extractfeatures• Computeputativematches• Loop:

– Hypothesize transformationT(smallgroupofputativematchesthatarerelatedbyT)

– Verifytransformation(searchforothermatchesconsistentwithT)

ObjectRecognitionLecture5&6–FindingFeatures,AffineInvariance,SIFT

Fortrainingimages:ExtractingkeypointsbySIFT.Creatingdescriptorsdatabase.

Performingdetailedgeometricfitcheckforeachcluster.

Forqueryimages:ExtractingkeypointsbySIFT.Foreachdescriptor- findingnearestneighborinDB.

Findingclusterofat-least3keypoints.

ImageRegistrationLecture5&6–FindingFeatures,AffineInvariance,SIFT

Lecture 5&6 –Finding Features, Affine Invariance,...

Documents

Affine-Resistant Watermarking

Materi7 Grafik(Trandform Affine)

Establishing Factorial Invariance for Multiple-Group ......invariance (Cheung & Rensvold, 2002). 22.1 Configural Invariance example, suppose that a researcher is measuring positive

Hinh hoc-affine

Affine Transformation. Affine Transformations In this lecture, we will continue with the discussion of the remaining affine transformations and composite

Géométrie Affine

Invariants to affine transform What is affine transform ?

Structuralism,Invariance, andUnivalence

Scale Invariance

GR Affine Variation

Affine cluster algebras

Affine Parameterisation 1

Relativistic Invariance (Lorentz invariance)

Affine Versus Non-Affine Fibril Kinematics in Collagen Networks

Invariance to Affine-Permutation Distortions...and the Karcher mean given multiple shapes–representing possibly different afﬁne-permutation distorted objects. 2. Shape Space: A

Invariance, Symmetries and Structural Realism Friedel ...philsci-archive.pitt.edu › 3643 › 1 › Invariance,Symmetries...pdf · Invariance, Symmetries and Structural Realism

Espace Affine

Affine Transformation

affine cipher

Affine Model by Piazzesi