Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Estimation of Transformations
簡韶逸 Shao-Yi Chien
Department of Electrical Engineering
National Taiwan University
Spring 2021
1
Outline
• Estimation – 2D Projective Transformation
2
[Slides credit: Marc Pollefeys]
Parameter Estimation
• 2D homographyGiven a set of (xi,xi’), compute H (xi’=Hxi)
• 3D to 2D camera projectionGiven a set of (Xi,xi), compute P (xi=PXi)
• Fundamental matrixGiven a set of (xi,xi’), compute F (xi’TFxi=0)
• Trifocal tensor Given a set of (xi,xi’,xi”), compute T
Number of Measurements Required• At least as many independent equations as degrees
of freedom required
• Example:
Hxx'=
=
11
λ
333231
232221
131211
y
x
hhh
hhh
hhh
y
x
2 independent equations / point8 degrees of freedom
4x2≥8
Approximate Solutions
• Minimal solution• 4 points yield an exact solution for H
• More points• Robust estimation algorithms, such as RANSAC
• No exact solution, because measurements are inexact (“noise”)
• Search for “best” according to some cost function• Algebraic or geometric/statistical cost
Gold Standard Algorithm
• Cost function that is optimal for some assumptions
• Computational algorithm that minimizes it is called “Gold Standard” algorithm
• Other algorithms can then be compared to it
Direct Linear Transformation(DLT)
ii Hxx = 0Hxx = ii
=
i
i
i
i
xh
xh
xh
Hx3
2
1
T
T
T
−
−
−
=
iiii
iiii
iiii
ii
yx
xw
wy
xhxh
xhxh
xhxh
Hxx12
31
23
TT
TT
TT
0
h
h
h
0xx
x0x
xx0
3
2
1
=
−
−
−
TTT
TTT
TTT
iiii
iiii
iiii
xy
xw
yw
( )Tiiii wyx = ,,x
0hA =i
→
Direct Linear Transformation(DLT)
• Equations are linear in h
0
h
h
h
0xx
x0x
xx0
3
2
1
=
−
−
−
TTT
TTT
TTT
iiii
iiii
iiii
xy
xw
yw
0hA =i
• Only 2 out of 3 are linearly independent
(indeed, 2 eq/pt)
Direct Linear Transformation(DLT)
• Equations are linear in h0hA =i
• Only 2 out of 3 are linearly independent
(indeed, 2 eq/pt)
0
h
h
h
x0x
xx0
3
2
1
=
−
−TTT
TTT
iiii
iiii
xw
yw
• Holds for any homogeneous
representation, e.g. (xi’,yi’,1)
Direct Linear Transformation(DLT)
• Solving for H
0Ah =0h
A
A
A
A
4
3
2
1
=
size A is 8x9 or 12x9, but rank 8
Trivial solution is h=09T is not interesting
1-D null-space yields solution of interest
pick for example the one with 1h =
Direct Linear Transformation(DLT)
• Over-determined solution
No exact solution because of inexact measurement
i.e. “noise”
0Ah = 0h
A
A
A
n
2
1
=
Find approximate solution
- Additional constraint needed to avoid 0, e.g.
- not possible, so minimize
1h =
Ah0Ah =
DLT Algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’},
determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) For each correspondence xi↔xi’ compute Ai. Usually
only the first two rows are needed.
(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A
(iii) Obtain SVD of A. Solution for h is the last column of V
(iv) Determine H from h
Inhomogeneous Solution
−=
−−−
'
'h~
''000'''
'''''000
ii
ii
iiiiiiiiii
iiiiiiiiii
xw
yw
xyxxwwwywx
yyyxwwwywx
Since h can only be computed up to scale,
pick hj=1, e.g. h9=1, and solve for 8-vector h~
Solve using Gaussian elimination (4 points) or using linear least-squares (more than 4 points)
However, if h9=0 this approach fails also poor results if h9 close to zero Therefore, not recommended
Note h9=H33=0 if origin is mapped to infinity
0
1
0
0
H100Hxl 0 =
=
T
Degenerate Configurations
x4
x1
x3
x2
x4
x1
x3
x2H? H’?
x1
x3
x2
x4
0Hxx = iiConstraints: i=1,2,3,4
TlxH 4
* =Define:
( ) 4444
* xxlxxH == kT
( ) 3,2,1 ,0xlxxH 4
* === iii
TThen,
H* is rank-1 matrix and thus not a homography
(case A) (case B)
If H* is unique solution, then no homography mapping xi→xi’(case B)If further solution H exist, then also αH*+βH (case A) (2-D null-space in stead of 1-D null-space)
Solutions from Lines
ii lHl T= 0Ah =
2D homographies from 2D lines
Minimum of 4 lines
Minimum of 5 points or 5 planes
3D Homographies (15 dof)
2D affinities (6 dof)
Minimum of 3 points or lines
Conic provides 5 constraints
Solutions from Mixed Type
• 2D homography• cannot be determined uniquely from the
correspondence of 2 points and 2 line
• can from 3 points and 1 line or 1 point and 3 lines
16
Cost Functions
• Algebraic distance
• Geometric distance
• Reprojection error
• Comparison
• Geometric interpretation
• Sampson error
Algebraic Distance
AhDLT minimizes
Ah=e residual vector
ie partial vector for each (xi↔xi’)
algebraic error vector
( )2
22
alg hx0x
xx0Hx,x
−−
−−==
TTT
TTT
iiii
iiii
iiixw
ywed
algebraic distance
( ) 2
2
2
1
2
21alg x,x aad += where ( ) 21
T
321 xx,,a == aaa
( ) 2222
alg AhHx,x eedi
i
i
ii === Not geometrically/statistically meaningfull, but given good normalization it works fine and is very fast (use for initialization for non-linear minimization)
Geometric Distancemeasured coordinates
estimated coordinates
true coordinates
xxx
( )2H
xH,xargminH ii
i
d =
Error in one image e.g. calibration pattern
( ) ( )221-
H
Hx,xxH,xargminH iiii
i
dd +=
Symmetric transfer error
d(.,.) Euclidean distance (in image)
( ) ( ) ( )
ii
iiii
i
ii ddii
xHx subject to
x,xx,xargminx,x,H22
x,xH,
=
+=
Reprojection error
Symmetric Transfer Error v.s. Reprojection Error
( ) ( )221- Hx,xxHx, + dd
( ) ( )22x,xxx, + dd
Symmetric Transfer Error
Reprojection Error
Comparison of Geometric and Algebraic Distances
−
−==
iiii
iiii
iiwxxw
ywwye
ˆˆ
ˆˆhA
Error in one image
( )Tiiii wyx = ,,x ( ) xHˆ,ˆ,ˆx ==T
iiii wyx
−
−
3
2
1
h
h
h
x0x
xx0TTT
TTT
iiii
iiii
xw
yw
( ) ( ) ( )222
iialgˆˆˆˆx,x iiiiiiii wxxwywwyd −+−=
( ) ( ) ( )( )( ) ii
iiiiiiii
wwd
wxwxwywyd
=
−+−=
ˆ/x,x
/ˆ/ˆˆ/ˆ/x,x
iialg
2/1222
ii
typical, but not , except for affinities 1=iw i3xhˆ =iw
➔ For affinities, DLT can minimize geometric distance
these two distance metrics are related, but not identical
Sampson Error
2
XX −
HνVector that minimizes the geometric error is the closest point on the variety to the measurement
XX
between algebraic and geometric error
XSampson error: 1st order approximation of
( ) 0XAh H ==C
( ) ( ) XH
HXH δX
XδX
+=+
CCC XXδX −= ( ) 0XH =C
( ) 0δX
X XH
H =
+
CC e−=XJδ
Find the vector that minimizes subject to Xδ e−=XJδXδ
XJ H
with
=
C
Find the vector that minimizes subject to Xδ e−=XJδXδ
Use Lagrange multipliers:
minimize
derivatives
( ) 0Jδ2λ-δδ XXX =+ eT
TT 0J2λ-δ2 X =
( ) 0Jδ2 X =+ e
λJδ X
T=
0λJJ T =+ e
( ) e1TJJλ −
−=
( ) e1TT
X JJJδ −
−=
XδXX += ( ) ee1TT
X
T
X
2
X JJδδδ −
==
Sampson Error
Sampson Error
2
XX −
HνVector that minimizes the geometric error is the closest point on the variety to the measurement
XX
between algebraic and geometric error
XSampson error: 1st order approximation of
( ) 0XAh H ==C
( ) ( ) XH
HXH δX
XδX
+=+
CCC XXδX −= ( ) 0XH =C
( ) 0δX
X XH
H =
+
CC e−=XJδ
Find the vector that minimizes subject to Xδ e−=XJδXδ
XJ H
with
=
C
( ) ee1TT
X
T
X
2
X JJδδδ −
== (Sampson error)
Sampson Approximation
A few points
(i) For a 2D homography X=(x,y,x’,y’)
(ii) is the algebraic error vector
(iii) is a 2x4 matrix, e.g.
(iv) Similar to algebraic error in fact, same as Mahalanobis distance
(v) Sampson error independent of linear reparametrization
(cancels out in between e and J)
(vi) Must be summed for all points
(vii) Close to geometric error, but much fewer unknowns
( ) ee1TT2
X JJδ −
=
eee T2 =
2
JJT e
( ) 3121
3T2T
11 /hxhx hyhwxywJ iiiiii+−=+−=X
J H
=
C
( )XHCe =
( )−
ee1TT JJ
Statistical Cost Function and Maximum Likelihood Estimation
• Optimal cost function related to noise model of measurement
• Assume zero-mean isotropic Gaussian noise (assume outliers removed)
( ) ( ) ( )222/xx,
2πσ2
1xPr de−=
( ) ( ) ( )22
i 2xH,x /
2iπσ2
1H|xPr
ide
i
−=
( ) ( ) +−= constantxH,xH|xPrlog2
i2iσ2
1id
Error in one image
Maximum Likelihood Estimate
( ) 2
i xH,x id Equivalent to minimizing the geometric error function
Statistical Cost Function and Maximum Likelihood Estimation
• Optimal cost function related to noise model of measurement
• Assume zero-mean isotropic Gaussian noise (assume outliers removed)
( ) ( ) ( )222/xx,
2πσ2
1xPr de−=
( )( ) ( ) ( )22
i2
i 2xH,xx,x /
2iπσ2
1H|xPr
+−
=ii dd
ei
Error in both images
Maximum Likelihood Estimate
( ) ( )2i
2
i x,xx,x ii dd +Equivalent to minimizing the reprojection error function
Mahalanobis Distance
• General Gaussian case
Measurement X with covariance matrix Σ
( ) ( )XXXXXX 1T2
−−=− −
22
XXXX−+−
Error in two images (independent)
22
XXXXii
ii
i
ii −+−
Varying covariances
Invariance to Transforms ?
xTx~ =
Txx~ =Hxx = x~H
~x~ =
TH~
TH 1-?
=
TxH~
xT =
TxH~
Tx -1=
will result change?
for which algorithms? for which transformations?
Non-invariance of DLT
Given and H computed by DLT,
and
Does the DLT algorithm applied to
yield ?
iiii xTx~,Txx~ ==
ii xx
ii x~x~ -1HTTH
~=
( ) iiiiie TxHTTxTx~H~
x~~ -1== ( ) iii e** THxxT ==
( ) ( ) hA,R~,~h~
A~
i2121i seesee ===TT
Effect of change of coordinates on algebraic error
=
10
tsRT
=
ss
Rt-
0RT
T
*
for similarities
so
( ) ( )iiii sdd x~H~
,x~Hx,x algalg=
Non-invariance of DLT
(T*: cofactor matrix)
Non-invariance of DLT
Given and H computed by DLT,
and
Does the DLT algorithm applied to
yield ?
iiii xTx~,Txx~ ==
ii xx
ii x~x~ -1HTTH
~=
( )
( )
( ) 1H~
subject tox~H~
,x~minimize
1H subject tox~H~
,x~minimize
1H subject toHx,xminimize
2
alg
2
alg
2
alg
=
=
=
i
ii
i
ii
i
ii
d
d
d
Invariance of Geometric Error
( ) ( ) ( )
( )ii
iiiiii
sd
ddd
Hx,x
HxT,xTTxHTT,xTx~H~
,x~ -1
=
==
Given and H,
and
Assume T’ is a similarity transformations
,xTx~,Txx~ iiii==
ii xx
,x~x~ ii -1HTTH
~=
Normalizing Transformations
• Since DLT is not invariant,
what is a good choice of coordinates?e.g. Isotropic scaling
• Translate centroid to origin
• Scale to a average distance to the origin
• Independently on both images2
1
norm
100
2/0
2/0
T
−
+
+
= hhw
whwOr
Importance of Normalization
0
h
h
h
0001
1000
3
2
1
=
−−−
−−−
iiiiiii
iiiiiii
xyxxxyx
yyyxyyx
~102 ~102 ~102 ~102 ~104 ~104 ~10211
orders of magnitude difference!
Without normalization With normalization
Normalized DLT AlgorithmObjective
Given n≥4 2D to 2D point correspondences {xi↔xi’},
determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) Normalize points
(ii) Apply DLT algorithm to
(iii) Denormalize solution
,x~x~ ii
inormiinormi xTx~,xTx~ ==
norm
-1
norm TH~
TH =
Employ this algorithm instead of the original DLT algorithm!• More accurate• Invariant to arbitrary choices of the scale and coordinate origin
Normalization is also called pre-conditioning
Iterative Minimization MethodsRequired to minimize geometric error
(i) Often slower than DLT
(ii) Require initialization
(iii) No guaranteed convergence, local minima
(iv) Stopping criterion required
Therefore, careful implementation required:
(i) Cost function
(ii) Parameterization (minimal or not)
(iii) Cost function ( parameters )
(iv) Initialization
(v) Iterations
Parameterization
Parameters should cover complete space and allow efficient estimation of cost
• Minimal or over-parameterized? e.g. 8 or 9
(minimal often more complex, also cost surface)
(good algorithms can deal with over-parameterization)
(sometimes also local parameterization)
• Parametrization can also be used to restrict transformation to particular class, e.g. affine
Function Specifications
(i) Measurement vector X N with covariance Σ
(ii) Set of parameters represented by vector P M
(iii) Mapping f : M → N. Range of mapping is surface S representing allowable measurements
(iv) Cost function: squared Mahalanobis distance
Goal is to achieve , or get as close as
possible in terms of Mahalanobis distance
( ) ( )( ) ( )( )PXPXPX 1T2fff −−=− −
( ) XP =f
Error in one image
( ) 2
i xH,x id
( )nf Hx,...,Hx,Hxh: 21→
( )hX f−
( ) ( )221- Hx,xxH,x iiii
i
dd +
Symmetric transfer error
( )nnf Hx,...,Hx,Hx,xH,...,xH,xHh: 21
-1
2
-1
1
-1 →
( )hX f−
Reprojection error
( ) ( )2i
2
i x,xx,x ii dd +
( )hX f−
X composed of 2n inhomogeneous coordinates of the points 𝑥𝑖
′
X composed of 4n-vector inhomogeneous
coordinates of the points𝑥𝑖 and 𝑥𝑖′
X composed of 4n-vector
Initialization
• Typically, use linear solution
• If outliers, use robust algorithm
• Alternative, sample parameter space
Iteration Methods
Many algorithms exist
• Newton’s method
• Levenberg-Marquardt
• Powell’s method
• Simplex method
Levenberg-Marquardt Algorithm
For a mapping function f with parameter vector
To an estimated measurement vector
We want to find p that can minimize , where
f(p) can be approximated as
with small and J=
➔Find to minimize
44
Levenberg-Marquardt Algorithm
Find to minimize
The least-square solution:
Augmented normal equation (with damping term ):
45
Hessian
Gold Standard AlgorithmObjective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine
the Maximum Likelyhood Estimation of H
(this also implies computing optimal xi’=Hxi)
Algorithm
(i) Initialization: compute an initial estimate using normalized DLT
or RANSAC
(ii) Geometric minimization of -Either Sampson error:
● Minimize the Sampson error
● Minimize using Levenberg-Marquardt over 9 entries of h
or Gold Standard error:
● compute initial estimate for optimal {xi}
● minimize cost over {H,x1,x2,…,xn}
● if many points, use sparse method
( ) ( )2i
2
i x,xx,x ii dd +
Robust Estimation
• What if set of matches contains gross outliers?
RANSAC: RANdom SAmple Consensus
Objective
Robust fit of model to data set S which contains outliers
Algorithm
(i) Randomly select a sample of s data points from S and
instantiate the model from this subset.
(ii) Determine the set of data points Si which are within a
distance threshold t of the model. The set Si is the
consensus set of samples and defines the inliers of S.
(iii) If the subset of Si is greater than some threshold T, re-
estimate the model using all the points in Si and terminate
(iv) If the size of Si is less than T, select a new subset and
repeat the above.
(v) After N trials the largest consensus set Si is selected, and
the model is re-estimated using all the points in the
subset Si
Distance Threshold
Choose t so probability for inlier is α (e.g. 0.95)
• Often empirically
• Zero-mean Gaussian noise σ then follows
distribution with m=codimension of model
2
⊥d2
m
(dimension+codimension=dimension space)
Codimension Model t 2
1 Line (l), Fundamental matrix (F) 3.84σ2
2 Homography (H), Camera Matrix (P) 5.99σ2
3 Trifocal tensor (T) 7.81σ2
How Many Samples?
Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99
( ) ( )( )sepN −−−= 11log/1log
( )( ) peNs
−=−− 111
proportion of outliers e
s 5% 10% 20% 25% 30% 40% 50%2 2 3 5 6 7 11 173 3 4 7 9 11 19 354 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 1177
Acceptable Consensus Set?
• Typically, terminate when inlier ratio reaches expected ratio of inliers
( )neT −= 1
Adaptively Determining the Number of Samples
e is often unknown a priori, so pick worst case, e.g. 50%, and adapt if more inliers are found, e.g. 80% would yield e=0.2
• N=∞, sample_count =0
• While N >sample_count repeat• Choose a sample and count the number of inliers
• Set e=1-(number of inliers)/(total number of points)
• Recompute N from e
• Increment the sample_count by 1
• Terminate( ) ( )( )( )s
epN −−−= 11log/1log
Robust Maximum Likelyhood Estimation
Previous MLE algorithm considers fixed set of inliers
Better, robust cost function (reclassifies)
( ) ( )
== ⊥
outlier
inlier ρ with ρ
222
222
i tet
teeed iR
Other Robust Algorithms
• RANSAC maximizes number of inliers
• LMedS minimizes median error
Automatic Computation of H
Objective
Compute homography between two images
Algorithm
(i) Interest points: Compute interest points in each image
(ii) Putative correspondences: Compute a set of interest point
matches based on some similarity measure
(iii) RANSAC robust estimation: Repeat for N samples
(a) Select 4 correspondences and compute H
(b) Calculate the distance d⊥ for each putative match
(c) Compute the number of inliers consistent with H (d⊥<t)
Choose H with most inliers
(iv) Optimal estimation: re-estimate H from all inliers by minimizing ML
cost function with Levenberg-Marquardt
(v) Guided matching: Determine more matches using prediction by
computed H
Optionally iterate last two steps until convergence
Determine Putative Correspondences
• Compare interest points
Similarity measure:
• SAD, SSD, ZNCC on small neighborhood
• If motion is limited, only consider interest points with similar coordinates
• More advanced approaches exist, based on invariance…• Such as SIFT
Example: robust computation
Interest points
(500/image)
Putative correspondences (268)
Outliers (117)
Inliers (151)
Final inliers (262)