Embed Size (px)
Citation preview
A Software for Complete Calibration ofMulticamera Systems
Tomas Svobodawith contributions from D. Martinec, T. Pajdla, O. Chum, T. Werner,
and J. Bouguet
Czech Technical University, Faculty of Electrical Engineering
Center for Machine Perception, Prague, Czech Republic
http://cmp.felk.cvut.cz/~svoboda
Computer Vision Lab, ETH Zurich
2/45Outline
Motivation
Problem definition
Proposed solution
Results
Applications
6/45Calibration
many tasks can be accomplished without knowing
anything about the cameras
however, many more when we know . . .
9/45
. . . camera internal parametersfrom geometry to pixels
y xy xz
v uNormalized image plane
Image planef1
Xu
xC
11/45Camera calibration is an old problem
for photogrammetrists (even older problem)
in computer vision
many methods exist
13/45
Classical approaches — plate at severalpositions
http://www.vision.caltech.edu/bouguetj/calib doc/
14/45Classical methods — revisited
Pros:
many methods (and free codes)
precise, even for complicated camera models
14/45Classical methods — revisited
Pros:
many methods (and free codes)
precise, even for complicated camera models
Cons (for multicamera systems):
many cameras → hand work is not an option
large working volume to fill → big calibration
objects/plates
15/45Our solution — overview
We assume at least approximately synchronized multicamera
(N ≥ 3) setup.
use 1-point calibration object easily detectable in images
wave the calibration point through the working volume
freely
this will create a virtual calibration object (but the 3D
position unknown!)
apply theoretical results from self-calibration field
estimate as complicated camera model as reasonable
validate the results
16/45Multiple cameras — Geometry
C1
C2
C3
C4
Xj
u1
ju2
j
u3
j
u4
j
P1
P2
P3
P4
Problem definition:
From uij points, for which λ
iju
ij = P
iXj holds
estimate Euclidean projection matrices Pi
and coordinates of the 3D points Xj
17/45Pinhole camera model
λij
uij
vij
1
= λiju
ij = PiXj , λi
j ∈ R+
j index points
i index camera
λij projective depths
uij point projections (we find them in images)
Xj 3D points (we do not know the positions!)
Pi camera matrices
18/45Multicamera linear model
Ws =
λ11
u11
v11
1
· · · λ1n
u1n
v1n
1
... ... ...
λm1
um1
vm1
1
· · · λmn
umn
vmn
1
=
P1
...
Pm
3m×4︸ ︷︷ ︸
P
[X1 · · ·Xn]4×n︸ ︷︷ ︸X
Self-calibration (Euclidean stratification)
Ws = PX = PH︸︷︷︸ H−1X︸︷︷︸ = PX ,
19/45What the software does:
1. Finds the projections uij of the laser pointer in the images.
19/45What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
19/45What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to
make scaled measurement matrix Ws complete.
19/45What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to
make scaled measurement matrix Ws complete.
4. Performs the rank 4 factorization of the matrix Ws to get
projective shape and motion and upgrades them to
Euclidean ones.
19/45What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to
make scaled measurement matrix Ws complete.
4. Performs the rank 4 factorization of the matrix Ws to get
projective shape and motion and upgrades them to
Euclidean ones.
5. Estimates the parameters of the non-linear distortion
19/45What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to
make scaled measurement matrix Ws complete.
4. Performs the rank 4 factorization of the matrix Ws to get
projective shape and motion and upgrades them to
Euclidean ones.
5. Estimates the parameters of the non-linear distortion
6. Optionally, if some true 3D information is known, aligns the
computed Euclidean structures with a world system.
19/45What the software does:
1. Finds the projections uij of the laser pointer in the images.
2. Discards misdetected points by pairwise RANSAC analysis.
3. Estimates projective depths λij and fills the missing points to
make scaled measurement matrix Ws complete.
4. Performs the rank 4 factorization of the matrix Ws to get
projective shape and motion and upgrades them to
Euclidean ones.
5. Estimates the parameters of the non-linear distortion
6. Optionally, if some true 3D information is known, aligns the
computed Euclidean structures with a world system.
Many cross-validation steps inside.
20/45Calibration object
A very standard laser pointer with a piece of transparent
plastic attached.
21/45Finding points
Needs to be a bit more clever than a simple thresholding
Statistical analysis of the images (almost) solves it.
22/45Finding points
Sub-pixel accuracy is desirable
Interpolated ROI
20 40 60
10
20
30
40
50
60
050
100
0
50
1000
0.02
0.04
Correlation coeffs
020
40
0
20
40100
150
200
250
Active ROI
020
40
0
20
400
0.5
1
1.5
x 10−3PSF approx by 2D Gaussian
/local/MultiCam/Data/20030615H
oengg/arctic1/arctic1.pvi.0001.jpg original
312 314 316 318 320 322 324 326 328
294
296
298
300
302
304
306
308
310
Around 100ms per image.
24/45We know
Ws =
λ11
u11
v11
1
· · · λ1n
u1n
v1n
1
... ... ...
λm1
um1
vm1
1
· · · λmn
umn
vmn
1
=
P1
...
Pm
3m×4
[X1 · · ·Xn]4×n
Ws = PX = PHH−1X = PX ,
However, some [uij, v
ij]> may be missing!
25/45
Estimation of λij
(Sturm & Triggs ECCV96)
uses the epipolar geometry
C1 C2
Xu1 u2l1 l2e1 e2
λij =
(eik × uij) · (Fikuk
j)‖eik × ui
j‖2 λkj
26/45We know
Ws =
λ11
u11
v11
1
· · · λ1n
u1n
v1n
1
... ... ...
λm1
um1
vm1
1
· · · λmn
umn
vmn
1
=
P1
...
Pm
3m×4
[X1 · · ·Xn]4×n
Ws = PX = PHH−1X = PX ,
However, some [uij, v
ij]> and λi
j may be missing!
28/45We know
Ws =
λ11
u11
v11
1
· · · λ1n
u1n
v1n
1
... ... ...
λm1
um1
vm1
1
· · · λmn
umn
vmn
1
=
P1
...
Pm
3m×4
[X1 · · ·Xn]4×n
Ws = PX = PHH−1X = PX ,
29/45Rank–4 factorization
Ws =
P1
...
Pm
3m×4
[X1 · · ·Xn]4×n
So, matrix Ws should have rank at most 4
Ws = USV> P1
...
Pm
3m×4
[X1 · · ·Xn]4×n = (U√S4)(
√S4V
>)
where S4 is the S with only 4 biggest diagonal values, rest is
zeroed.
30/45We know
Ws = PX = PHH−1X = PX ,
We must find a 4× 4 matrix H which upgrades the projective
structures P, X to metric ones, P, X.
31/45
Euclidean stratification(Pollefeys et al, Hartley, . . . )
based on the idea of absolute quadric (conic)
Pi = µi
[KiRi Kiti
]
PiΩ∞Pi> ∼ KiKi>
where
Ω∞ =
1 0 0 00 1 0 00 0 1 00 0 0 0
32/45Euclidean stratification cont.
absolute conic exists also in the projective world!
KiKi> ∼ (PiH−1)(HΩ∞H>)(H−>Pi>)
KiKi> ∼ PiΩ∞Pi>
We know the projective Pi. The projective Ω∞ is 4× 4symmetric.
Once Ω∞ is known, then we can compute H from
Ω∞ = HΩ∞H>
by eigenvalue decomposition and get the sought Euclidean
structures Pi = PiH and Xj = H−1Xj.
33/45
Euclidean stratification — Example ofsolution
assume everything is known except focal lenghts
Ki =
f i 0 ui0
0 αif i vi0
0 0 1
→ KiKi> =
f i2 0 00 f i2 00 0 1
Remember that KiKi> ∼ PiΩ∞Pi>
(PiΩ∞Pi>)11 − (PiΩ∞Pi>)22 = 0
(PiΩ∞Pi>)12 = 0
(PiΩ∞Pi>)13 = 0
(PiΩ∞Pi>)23 = 0
Each camera contributes by 4 contraints.
34/45We have the metric linear model
Ws = PX
Estimation of non-linear distortion starts from
Xj ↔ uij
correspondences. We use the CalTech package
http://www.vision.caltech.edu/bouguetj/calib doc/
Then it goes back, adapt parameters and . . .
35/45Aligning the results with the world
−2
0
2
−6
−4
−2
0
2
−6
−4
−2
0
2
4
10
11
13
1
9
2
8
12
reconstructed points/camera setup only inliers are used
3
16
15
14
7
4
6
5
−3−2
−10
12
3
−2
−1
0
1
2
3
4
10
9
8
7
6
11
16
Graphical Output Validation: View from the top camera
12
13
15
14
1
2
3
4
5
User provides some 3D information. Example: “Cameras No.
11,13,15 define the xy plane”.
36/45Results — Filling points
0 100 200 300 400 500 600 700
0
100
200
300
400
500
measured, o, vs reprojected, +, 2D points (camera: 4)
−400 −200 0 200 400 600 800 1000 1200 1400 1600−500
0
500
1000
measured, o, vs reprojected, +, 2D points (camera: 40)
The calibration “point” needs not to be visible in all cameras!
37/45Results — Calibrated setups
−3 −2 −1 0 1 2 3
−2
−1
0
1
2
3
10
9
8
7
6
11
16
Graphical Output Validation: View from the top camera
12
15
13
14
5
3
4
2
1−1 0 1
−2
−1
0
1
2
3 71
70
80
Graphical Output Validation: Aligned data
72
81
82
38/45Results — Linear model
0 100 200 300 400 500 600
0
100
200
300
400
500measured, o, vs reprojected, +, 2D points (camera: 12)
0 2 4 6 8 10 12 14 16 180
1
2
3
4
5
6
7
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
39/45Results — Complete model
0 100 200 300 400 500 600 700
0
100
200
300
400
500
measured, o, vs reprojected, +, 2D points (camera: 12)
0 2 4 6 8 10 12 14 16 180
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
Very fine results from (almost) nothing!
40/45Results — Simple setup
−3−2−101
−1
0
1
2
3
4
5
6
4
reconstructed points/camera setup only inliers are used 3
2
2 3 40
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
41/45
Application example — volumetricreconstruction
I know, it is just toy example. Still, it shows that the metric is
OK.
45/45Summary
waving the point object is the only hand work required
no user interaction
complete calibration of 16 camera setup may be done in
60-90 minutes (95% computation)
Codes, sample data, papers, etc. downloadable from
http://cmp.felk.cvut.cz/˜svoboda/SelfCal
y xy xz
v uNormalized image plane
Image planef1
Xu
xC
C1
C2
C3
C4
Xj
u1
ju2
j
u3
j
u4
j
P1
P2
P3
P4
Problem definition:
From uij points, for which λ
iju
ij = P
iXj holds
estimate Euclidean projection matrices Pi
and coordinates of the 3D points Xj
Interpolated ROI
20 40 60
10
20
30
40
50
60
050
100
0
50
1000
0.02
0.04
Correlation coeffs
020
40
0
20
40100
150
200
250
Active ROI
020
40
0
20
400
0.5
1
1.5
x 10−3PSF approx by 2D Gaussian
/local/MultiCam/Data/20030615H
oengg/arctic1/arctic1.pvi.0001.jpg original
312 314 316 318 320 322 324 326 328
294
296
298
300
302
304
306
308
310
C1 C2
Xu1 u2l1 l2e1 e2
−2
0
2
−6
−4
−2
0
2
−6
−4
−2
0
2
4
10
11
13
1
9
2
8
12
reconstructed points/camera setup only inliers are used
3
16
15
14
7
4
6
5
−3−2
−10
12
3
−2
−1
0
1
2
3
4
10
9
8
7
6
11
16
Graphical Output Validation: View from the top camera
12
13
15
14
1
2
3
4
5
0 100 200 300 400 500 600 700
0
100
200
300
400
500
measured, o, vs reprojected, +, 2D points (camera: 4)
−400 −200 0 200 400 600 800 1000 1200 1400 1600−500
0
500
1000
measured, o, vs reprojected, +, 2D points (camera: 40)
−3 −2 −1 0 1 2 3
−2
−1
0
1
2
3
10
9
8
7
6
11
16
Graphical Output Validation: View from the top camera
12
15
13
14
5
3
4
2
1
−1 0 1
−2
−1
0
1
2
3 71
70
80
Graphical Output Validation: Aligned data
72
81
82
0 100 200 300 400 500 600
0
100
200
300
400
500measured, o, vs reprojected, +, 2D points (camera: 12)
0 2 4 6 8 10 12 14 16 180
1
2
3
4
5
6
7
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
0 100 200 300 400 500 600 700
0
100
200
300
400
500
measured, o, vs reprojected, +, 2D points (camera: 12)
0 2 4 6 8 10 12 14 16 180
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
−3−2−101
−1
0
1
2
3
4
5
6
4
reconstructed points/camera setup only inliers are used 3
2
2 3 40
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Id of the camera
2D error: mean (blue), std (red)
pixe
ls
