Embed Size (px)
Citation preview
Summary Page
Robust 6DOF Motion Estimation for Non-Overlapping, Multi-Camera Systems• Is this a system paper or a regular paper? This is a regular paper.
• What is the main contribution in terms of theory, algorithms and approach?
The paper introduces a novel, robust approach for 6DOF motion estimation of a multi-camera system with non-overlapping views. The proposed approach is the first that can determine the rotation, translation direction and scale oftranslation for a two camera system with non-overlapping views.
Our technique is based on the observation that, given the epipolar geometry of one of the cameras which gives 5DOF ofthe multi-camera system, the position of the epipole in all other cameras is restricted to a line in the images. Hence thescale as the sixth degree of freedom of the camera system motion describes a linear subspace. The algorithm works intwo phases, first estimating the essential matrix of one camera using the five point algorithm of Nister [11] for calibratedcameras and second calculating the scale of translation using one or more correspondences from any other camera usingthe multi-camera scale constraint we have developed. Additionally, we give a geometric interpretation of the scaleconstraint and analyze critical camera system motions for which the scale cannot be observed.
(a) (b)Multi-camera system (a) Overlapping vs. (b) non-overlapping Images from one camera cluster
Multi-camera systems
• Describe the types of experiments and the novelty of the results. If applicable, provide comparison to the state of the artin this area.
To demonstrate the performance of our algorithm we embedded it into a full structure from motion system. We recon-struct a camera path using single camera calibrated structure from motion. The absolute scale of this reconstructionaccumulates error with time. Starting with the reconstruction correctly scaled, we measure the accumulated scale errorof the camera motion using highly accurate GPS/INS measurements. We compare this scale drift to the that foundusing only measurements from our 6DOF multi-camera motion estimation algorithm and demonstrate that our systemcan determine the reconstructions’s absolute scale. This clearly demonstrates that the additional constraint advancesmulti-camera structure from motion to achieve robust absolute scale reconstructions without overlapping fields of view.
Additionally, we evaluate our technique with experiments on real data to demonstrate the algorithm’s rotation and scaledtranslation measurement accuracy and we perform a similar analysis using synthetic data over varying levels of noiseand different camera system motions.
0 500 1000 15000
200
400
600
800
1000
1200
Meters North
Met
ers
Eas
t
Locations in Vehicle Path Where Scale Can Be Estimated
Motion Used for Results
0 20 40 60 80 100 120 140 160 180 2000
0.2
0.4
0.6
0.8
1
Sequence Frame Number
|Rec
onst
ruct
ion
Tra
nsla
tion|
/|T
rue
Tra
nsla
tion|
(1.
0 =
No
Err
or) Measurement of Absolute Scale Error in a Reconstruction
True Scale Error of ReconstructionEstimated Scale Error of Reconstruction
Map of camera path showing where scale can be estimated Single camera reconstruction scale drift
Robust 6DOF Motion Estimation for Non-Overlapping, Multi-Camera Systems
Brian Clipp+, Jae-Hak Kim∗, Jan-Michael Frahm+, Marc Pollefeys+ and Richard Hartley∗+Department of Computer Science
The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA∗Research School of Information Sciences and Engineering
The Australian National University, Canberra, ACT, Australia{bclipp,jmf,marc}@cs.unc.edu
{Jae-Hak.Kim, Richard.Hartley}@anu.edu.au
Abstract
This paper introduces a novel, robust approach for6DOF motion estimation of a multi-camera system withnon-overlapping views. The proposed approach is the firstthat is able to solve the pose estimation, including scale,for a two camera system with non-overlapping views. Incontrast to previous approaches, it degrades gracefully ifthe motion is close to degenerate. For degenerate motionsthe technique estimates the remaining 5DOF. The proposedtechnique is evaluated on real and synthetic sequences.
1. Introduction
Recently, interest has grown in motion estimation formulti-camera systems as these systems have been used tocapture ground based and indoor data sets for reconstruc-tion [16, 17]. To combine high resolution and a fast frame-rate with a wide field-of-view, the most effective approachoften consists of combining multiple video cameras into acamera cluster. Some systems have all cameras mounted to-gether in a single location, eg. [1, 2, 3], but it can be difficultto avoid losing part of the field of view due to occlusion (i.e.typically requiring camera cluster placement high up on aboom). Alternatively, for mounting on a vehicle the systemcan be split into two clusters so that one can be placed oneach side of the vehicle and occlusion problems are mini-mized. We will show that by using a system of two cam-era clusters, consisting of one or more cameras each, sepa-rated by a known transformation, the six degrees of freedom(DOF) of camera system motion, including scale, can be re-covered.
An example of a multi-camera system for the capture ofground based video is shown in Figure 1. It consists of twocamera clusters, one on each side of a vehicle. The cameras
Figure 1. Example of a multi-camera system on a vehicle
are attached tightly to the vehicle and can be considered arigid object. This system is used for experimental evalua-tion of our approach.
Computing the scale, structure and camera motion of ageneral scene is an important application of our scale es-timation approach. In [12] Nister et al. investigated theproperties of visual odometry for single-camera and stereo-camera systems. Their analysis showed that a single camerasystem is not capable of maintaining a consistent scale overtime. Their stereo system is able to maintain absolute scaleover extended periods of time by using a known baselineand cameras with overlapping fields of view. Our approacheliminates the requirement for overlapping fields of viewand is able to maintain the absolute scale over time.
The remainder of the paper is organized as follows. Thenext section discusses the related work. In section 3 we in-troduce our novel solution to finding the 6DOF motion ofa two camera system with non-overlapping views. We de-rive the mathematical basis for our technique in section 4as well as give a geometrical interpretation of the scale con-straint. The algorithm used to solve for the scaled motion isdescribed in section 5. Section 6 discusses the evaluation ofthe technique on synthetic data and on real imagery.
2
2. Related WorkIn recent years much work has been done on egomotion
estimation of multi-camera systems. Nister et al. [12] pro-posed a technique that used a calibrated stereo camera sys-tem with overlapping fields of view for visual navigation.The proposed algorithm employed a stereo camera systemto recover 3D world points up to an unknown Euclideantransformation. In [7] Frahm et al. introduced a 6DOFestimation technique using a multi-camera system. Theirapproach assumed overlapping camera views to obtain thescale of the camera motion. In contrast, our technique doesnot require any overlapping views. In [15] Tariq and Del-laert proposed a 6DOF tracker for a multi-camera systemfor head tracking. Their multi-camera motion estimation isbased on known 3D fiducials. The position of the multi-camera system is computed from the fiducial positions witha MAP based optimization. Our algorithm does not requireany information about the observed scene. Therefore, it hasa much broader field of application.
Another class of approaches is based on the generalizedcamera model [8, 13] of which a stereo/multi-camera sys-tem is a special case. A generalized camera is a camerawhich can have different centers of projection for each pointin the world space. An approach to the motion estimation ofa generalized camera was proposed by Stewenius et al. [14].They showed that there are up to 64 solutions for the rela-tive position of two generalized cameras given 6 point cor-respondences. Their method delivers a rotation, translationand scale of a freely moving generalized camera. One ofthe limitations of their approach is that centers of projectioncannot be collinear. This limitation naturally excludes alltwo camera systems as well as a system of two camera clus-ters where the cameras of the cluster have approximatelythe same center of projection. Our technique addresses thisclass of commonly used two/multi-camera systems and de-livers only up to ten solutions. The next section will in-troduce our novel approach for estimating a multi-camerasystem’s 6DOF motion.
3. 6DOF Multi-camera MotionThe proposed approach addresses the 6DOF motion es-
timation of multi-camera systems with non-overlappingfields of view. Previous approaches to 6DOF motion es-timation have used camera configurations with overlappingfields of view, which allow correspondences to be triangu-lated simultaneously across multiple views with a known,rigid baseline. Our approach uses a temporal baseline wherepoints are only visible in one camera at a given time. Thedifference in the two approaches is illustrated in figure 2.
Our technique assumes that we can establish at least fivetemporal correspondences in one of the cameras and onetemporal correspondence in any additional camera. In prac-
(a) (b)Figure 2. (a) Overlapping stereo camera pair, (b) Non-overlappingmulti-camera system
tice this assumption is not a limitation, as a reliable estima-tion of camera motion requires multiple correspondencesfrom each camera due to noise.
The essential matrix which defines the epipolar geome-try of a single freely moving calibrated camera can be esti-mated from five points. Nister proposed an efficient algo-rithm for this estimation in [11]. It delivers up to ten validsolutions for the epipolar geometry. The ambiguity can beeliminated with additional points. With oriented geometrythe rotation and the translation up to scale of the cameracan be extracted from the essential matrix. Consequently asingle camera provides 5DOF of the camera motion. Theremaining degree is the scale of the translation. Given these5DOF of multi-camera system motion (rotation and trans-lation direction) we can compensate for the rotation of thesystem. Our approach is based on the observation that giventhe temporal epipolar geometry of one of the cameras, theposition of the epipole in each of the other cameras of themulti-camera system is restricted to a line in the image.Hence the scale as the remaining degree of freedom of thecamera motion describes a linear subspace.
In the next section, we derive the mathematical basis ofour approach to motion recovery.
4. Two Camera System – Theory
We consider a system involving two cameras, rigidlycoupled with respect to each other. The cameras are as-sumed to be calibrated. Figure 3 shows the configurationof the two-camera system. The cameras are denoted by C1
and C2, at the starting position and C′1 and C′
2 after a rigidmotion.
We will consider the motion of the camera-pair to a newposition. Our purpose is to determine the motion using im-age measurements. It is possible through standard tech-niques to compute the motion of the cameras up to scale,by determining the motion of just one of the cameras usingpoint correspondences from that camera. However, fromone camera, motion can be determined only up to scale. Thedirection of the camera translation may be determined, but
not the magnitude of the translation. It will be demonstratedin this paper that a single correspondence from the secondcamera is sufficient to determine the scale of the motion,that is, the magnitude of the translation. This result is sum-marized in the following theorem.
Theorem 1. Let a two camera system have initial con-figuration determined by camera matrices P1 = [I | 0]and P2 = [R2 | − R2C2]. Suppose it moves rigidly toa new position for which the first camera is specified byP′1 = [R′1 | − λR′1C
′1]. Then the scale of the translation,
λ, is determined by a single point correspondence x′ ↔ xseen in the second camera according to the formula
x′>Ax + λx′>Bx = 0 (1)
where A = R2R′1 [(R′1> − I)C2]×R2
> and B =R2R′1 [C′
1]×R2>. In this paper [a]×b denotes the skew-
symmetric matrix inducing the cross product a× b.E2λC1' C1 R1'TC2C2λC1'+R1'TC2Figure 3. Motion of a multi-camera system consisting of tworigidly coupled conventional cameras.
In order to simplify the derivation we assume that the co-ordinate system is centered on the initial position of the firstcamera, so that P1 = [I | 0]. Any other coordinate systemis easily transformed to this one by a Euclidean change ofcoordinates.
Observe also that after the motion, the first camera hasmoved to a new position with camera center at λC′
1. Thescale is unknown at this point because in our method wepropose as a first step determining the motion of the cam-eras by computing the essential matrix of the first cameraover time. This allows us to compute the motion up to scaleonly. Thus the scale λ remains unknown. We now proceedto derive Theorem 1. Our immediate goal is to determinethe camera matrix for the second camera after the motion.First note that the camera P′1 may be written as
P′1 = [I | 0]
[R′1 −λR′1C
′1
0> 1
]= P1T .
where the matrix T, is the Euclidean transformation inducedby the motion of the camera pair. Since the second camera
undergoes the same Euclidean motion, we can compute thecamera P′2 to be
P′2 = P2T
= [R2 | − R2C2]
[R′1 −λR′1C
′1
0> 1
]= [R2R
′1 | − λR2R
′1C
′1 − R2C2]
= R2R′1[I | − (λC′
1 + R′1>C2)] . (2)
From the form of the two camera matrices P2 and P′2, wemay compute the essential matrix E2 for the second camera.
E2 = R2R′1[λC′
1 + R′1>C2 −C2]×R2
>
= R2R′1[R
′1>C2 −C2]×R2
> + λR2R′1[C
′1]×R2
>(3)
= A + λB .
Now, given a single point correspondence x′ ↔ x asseen in the second camera, we may determine the value ofλ, the scale of the camera translation. The essential matrixequation x′>E2x = 0 yields x′>Ax + λx′>Bx = 0, andhence:
λ = −x′>Axx′>Bx
= −x′>
(R2R′1[R
′1>C2 −C2]×R2
>)x
x′>(R2R′1[C
′1]×R2
>)x
(4)
.So each correspondence in the second camera provides a
measure for the scale. In the next section we give a geomet-ric interpretation for this constraint.
4.1. Geometric Interpretation
The situation may be understood via a different geomet-ric interpretation, shown in Figure 4. We note from (2)that the second camera moves to a new position C′
2(λ) =R′1>C2 + λC′
1. The locus of this point for varying valuesof λ is a straight line with its direction vector C′
1, passingthrough the point R′1
>C2. From its new position, the cam-era observes a point at position x′ in its image plane. Thisimage point corresponds to a ray v′ along which the 3Dpoint X must lie. If we think of the camera as moving alongthe line C′
2(λ) (the locus of possible final positions of thesecond camera center), then this ray traces out a plane Π;the 3D point X must lie on this plane.
On the other hand, the point X is also seen (as imagepoint x) from the initial position of the second camera, andhence lies along a ray v through C2. The point where thisray meets the plane Π must be the position of the point X.In turn this determines the scale factor λ.
4.2. Critical configurations
This geometric interpretation allows us to identify crit-ical configurations in which the scale factor λ cannot be
Translation
Rotation
R′>
1C2
λc11′
ΠC2′′
C1′′
C2
C1
C1′
nΠ
C2′
X
v′
v
Figure 4. The 3D point X must lie on the plane traced out by theray corresponding to x′ for different values of the scale λ. It alsolies on the ray corresponding to x through the initial camera centerC2.
determined. As shown in Figure 4, the 3D point X is theintersection of the plane Π with a ray v through the cameracenter C2. If the plane does not pass through C2, then thepoint X can be located as the intersection of plane and ray.Thus, a critical configuration can only occur when the planeΠ passes through the second camera center, C2.
According to the construction, the line C′2(λ) lies on the
plane Π. For different 3D points X, and corresponding im-age measurement x′, the plane will vary, but always containthe line C′
2(λ). Thus, the planes Π corresponding to dif-ferent points X form a pencil of planes hinged around theaxis line C′
2(λ). Unless this line actually passes throughC2, there will be at least one point X for which C2 does notlie on the plane Π, and this point can be used to determinethe point X, and hence the scale.
Finally, if the line C′2(λ) passes through the point C2,
then the method will fail. In this case, the ray correspondingto any point X will lie within the plane Π, and a uniquepoint of intersection cannot be found.
In summary, if the line C′2(λ) does not pass through the
initial camera center C2, almost any point correspondencex′ ↔ x may be used to determine the point X and the trans-lation scale λ. The exceptions are point correspondencesgiven by points X that lie in the plane defined by the cam-era center C2 and the line C′
2(λ) as well as far away pointsfor which Π and v are almost parallel.
If on the other hand, the line C′2(λ) passes through the
center C2, then the method will always fail. It may beseen that this occurs most importantly if there is no cam-era rotation, namely R′1 = I. In this case, we see thatC′
2(λ) = C2 + λC′1, which passes through C2. It is easy to
give an algebraic condition for this critical condition. SinceC′
1 is the direction vector of the line, the point C2 will
ΘλC1'+R1'TC2 R1'TC2λC1' C1 C2Θ = Angle Between Rotation Induced Translation for C2 and Camera System TranslationFigure 5. Rotation Induced Translation to Translation Angle
Center of Rotation λC1'+R1'TC2R1'TC2λC1' C1 C2Figure 6. Critical motion due to constant rotation rate
lie on the line precisely when the vector R′1>C2 − C2 is
in the direction C′1. This gives a condition for singularity
(R′1>C2 − C2) × C′
1 = 0, or rearranging this expression,and observing that the vector C2 × C′
1 is perpendicular tothe plane of the three camera centers C2, C′
1 and C1 (thelast of these being the coordinate origin), we may state:
Theorem 2. The critical condition for singularity for scaledetermination is
(R′1>C2)×C′
1 = C2 ×C′1 .
In particular, the motion is not critical unless the axis of ro-tation is perpendicular to the plane determined by the threecamera centers C2, C′
1 and C1.
Intuitively, critical motions occur when the rotation in-duced translation R′1
>C2 − C2 is aligned with the transla-tion C′
1. In this case the angle Θ in Fig. 5 is zero. Themost common motion which causes a critical condition iswhen the camera system translates but has no rotation. An-other common but less obvious critical motion occurs whenboth camera paths move along concentric circles. This con-figuration is illustrated in figure 6. A vehicle borne multi-camera system turning at a constant rate undergoes criticalmotion, but not when it enters and exits a turn.
Figure 7. Algorithm for estimating 6DOF motion of a multi-camera system with non-overlapping fields of view.
Detecting critical motions is important to determiningwhen the scale estimates are reliable. One method to deter-mine the criticality of a given motion is to use the approachof [6]. We need to determine the dimension of the spacewhich includes our estimate of the scale. To do this wedouble the scale λ and measure the difference in the frac-tion of inliers to the essential matrix of our initial estimateand the doubled scale essential matrix. If a large propor-tion of inliers are not lost when the scale is doubled thenthe scale is not observable from the data. If the scale is ob-servable the deviation from the estimated scale value wouldcause the correspondences to violate the epipolar constraint,which means they are outliers to the constraint for the dou-bled scale. When the scale is ambiguous doubling the scaledoes not cause correspondences to be classified as outliers.This method proved to work practically on real data sets.
5. AlgorithmFigure 7 shows an algorithm to solve relative motion
of two generalized cameras from 6 rays with two centerswhere 5 rays meet one center and sixth ray meets the othercenter. First, we use 5 correspondences in one ordinarycamera to estimate an essential matrix between two framesin time. The algorithm used to estimate the essential ma-trix from 5 correspondences is the method by Nister [11]. Itis also possible to use a simpler algorithm which gives thesame result developed by Li and Hartley [10]. The 5 corre-spondences are selected by the RANSAC (Random SampleConsensus) algorithm [5]. The distance between a selectedfeature and its corresponding epipolar line is used as an in-lier criterion in the RANSAC algorithm. The essential ma-trix is decomposed into a skew-symmetric matrix of transla-tion and a rotation matrix. When decomposing the essentialmatrix into rotation and translation the chirality constraint
is used to determine the correct configuration [9]. At thispoint the translation is recovered up to scale.
To find the scale of translation, we use Eq. 4 withRANSAC. One correspondence is randomly selected fromthe second camera and is used to calculate a scale valuebased on the constraint given in Eq. 4.
Based on this scale factor, the translation direction androtation of the first camera, and the known extrinsics be-tween the cameras, an essential matrix is generated for thesecond camera. Inlier correspondences in the second cam-era are then determined based on their distance to the epipo-lar lines. A linear least squares calculation of the scalefactor is then made with all of the inlier correspondencesfrom the second camera. This linear solution is refinedwith a non-linear minimization technique using the GNCfunction [4] which takes into account the influence of allcorrespondences, not just the inliers of the RANSAC sam-ple, in calculating the error. This error function measuresthe distance of all correspondences to their epipolar linesand smoothly varies between zero for perfect correspon-dence and one for an outlier with distance to the epipolarline greater than some threshold. One could just as easilytake single pixel steps from the initial linear solution in thedirection which maximizes inliers, or equivalently minimiz-ing the robust error function. The non-linear minimizationsimply allows us to select step sizes depending on the sam-pled Jacobian of the error function, which should convergefaster than single pixel steps and allows for sub-pixel preci-sion.
Following refinement of the scale estimate, the inlier cor-respondences of the second camera are calculated and theirnumber is used to score the current RANSAC solution. Thefinal stage in the scale estimation algorithm is a bundle ad-justment of the multi-camera system’s motion. Inliers arecalculated for both cameras and they are used in a bundleadjustment refining the rotation and scaled translation of thetotal, multi-camera system.
While this algorithm is described for a system consistingof two cameras it is relatively simple to extend the algo-rithm to use any number of rigidly mounted cameras. TheRANSAC for the initial scale estimate, initial linear solu-tion and non-linear refinement are performed over corre-spondences from all cameras other than the camera used inthe five point pose estimate. The final bundle adjustment isthen performed over all of the system’s cameras.
6. ExperimentsWe begin with results using synthetic data to show the al-
gorithm’s performance over varying levels of noise and dif-ferent camera system motions. Following these results weshow the system operating on real data and measure its per-formance using data from a GPS/INS (inertial navigationsystem). The GPS/INS measurements are post processed
−0.2 0 0.2 0.4 0.6 0.8 1 1.2−0.05
0
0.05
0.1
0.15
0.2Rotation Angle Error
Gaussian Noise Standard Deviation (pixels)
Rot
atio
n E
rror
(de
gree
s)
Figure 8. Angle Between True and Estimated Rotations, SyntheticResults of 100 Samples Using Two Cameras
and are accurate to 4cm in position and 0.03 degrees in ro-tation, providing a good basis for error analysis.
6.1. Synthetic Data
We use results on synthetic data to demonstrate the per-formance of the 6DOF motion estimate in the presence ofvarying levels of gaussian noise on the correspondencesover a variety of motions. A set of 3D points was generatedwithin the walls of an axis-aligned cube. Each cube wallconsisted of 5000 3D points randomly distributed within a20m x 20m x 0.5m volume. The two-camera system, whichhas an inter-camera distance of 1.9m, a 100o angle betweenoptical axes and non-overlapping fields of view, is initiallypositioned at the center of the cube, with identity rotation.A random motion for the camera system was then gener-ated. The camera system’s rotation was generated from auniform ±6o distribution sampled independently in eachEuler angle. Additionally, the system was translated by auniformly distributed distance of 0.4m to 0.6m in a randomdirection. A check for degenerate motion is performed bymeasuring the distance between the epipole of the secondcamera (see Fig. 3) due to rotation of the camera system andthe epipole due to the combination of rotation and transla-tion. Only results of non-degenerate motions with epipoleseparations equivalent to a 5o angle between the translationvector and the rotation induced translation vector(see Fig. 5)are shown. Results are given for 100 sample motions foreach of the different values of normally distributed, zeromean Gaussian white noise added to the projections of the3D points into the system’s cameras. The synthetic camerashave calibration matrices and fields of view which matchthe cameras used in our real multi-camera system. Eachreal camera has an approximately 40o x 30o field of viewand a resolution of 1024 x 768 pixels.
Results on synthetic data are shown in figures 8 to11. One can see that the system is able to estimate therotation (Fig. 8) and translation direction (Fig. 9) wellgiven noise levels that could be expected using a 2D fea-ture tracker on real data. Figure 10 shows a plot of‖Test − Ttrue‖ / ‖Ttrue‖. This ratio measures both the ac-curacy of the estimated translation direction, as well as
−0.2 0 0.2 0.4 0.6 0.8 1 1.2−1
0
1
2
3
4Translation Direction Error
Gaussian Noise Standard Deviation (pixels)
Tra
nsla
tion
Dire
ctio
n E
rror
(de
gree
s)
Figure 9. Angle Between True and Estimated Translation Vectors,Synthetic Results of 100 Samples Using Two Cameras
−0.2 0 0.2 0.4 0.6 0.8 1 1.2−0.5
0
0.5
1Translation Error Ratio
Gaussian Noise Standard Deviation (pixels)
|Tes
t−T
true
| / |T
true
|
Figure 10. Scaled Translation Vector Error, Synthetic Results of100 Samples Using Two Cameras
−0.2 0 0.2 0.4 0.6 0.8 1 1.20
0.5
1
1.5
2Scale Error Ratio
Gaussian Noise Standard Deviation (pixels)
|Tes
t| / |T
true
|
Figure 11. Scale Ratio, Synthetic Results of 100 Samples UsingTwo Cameras
the scale of the translation and would ideally have a valueof zero because the true and estimated translation vectorswould be the same. Given the challenges of translation esti-mation and the precise rotation estimation we use this ratioas the primary performance metric for the 6DOF motion es-timation algorithm. The translation vector ratio along withthe rotation error plot demonstrate that the novel system per-forms well given a level of noise that could be expected inreal tracking results.
6.2. Real data
For a performance analysis on real data we collectedvideo using an eight camera system mounted on a vehicle.The system included a highly accurate GPS/INS unit whichallows comparisons of the scaled camera system motion cal-culated with our method to ground truth measurements. Theeight cameras have almost no overlap to maximize the totalfield of view and are arranged in two clusters facing toward
0 50 100 150 2000
2
4
6
8
10Translation Direction Error
Sequence Frame Number
Tra
nsla
tion
Err
or (
deg)
Points Where Scale Can Be Estimated
Figure 12. Angle Between True and Estimated Translation Vec-tors, Real Data with Six Cameras
0 50 100 150 2000
0.2
0.4
0.6
0.8
1Rotation Error
Sequence Frame Number
Rot
atio
n E
rror
(de
g)
Points Where Scale Can Be Estimated
Figure 13. Angle Between True and Estimated Rotations, RealData with Six Cameras
the opposite sides of the vehicle. In each cluster the cameracenters are within 25cm of each other. A camera cluster isshown in Fig. 1. The camera clusters are separated by ap-proximately 1.9m and the line between the camera clustersis approximately parallel with the rear axle of the vehicle.Three of the four cameras in each cluster cover a horizontalfield of view on each side of the vehicle of approximately120o x 30o. A fourth camera points to the side of the vehi-cle and upward. Its principle axis has an angle of 30o withthe horizontal plane of the vehicle which is colinear withthe optical axes of the other three cameras.
In these results on real data we take advantage of thefact that we have six horizontal cameras and use all of thecameras to calculate the 6DOF system motion. The upwardfacing cameras were not used because they only recordedsky in the sequence. For each pair of frames recorded atdifferent times, each camera in turn is selected and the fivepoint pose estimate is performed for that camera. The othercameras are then used to calculate the scaled motion of thecamera system using the five point estimate from the se-lected camera as an initial estimate of the camera system ro-tation and translation direction. The 6DOF motion solutionfor each camera selected for the five point estimate is scoredaccording to the fraction of inliers of all other cameras. Themotion with the largest fraction of inliers is selected as the6DOF motion for the camera system.
In table 1 we show the effect of critical motions de-scribed in section 4.2 over a sequence of 200 frames. Crit-ical motions were detected using the QDEGSAC [6] ap-proach described in that section. Even with critical motion
0 50 100 150 2000
1
2
3
4
5Ratio of Estimated Translation Length to True Translation Length
Sequence Frame Number
|Tes
t| / |T
true
|
Points Where Scale Can Be Estimated
Figure 14. Scale Ratio, Real Data with Six Cameras
0 50 100 150 2000
1
2
3
4
5Translation Error Ratio
Sequence Frame Number
|Tes
t−T
true
| / |T
true
|
Points Where Scale Can Be Estimated
Figure 15. Scaled Translation Vector Difference from GroundTruth, Real Data with Six Cameras
‖Test−Ttrue‖‖Ttrue‖ 0.23± 0.19‖Test‖‖Ttrue‖ 0.90± 0.28
Table 1. Relative translation vector error including angle and errorof relative translation vector length mean± std.dev.
the system degrades to the standard 5DOF motion estima-tion from a single camera and only the scale remains am-biguous as shown by the translation direction and rotationangle error in Figures 12 and 13. This graceful degradationto the one camera motion estimation solution means that thealgorithm solves for all of the possible degrees of freedomof motion given the data provided to it.
In this particular experiment the system appears to con-sistently underestimate the scale with our multi-camera sys-tem when the motion is non-critical. This is likely due to acombination of error in the camera system extrinsics anderror in the GPS/INS ground truth measurements.
Figure 16 shows the path of the vehicle mounted multi-camera system and locations where the scale can be esti-mated. From the map it is clear that the scale cannot beestimated in straight segments as well as in smooth turns.This is due to the constant rotation rate critical motion con-dition described in section 4.2. We selected a small sectionof the camera path circled in figure 16 and used a calibratedstructure from motion (SfM) system similar to the systemused in [11] to reconstruct the motion of one of the sys-tem’s cameras. For a ground truth measure of scale erroraccumulation we scaled the distance traveled by a camerabetween two frames at the beginning of this reconstructionto match the true scale of the camera motion according to
0 500 10000
500
1000
1500
Meters East
Met
ers
Nor
th
Locations in Vehicle Path Where Scale Can Be Estimated
Motion Used In Results
Figure 16. Map of vehicle motion showing points where scale canbe estimated
0 20 40 60 80 100 120 140 160 180 2000
0.2
0.4
0.6
0.8
1
Sequence Frame Number
|Rec
onst
ruct
ion
Tra
nsla
tion|
/|T
rue
Tra
nsla
tion|
(1.
0 =
No
Err
or) Measurement of Absolute Scale Error in a Reconstruction
True Scale Error of ReconstructionEstimated Scale Error of Reconstruction
Figure 17. Scaled structure from motion reconstruction
the GPS/INS measurements. Figure 17 shows how errorin the scale accumulates over the 200 frames (recorded at30 frames per second) of the reconstruction. We then pro-cessed the scale estimates from the 6DOF motion estima-tion system with a Kalman filter to determine the scale ofthe camera’s motion over many frames and measured theerror in the SfM reconstruction scale using only our algo-rithm’s scale measurements. The scale drift estimates fromthe 6DOF motion estimation algorithm clearly measure thescale drift and provide a measure of absolute scale.
7. ConclusionIn this paper we have introduced a novel algorithm that
solves for the 6DOF motion of a multi-camera system withnon-overlapping fields of view. The proposed techniqueis the first that can estimate the absolute scale for a twocamera system with non-overlapping views. We have pro-vided a complete analysis of the critical motions of themulti-camera system that make the absolute scale unob-servable. Our algorithm can detect these critical motionsand gracefully degrades to the estimation of the epipolar
geometry. We have demonstrated the performance of oursolution through both synthetic and real motion sequences.Additionally, we embedded our novel algorithm in a struc-ture from motion system to demonstrate that our techniqueadvances multi-camera structure from motion to allow thedetermination absolute scale without requiring overlappingfields of view.
References[1] Immersive Media Camera Systems Dodeca 2360,http://
www.immersivemedia.com/. 2[2] Imove GeoView 3000, http://www.imoveinc.com/
geoview.php. 2[3] Point Grey Research Ladybug2, http://www.ptgrey.
com/products/ladybug2/index.asp. 2[4] A. Blake and A. Zisserman. Visual reconstruction. MIT
Press, Cambridge, MA, USA, 1987. 6[5] M. A. Fischler and R. C. Bolles. Random sample consensus:
a paradigm for model fitting with applications to image anal-ysis and automated cartography. Commun. ACM, 24(6):381–395, 1981. 6
[6] J. Frahm and M. Pollefeys. Ransac for (quasi-)degeneratedata (qdegsac). pages I: 453–460, 2006. 6, 8
[7] J.-M. Frahm, K. Koser, and R. Koch. Pose estimation forMulti-Camera Systems. In DAGM, 2004. 3
[8] M. D. Grossberg and S. K. Nayar. A general imaging modeland a method for finding its parameters. In ICCV, pages108–115, 2001. 3
[9] R. I. Hartley and A. Zisserman. Multiple View Geometryin Computer Vision. Cambridge University Press, ISBN:0521540518, second edition, 2004. 6
[10] H. Li and R. Hartley. Five-point motion estimation madeeasy. In ICPR (1), pages 630–633. IEEE Computer Society,2006. 6
[11] D. Nister. An efficient solution to the five-point relativepose problem. In Int. Conf. on Computer Vision and PatternRecognition, pages II: 195–202, 2003. 1, 3, 6, 8
[12] D. Nister, O. Naroditsky, and J. Bergen. Visual odometry. InIEEE Computer Society Conference on Computer Vision andPattern Recognition, volume 1, pages 652–659, 2004. 2, 3
[13] R. Pless. Using many cameras as one. In CVPR03, pages II:587–593, 2003. 3
[14] H. Stewenius, D. Nister, M. Oskarsson, and K. Astrom. So-lutions to minimal generalized relative pose problems. InWorkshop on Omnidirectional Vision, Beijing China, Oct.2005. 3
[15] S. Tariq and F. Dellaert. A Multi-Camera 6-DOF PoseTracker. In IEEE and ACM International Symposium onMixed and Augmented Reality (ISMAR), 2004. 3
[16] M. Uyttendaele, A. Criminisi, S. Kang, S. Winder,R. Szeliski, and R. Hartley. Image-based interactive explo-ration of real-world environments. IEEE Computer Graphicsand Applications, 24(3):52–63, 2004. 2
[17] xxx. xxx. In xxx, 2006. 2
