Motion compensation for three-dimensional measurementsof macroscopic objects using fringe projection

Andreas Breitbarth*, Peter Kühmstedt*, Gunther Notni*, Joachim Denzler*** Fraunhofer Institute for Applied Optics and Precision Engineering IOF

** Friedrich Schiller University Jena, Chair for Computer Vision

mailto:andreas.breitbarth@iof.fraunhofer.de

For three-dimensional reconstruction using fringe projection and under motionpoint correspondences between different 2D images and global phase shifts areno longer constant during the complete process. One possibility to address thisissue is the use of motion compensation as described in this article. Simulationresults and remaining limitations are indicated.

1 Introduction

Three-dimensional reconstruction with measure-ment systems using fringe projection is widely usedin non-static applications such as measurement ofteeth in-side mouth, measurements using hand-heldsensors [1], and inspections at assembly lines. Inparticular, in the last few years the demand of fastand cost-effective measurement systems becomesevident. Therefore, it is recommended to measuremoving objects continuously instead of the cur-rent start-stop regimes. There are two potential ap-proaches to address this issue: 1. the usage of high-speed components for projection and image acquisi-tion (∧= quasi-static situation for 3D measurements)or 2. algorithms for motion compensation. In the cur-rent study, standard hardware was used and a novelalgorithm for motion compensation was developed.

2 (Quasi-)Static situation

For 3D reconstruction it is required to find correspon-dences between at least two images Icn from differ-ent cameras c, in which one camera also could be aprojector (∧= inverse camera). This is achieved in asuccessive process for each point of the master im-age (arbitrary selected): Determine 2D point in themaster image, search for the corresponding pointin the second image, and finally triangulate with aknown system setup. The 2D search area for cor-respondences will be constraint through the activeprojection of the fringe pattern in two projection di-rections to only one 2D point. To accomplish high ro-bustness and accuracy in 3D reconstruction, a com-plex sequence of pattern is needed which requires along acquisition time using standard hardware.

In more detail, the procedure for 3D reconstructionwith static setup is performed as follows:

1. Acquisition of fringe pattern with phase shifting:Each image can be discribed with

Icn(x, y) = a(x, y)+b(x, y)·cos[φc(x, y)+∆φn] (1)

where ∆φn is the phase shift between two se-quently fringe pattern in degree.

2. Calculate raw fine phases Φc(Ick) ∈ [−π, π] with

φc(x, y) = arctanIc4(x, y)− Ic2(x, y)

Ic1(x, y)− Ic3(x, y)(2)

if you use k = 4 fringe pattern per direction.

3. Unwrap raw fine phases by using e.g. gray codepattern to get global unique fine phases Φc′.

4. Calculate final 3d point cloud by triangulation bet-ween the fine phase maps Φc′ of at least twocameras c (or camera and projector).

3 Measurements under motion

In case of measurements with moving objects and/orsensor systems at least two previous assumptionsare no longer given:

• uniform global phase shift ∆φn

• unique pixel correspondences (x, y) over thewhole measurement sequence, which are re-quired for equation (2)

These given facts can be corrected by refinementof the static situation. Therefore, two important func-tions have to be added: First, a step of motion esti-mation with the result of motion vectors T ∈ R6 andsecond, a step of motion compensation using T .

6D motion estimation is unfeasible using 2D inputdata. Certainly, there are methods existing to esti-mate motion out of 2D images, e.g. autocorrelation.However, several kinds of motion are not convenient.To bypass this restiction, we used 3D data for motionestimation. An established algorithm to minimize er-rors or distances of point clouds is the approach ofiterative closest points (ICP) [2]. It estimates the besttransformation between two sets of given points in-dependent of their dimension. The first step of anal-ysis is the correlation of two 3D sets which requires

DGaO Proceedings 2012 – http://www.dgao-proceedings.de – ISSN: 1614-8436 – urn:nbn:de:0287-2012-XXXX-Y

3D input data and is realised by fourier analysisof each individual 2D fringe image [3]. As result, acoarse 3D point cloud for each 2D fringe image iscreated that is distinct from the final 3D point cloudbut sufficient as input for the ICP algorithms.

The motion compensation as second step com-prises both a rear projection of the coarse 3D pointclouds with known motion in the camera images anda rear projection to the projection plain. The last-mentioned step is required for determination of thelocal phase shifts ∆φn(x, y) in projection, which arenow depending on the local object height and the lo-cal motion. Due to the previous system calibration,the projection matrices are known. Rear projectionof 3D point clouds into the camera images result innew aligned intensity images Icn(x1

′, y1′). These new

intensity images fulfil the equation (2).

4 Results and limitations

Figure 1 illustrates the differences between motioncompensation and non-compensated input imagesand phase shift values ∆φn(x, y) for the final 3D re-sult. Without compensation (Fig. 1, left) various er-rors occur, particularly at object parts with distinctive3D structure. In contrast, using motion compensa-tion (Fig. 1, right) a smooth and for one point of viewcomplete 3D result is generated.

Fig. 1 Final 3d result (color-coded). Left: Disturbed imagewithout compensation. Right: Smooth image using motioncompensation.

The results look promising, however, there are stillremaining limitations, e.g. changes in lighting dur-ing the measurement sequence. This is describedby b(x, y) in equation (1) and is constant for staticscenes, the most translations and rotations aroundan axis which is parallel to the principal axis of theprojector. For arbitrary relative motion between themeasurement object and the projection system light-ing alterations have to be integrated in the compen-sation model which is part of further research.

In addition, the accuracy of the current framework islimited by two error sources: Error in motion estima-tion and motion blurred input images. The first men-tioned error source is mainly affected by poorly 3Dpoint clouds as result of fourier analysis of individualfringe images. Figure 2 demonstrates the correlation

between the error of motion vector estimation andthe relative 3D error in the simulation setup.

Fig. 2 Influence of motion vector estimation errors ε(T ) onthe relative 3D error.

The influence of blurred input images on the final 3Derror is indicated in figure 3. Exemplary the worstcase (fringe shifting and motion direction are parallelto each other) is presented. In case of fringe shiftingis orthogonally to the motion direction, the influenceof blurred images is quite low.

Fig. 3 Influence of motion blurred input images on the rel-ative 3D error.

5 Conclusion

The general approach for 3D reconstruction ofmacroscopic objects in motion is reliable in the ma-jority of cases, particularly in translations, agitationsand minor motion. However, there are still remainingchallenges, e.g. changes in lighting and major mo-tion (blurred input images). Thus, further studies willbe needed to overcome these limitations.

[1] C. Munkelt, I. Schmidt, C. Bräuer-Burchardt, P. Kühm-stedt, and G. Notni, “Cordless portable multi-viewfringe projection system for 3D reconstruction,” inCVPR (2007).

[2] S. Rusinkiewicz and M. Levoy, “Efficient Variants of theICP Algorithm,” in Third International Conference on3D Digital Imaging and Modeling (3DIM) (2001).

[3] M. Takeda and K. Mutoh, “Fourier transform profilom-etry for the automatic measurement of 3-D objectshapes,” Applied Optics 22(24), 3977–3982 (1983).

DGaO Proceedings 2012 – http://www.dgao-proceedings.de – ISSN: 1614-8436 – urn:nbn:de:0287-2012-XXXX-Y