1

Joint depth and color camera calibration withdistortion correction

Daniel Herrera C., Juho Kannala, and Janne Heikkila

Abstract—We present an algorithm that simultaneously calibrates two color cameras, a depth camera, and the relative posebetween them. The method is designed to have three key features: accurate, practical, and applicable to a wide range of sensors.The method requires only a planar surface to be imaged from various poses. The calibration does not use depth discontinuitiesin the depth image which makes it flexible and robust to noise. We apply this calibration to a Kinect device and present anew depth distortion model for the depth sensor. We perform experiments that show an improved accuracy with respect to themanufacturer’s calibration.

Index Terms—camera calibration, depth camera, camera pair, distortion, Kinect

F

1 INTRODUCTION

COLOR and depth information provide comple-mentary cues about a scene. Many applications

need to capture both simultaneously, like scene recon-struction and image based rendering. This requires atleast two sensors as no single sensor is able to captureboth. A basic device for scene reconstruction is adepth and color camera pair, which consists of a colorcamera rigidly attached to a depth sensor (e.g. time-of-flight (ToF) camera, laser range scanner, structuredlight scanner). The increasingly popular Kinect deviceis an example of such a camera pair.

In order to reconstruct a scene from the camera pairmeasurements the system must be calibrated. Thisincludes internal calibration of each camera as well asrelative pose calibration between the cameras. Colorcamera calibration has been studied extensively [2],[3]. For depth sensors, different calibration methodshave been developed depending on the technologyused. ToF cameras simultaneously produce an inten-sity and a depth image from the same viewpoint,which simplifies calibration because color discontinu-ities can be accurately localized [4]. Most structuredlight systems can calibrate the projector and cameraseparately. However, if the internals of the device arenot open, we might not have access to the originalintensity images. The Kinect device, for example, usesan infrared camera to detect a projected dot pattern.However, it returns a processed image that is notaligned with the original infrared image.

There is a particular need to calibrate the Kinectdevice because it delivers depth information in Kinectdisparity units (kdu) whose conversion to metric unitschanges for each device. Furthermore, independentcalibration of the cameras may not yield the optimalsystem parameters, and a comprehensive calibrationof the system as a whole could improve individual

Authors are with the Center for Machine Vision Research, University ofOulu, Finland.E-mail: {dherrera, jkannala, jth}@ee.oulu.fi

calibration as it uses all the available information.Depth cameras have been observed to suffer from

complicated geometric distortions due to the pro-cessing performed and the inevitable tolerances intheir manufacturing. Whereas a radial and tangentialdistortion model is usually sufficient to correct the2D pixel positions in color cameras, depth camerasrequire a more complicated model to correct the 3Dmeasurement volume.

1.1 Previous workA standard approach is to calibrate the cameras in-dependently and then calibrate only the relative posebetween them [5], [6], [7]. This may not be the optimalsolution as measurements from one camera can im-prove the calibration of the other camera. Moreover,the independent calibration of a depth camera mayrequire a high precision 3D calibration object that canbe avoided using joint calibration.

Fuchs and Hirzinger [4] propose a multi-splinemodel for time-of-flight (ToF) cameras. Their modelhas a very high number of parameters and it requiresa robotic arm to know the exact pose of the camera.Lindner and Kolb [8] use a high resolution colorcamera to determine the pose of the camera, remov-ing the need for a robotic arm. Lichti [9] proposesa calibration method for an individual laser rangescanner using only a planar calibration object. It per-forms a comprehensive calibration of all parameters.However, it relies on the depth camera deliveringradiometric intensity and range for each pixel. Thisis not directly applicable to a camera pair becausethe color and depth information are not taken fromthe same reference frame. Zhu et al. [10] describe amethod for fusing depth from stereo and ToF cameras.Their calibration uses the triangulation from the stereocameras as ground truth. This may not be optimal asit ignores the possible errors in stereo triangulationand measurement uncertainties.

2

ToF cameras are known to present distortions bothon the optical ray direction and on the measureddepth. Kim et al. [11] show a principled approachto correcting these distortions. Cui et al. [12] showthat the depth distortion of ToF cameras is radiallysymmetric and scene dependant. Thus they estimatenew distortion correction parameters for each image.The Kinect device has also shown radially symmetricdistortions [13]. However, being a structured lightsensor, the nature of the distortions is different.

Kinect devices are calibrated during manufactur-ing with a proprietary algorithm. The calibrated pa-rameters are stored in the device’s internal memoryand are used by the official drivers to perform thereconstruction. This is adequate for casual use, butwe have observed that the manufacturer’s calibrationdoes not correct the depth distortion. Other calibrationalgorithms have been developed by the Kinect com-munity. The first algorithms (e.g. [14]) calibrated onlythe intrinsics (focal length and principal point) of theinfrared camera but did not calibrate the parametersto convert kinect disparity units to meters. In our previ-ous work [15], we make a comprehensive calibrationof all parameters of the camera pair. However, depthdistortion was not corrected. Using a similar formu-lation, Zhang and Zhang [16] augment our previouswork with correspondences between the color anddepth images, but still do not address distortion ofthe depth values. Smısek et al. [13] include a depthdistortion correction component as the average ofthe residuals in metric coordinates. We propose adisparity distortion correction that depends on theobserved disparity which further improves accuracy.

1.2 Motivation

As a motivation for our work, we propose threerequirements that an optimal calibration algorithmmust have. To the best of our knowledge, no availablecalibration algorithm for a depth and color camerapair fulfills all three criteria.

Accurate: The method should provide the best com-bination of intrinsic and extrinsic parameters thatminimizes the reprojection error for both cameras overall calibration images. This may seem like an obviousprinciple but we stress it because partial calibrations,where each camera is calibrated independently andthe relative pose is estimated separately, may notachieve the lowest reprojection error.

Practical: It should be practical to use with readilyavailable materials. A high precision 3D calibrationobject is not easy/cheap to obtain and a robotic armor a high precision mechanical setup to record theexact pose of the camera pair is usually not practical,whereas a planar surface is usually readily available.

Widely applicable: To be applicable to a wide range ofdepth sensors, one cannot assume that color disconti-nuities are visible on the depth image. Moreover, some

Fig. 1. Top: Sample calibration images from theexternal camera. Bottom: Disparity images. Note theinaccuracies at the edges and that the checkerboard isnot visible.

depth sensors, like the one used for our experiments,may not provide accurate measurements at sharpdepth discontinuities. Thus, neither color nor depthdiscontinuities are suitable features for depth cameracalibration. The method should use features based ondepth measurements that are most reliable for a widerange of cameras (e.g. planes).

Finally, the increasing popularity of the Kinect de-vices provides an additional motivation for our re-search. For example, the work from Shotton et al. [17]based on the Kinect was selected as the best paper inCVPR 2011. We believe that many applications wouldbenefit from improved accuracy. We have previouslyreleased a Kinect calibration toolbox [15] that hasbeen well received by the developer community. Withthis work we aim to provide an improved calibrationalgorithm for the Kinect community.

1.3 Overview of the approach

We use a planar checkerboard pattern for calibrationwhich can be constructed from any readily availableplanar surface (e.g. a flat table, a wall). We usemultiple views of the calibration plane and for eachview all cameras take an image. The checkerboard cor-ners provide suitable constraints for the color images,while the planarity of the points provides constraintson the depth images. The pixels at the borders ofthe calibration object can be ignored and thus depthdiscontinuities are not needed. Figure 1 shows sampleimages from the external and depth cameras used forcalibration. The three orientations shown constrain thethree dimensions of the relative translation betweendepth and color cameras.

In the color images, the checkerboard corners areextracted. In the depth images, the area containing theplane is located. To initialize the depth camera intrin-sics, the user also selects the corners of the calibrationplane in the depth images. A standard calibrationbased on user selected corners [3] is performed oneach image individually to initialize the calibration.An iterative non-linear bundle adjustment is then

3

performed so that our formulation allows for a closed-form solution of the disparity distortion parameters.

2 CALIBRATION MODEL

Our setup consists of a depth and color camera pairwith an optional high resolution color camera rigidlyattached. Although the camera pair is sufficient forcalibration, in the case of a Kinect device, the internalcolor camera has low quality. Therefore, if one needshigh quality color images with depth maps, an exter-nal camera is very useful. Our system calibrates bothcolor cameras simultaneously. We will refer to thehigh resolution camera as the external camera, whilethe color camera from the camera pair will be referredto simply as the color camera. Our implementationand experiments use the Kinect sensor, which consistsof a projector-camera pair as the depth sensor thatmeasures per pixel disparity. The external camera isa Canon EOS 5D Mark II.

2.1 Color camera intrinsicsWe use a similar intrinsic model as Heikkila [2] whichconsists of a pinhole model with radial and tangentialdistortion correction. The projection of a point fromcolor camera coordinates xc = [xc, yc, zc]

> to colorimage coordinates pc = [uc, vc]

> is obtained throughthe following equations. The point is first normalizedby xn = [xn, yn]

> = [xc/zc, yc/zc]>. Distortion is then

performed:

xg =[2k3xnyn + k4(r

2 + 2x2n)k3(r

2 + 2y2n) + 2k4xnyn

](1)

xk = (1 + k1r2 + k2r

4 + k5r6)xn + xg (2)

where r2 = x2n + y2n and kc = [k1, . . . , k5] is a vectorcontaining the distortion coefficients.

Finally, the image coordinates are obtained:[ucvc

]=

[fcx 00 fcy

] [xkyk

]+

[u0cv0c

](3)

where f c = [fcx, fcy] are the focal lengths and p0c =[u0c, v0c] is the principal point. The same model ap-plies to the color and external cameras. The modelfor each camera is described by Lc = {f c,p0c,kc}.

2.2 Depth camera intrinsicsIn our experiments we used the Kinect as a depthcamera. Yet, the method allows different kinds ofdepth sensors by replacing this intrinsic model. TheKinect’s depth sensor consists of an infrared projectorthat emits a constant pattern and an infrared camerathat measures the disparity between the observedpattern and a pre-recorded image at a known constantdepth. The output consists of an image of scaleddisparity values in Kinect disparity units.

The transformation between depth camera coordi-nates xd = [xd, yd, zd]

> and depth image coordinate

pd = [ud, vd]> follows a similar model to that used for

the color camera. Eq. (3) is used with the respectiveparameters fd and p0d. However, whereas the colorcamera’s distortion is defined in terms of the forwardmodel (world to image), we define the geometricdistortion of the depth camera in terms of the back-ward model (image to world). This is computationallyconvenient in our case because our formulation of thebundle-adjustment in Section 3.3 backward projectsoptical rays for the depth camera but forward projectsoptical rays for the color camera. Further, previousstudies have shown that the lens distortion modeldefined by Eqs. (1) and (2) works well in both ways[2], [18]. Thus, the geometric distortion model for thedepth camera is obtained by simply switching the roleof xn and xk in Eqs. (1) and (2).

The relation between the obtained disparity valued and the depth zd contains two parts: a scaledinverse and a distortion correction. The scaled inversehas been observed by most previous calibration ap-proaches to fit the observations, it is modeled by theequation:

zd =1

c1dk + c0(4)

where c1 and c0 are part of the depth camera intrinsicparameters to be calibrated and dk is the undistorteddisparity (i.e. after distortion correction). Note that forall Kinect devices c1 is negative, which means thathigher disparity values correspond to points fartheraway from the camera (the opposite of traditionalstereo disparity units).

When calibrated using only Equation (4) (i.e. with-out distortion correction) the Kinect displays a fixederror pattern in the measurements (Figure 2). Becausethe internal algorithms of the Kinect are proprietaryand closed, it is not possible to pinpoint the exactnature of this distortion. However, we can correctit based on observations. It was suggested in [13]that this distortion could be corrected by applying aspatially varying offset Zδ to the calculated depth:

zdd = zd +Zδ(u, v) (5)

and it was observed that this usually reduces thereprojection error. However, we have found that amore accurate calibration is made by correcting forthe distortion directly in disparity units.

The shape of the error pattern is constant but itsmagnitude decreases as the distance from the objectincreases. To demonstrate this decay we took the er-rors from planes at several distances and normalizedthem (dividing all images by Figure 2). Figure 3 showsthe resulting median values for each measured dispar-ity. The normalized error fits well to an exponentialdecay. This led us to construct a distortion model thathas per-pixel coefficients and decays exponentiallywith increasing disparity.

4

Fig. 2. Error residuals (kdu) without distortion correc-tion of a plane at 0.56m (left) and 1.24m (right).

300 400 500 600 700 800 900 1000−1

0

1

2

3

4

5

Measured disparity (kdu)

Median of normalized error

Fitted exponential

Fig. 3. Distortion magnitude with increasing disparity.

We use a spatially varying offset that decays as theKinect disparity increases:

dk = d+Dδ(u, v) · exp(α0 − α1d) (6)

where d is the distorted disparity as returned by theKinect, Dδ contains the spatial distortion pattern, andα = [α0, α1] models the decay of the distortion effect.

Note that this models does not enforce any smooth-ness on the spatial distortion pattern. To properlyconstrain this pattern it is enough to include some(four) images of a flat surface that spans the entiredepth image. We add images of an empty wall atseveral depths. These images do not need the checker-board pattern since they are only needed to constrainthe distortion pattern. This ensures that all pixelsin the depth image have samples to estimate theircoefficients Dδ(u, v).

Although this disparity distortion model was devel-oped with the Kinect in mind, it bears similarities withthe model of a ToF camera. Kim et al. [11] obtainedresults similar to Figure 3, except that they fit a 6th

degree polynomial instead of an exponential. Further-more, the calibration of this ToF camera model issimpler because they don’t use per-pixel coefficients.

Equations (4) and (6) are used when measureddisparities are transformed to metric coordinates, alsoknown as the backward model. The inverse of thesefunctions, the forward model, is also needed to com-pute the reprojection errors. The inverse of Equation(4) is straightforward:

dk =1

c1zd− c0c1

(7)

But the inverse of Equation (6) is a bit more involvedbecause of the exponential. We perform two variable

substitutions to isolate the exponential product:

y =exp(α0 − α1dk + α1Dδ(u, v)y)

where y =dk − dDδ(u, v)

y =exp(α1Dδ(u, v)y) exp(α0 − α1dk)

−yα1Dδ(u, v)

= exp(−y) exp(α0 − α1dk)

where y = −yα1Dδ(u, v)

y exp(y) =− α1Dδ(u, v) exp(α0 − α1dk)

The product can be solved using the Lambert Wfunction [19]. The Lambert W function is the solutionto the relation W (z) exp(W (z)) = z.

y =W (−α1Dδ(u, v) exp(α0 − α1dk))

(d− dk)α1 =W (−α1Dδ(u, v) exp(α0 − α1dk))

d = dk +W (−α1Dδ(u, v) exp(α0 − α1dk))

α1(8)

Although the Lambert W function is a trascendentalfunction, there are many accurate approximations inthe literature [19] and modern mathematical packagesinclude implementations of it (e.g. Matlab).

The model for the depth camera is described byLd = {fd,p0d,kd, c0, c1,Dδ,α}, where the first 3parameters come from the model described in section2.1 and the last 4 are used to transform disparity todepth values.

2.3 Extrinsics and relative pose

Figure 4 shows the different reference frames presentin a scene. Points from one reference frame can betransformed to another using a rigid transformationdenoted by T = {R, t}, where R is a rotation andt a translation. For example, the transformation of apoint xw from world coordinates {W} to color cameracoordinates {C} follows xc = WRCxw +W tC .

Reference {Vi} is anchored to the corner of thecalibration plane of image i and is only used forinitialization. The relative poses (DTC and ETC) areconstant, while each image has its own world tocamera pose WiTC . By design, the table and thecheckerboard are coplanar but the full transformationbetween {V } and {W} is unknown.

3 CALIBRATION METHOD

A block diagram of our calibration method is pre-sented in Figure 5. The individual steps are describedin the following sections.

5

{C}

{D}

{Wi}{Vi}

zvzw

ViTD

DTC

WiTC

{E}

ETC

WiTE

Camera Pair

Fig. 4. Reference frames and transformations. {D},{C}, and {E} are the depth, color, and external cam-eras. For image i, {Vi} is attached to the calibrationplane and {Wi} is the calibration pattern.

Relative poseestimation

Checkerboardcorners C

Cornerbased calib

Planecorners

Ld and DTCrefinement

Lc,Le,Ld, WiTC ,ETC ,DTC

Globalrefinement

Cornerbased calib

Disparity images

Cornerbased calib

Checkerboardcorners E

Lc, WiTC

Ld

Le

WiTE

ViTDDisparity distortion

estimation (Dδ)

Fig. 5. Calibration algorithm. Before dashed line: ini-tialization. After dashed line: non-linear minimization.

3.1 Corner based calibration

The calibration of a color camera is a well studiedproblem, we use Zhang’s method [3] to initialize thecamera parameters. Briefly, the steps are the follow-ing. The checkerboard corners are extracted from theintensity image. A homography is then computed foreach image using the known corner positions in worldcoordinates {Wi} and the measured positions in theimage. Each homography then imposes constraints onthe camera parameters which are solved with a linearsystem of equations. The distortion coefficients areinitially set to zero.

The same method is used to initialize the depthcamera parameters. However, because the checker-board is not visible in the depth image, the fourcorners of the calibration plane are extracted (the grayplane in Figure 1). These corners are very noisy andare only used here to obtain an initial guess. Thehomography is thus computed between {Vi} and {D}also using Zhang’s method. This initializes the focallengths, principal point, and the transformation ViTD.Using these initial parameters we obtain an estimatefor the expected depth of each selected corner. Withthis expected depth and the measured disparity anoverdetermined system of linear equations is builtusing (4), which gives an initial guess for the depthparameters (c0 and c1).

3.2 Relative pose estimationThe relative pose between the external and color cam-eras can be obtained directly because their pose withrespect to the same reference frame {W} is known.For the depth camera, however, only the pose withrespect to {V } is known, which is not aligned to {W}.To obtain the relative pose CTD we take advantage ofthe fact that {V } and {W} are coplanar by design.We extract the plane equation in each reference frameand use it as a constraint. We define a plane using theequation n>x− δ = 0 where n is the unit normal andδ is the distance to the origin.

If we divide a rotation matrix into its colums R =[r1, r2, r3] and choose the parameters of the plane inboth frames as n = [0, 0, 1]> and δ = 0, the planeparameters in camera coordinates are:

n = r3 and δ = r>3 t (9)

where we use WiRC and WitC for the color cameraand ViRD and VitD for the depth camera.

As mentioned by Unnikrishnan and Hebert [6] therelative pose can be obtained in closed form fromseveral images. The plane parameters for each imageare concatenated in matrices of the form: M c =[nc1,nc2, ...,ncn], bc = [δc1, δc2, ..., δcn], and likewisefor the depth camera to form Md and bd. The relativetransformation is then:

CR′D =MdM>c (10)

CtD = (M cM>c )−1M c(bc − bd)> (11)

Due to noise CR′D may not be orthonormal. We obtaina valid rotation matrix through SVD using: CRD =UV > where USV > is the SVD of CR′D.

3.3 Non-linear minimizationThe calibration method aims to minimize theweighted sum of squares of the measurement repro-jection errors over all parameters (Lc,Ld,Le,ETC ,DTC ,and ,WiTC for all images i). The error for the colorcamera is the Euclidean distance between the mea-sured corner position pc and its reprojected positionpc (the same for the external camera with pe andpe respectively). Whereas for the depth camera it isthe difference between the measured disparity d andthe predicted disparity d. The predicted disparity isobtained by calculating the distance along the opticalray of the calibration plane and transforming to dis-parity using Eqs. (7) and (8). Because the errors havedifferent units, they are weighted using the inverse ofthe corresponding measurement variance (σ2

c , σ2d, σ2

e ).The resulting cost function is:

c =

∑‖pc − pc‖

2

σ2c

+

∑(d− d)2

σ2d

+

∑‖pe − pe‖

2

σ2e

(12)

Note that (12) is highly non-linear and depends ona lot of parameters (Dδ contains 307200 entries). To

6

separate the optimization of the disparity distortionparameters from the rest we make a slight modi-fication to Equation (12). Instead of comparing thereprojection in measured disparity space, we calculatethe residuals in undistorted disparity space:

c =

∑‖pc − pc‖

2

σ2c

+

∑(dk − dk)2

σ2d

+

∑‖pe − pe‖

2

σ2e

(13)It is also possible to optimize Eq. (12) by inverting

the roles of Eqs. (6) and (8). However, including theLambert W function in the backward camera modelwould make it cumbersome to use for transformingmeasurements into 3D points. We tested both ap-proaches and found no practical advantage of min-imizing Eq. (12) over (13).

The optimization has three steps as shown in Fig.5. The initialization gives a very rough guess of thedepth camera parameters and relative pose, whereasthe color camera parameters have fairly good initialvalues. Thus, the first step optimizes only Ld and DTCwith all other parameters fixed. The optimization thencontinues iteratively with two alternating steps. In thefirst step Dδ is kept constant and Equation (13) isminimized using the Levenberg-Marquardt algorithmover all other parameters. In the second step thespatial disparity distortion pattern Dδ is optimizedindependently for each pixel. The initial values of thedepth distortion model (α and Dδ) are not criticaland initially assuming zero for both has proven toyield accurate results. The algorithm iterates until theresiduals are stable.

3.4 Disparity distortion estimationOptimizing Dδ separately is more efficient becausethe entries in Dδ are independent from each otherand the estimation of Dδ(u, v) takes into account onlymeasurements obtained from pixel (u, v). Moreover,when the other parameters are fixed we can solve forDδ(u, v) in closed-form.

Each disparity measurement d is first undistortedusing Equation (6). We compute a predicted disparitydk using the distance to the plane and Equation (7).We minimize the following cost function to obtain thedistortion parameters:

cd =∑

images

∑u,v

(d+Dδ(u, v) · exp(α0 − α1d)− dk)2

(14)This minimization is straightforward because Equa-tion (14) is quadratic in each Dδ(u, v) and hence theoptimal value of each Dδ(u, v) is obtained by solvinga linear equation.

For comparison we also calibrated using the modelof Smısek et al. [13]. The value of Zδ(u, v) is calculatedas the mean difference between measured depth zdand expected depth zd:

Zδ(u, v) =

∑N zd − zdN

(15)

(a) Our method: Dδ (b) Smısek et al.: ZδFig. 6. Obtained distortion spatial patterns.

TABLE 1Calibration with different distortion models. Std.

deviation of residuals with a 99% confidence interval.

Color Depth External±0.02 px ±0.002 kdu ±0.05 px

A1No correction 0.42 1.497 0.83Smısek [13] 0.32 1.140 0.72Our method 0.28 0.773 0.64

A2No correction 0.36 1.322 0.83Smısek [13] 0.33 0.884 0.85Our method 0.38 0.865 0.79

B1No correction 0.56 1.108 0.97Smısek [13] 0.62 1.300 0.91Our method 0.57 0.904 0.85

4 RESULTS

The color camera of the Kinect delivers images witha resolution of 1280x1024, whereas the resolution ofthe external camera was 2784x1856. Three differentdata sets were captured (A1, A2, and B1). Two ofthem were captured with the same Kinect (A1 andA2) and one with a different Kinect (B1). For each set,the captured images were divided into calibration andvalidation groups with 60 and 14 images respectively.The calibration images were used to estimate all theparameters in the model, then the intrinsic parameterswere kept fixed to estimate only the rig’s pose (WTC)for the validation images. All results presented herewere obtained from the validation images.

We implemented our algorithm in Matlab. The codehas been released as a toolbox for the research com-munity. It can be found in the same website as ourprevious work [15]. We used 60 plane orientationsfor calibration. However, we found that comparableaccuracy is achieved with only 20 positions: 4 for eachorientation shown in Fig. 1 at different depths, and 4of a flat surface that covers the entire depth image.The calibration with 60 positions takes 15min on a2.4GHz computer, but only 3min with 20 positions.

Normally, the different calibrations (A1, A2, andB1) would produce slightly different error variances(σc,σd, and σe). To compare the data sets the varianceswere kept constant (σc = 0.18px, σd = 0.9kdu, andσe = 0.30px).

7

TABLE 2Calibration without the external camera. Std. deviation

of residuals with a 99% confidence interval.

Color Depth±0.02 px ±0.002 kdu

A1 0.26 0.765A2 0.36 0.873B1 0.60 0.902

4.1 Calibration accuracyFigure 6 shows the obtained spatial patterns for thedistortion correction using Eqs. (6) and (5). We canobserve a very similar pattern in both images. Table1 shows a comparison of the calibration results ob-tained using both types of distortion correction andno correction. The three models were calibrated usingthe same data sets and the table shows the results ofvalidation against the respective validation set.

The distortion correction proposed by Smısek etal. [13] improves the reprojection error for data setsA1 and A2. However, because it reduces the errorin metric space it increases the reprojection errorfor B1. In contrast, our approach produces the bestresults for all cases. The standard deviation of thereprojection errors for all sets were both very low(< 1px and < 1kdu), which demonstrates an accuratecalibration. Also note that even though no spatialsmoothness was enforced for the spatial distortionpattern, the obtained pattern is smooth, proving thatthe procedure provides enough constraints.

Table 2 shows the results of calibration without theexternal camera. The calibration accuracy remains thesame. The external camera is thus not necessary foran accurate calibration. Still, its joint calibration is auseful feature for many applications that need a highquality external camera.

4.2 Comparison with manufacturer calibrationThe drivers provided by the manufacturer (Prime-sense) use factory calibrated settings to convert thedisparity measurements to 3D points. We used thesecalibration parameters and compared their perfor-mance to that of our calibration. Using the disparityimages from the A2 data set, both calibrations wereused to obtain 3D points. The calibration of ourmethod was done with the A1 data set to avoid anybias. A plane was fitted to each cloud of points and themeasured depth was compared to the expected basedon the plane’s depth at the given pixel. The errormeasurements are shown in Figure 7 for both calibra-tions. The measurements were grouped by depth in64 bins from 0.4m to 3.7m. For each bin, the standarddeviation of the error was plotted.

The uncertainty was also simulated using the cal-ibrated model. For a given depth, the expected dis-parity for each pixel was calculated using Equations(7) and (8). Gaussian noise (µ = 0 and σ = 0.6) was

0 0.5 1 1.5 2 2.5 3 3.5 40

5

10

15

20

25

30

35

Depth (m)

Err

or s

td. d

ev. (

mm

)

Depth uncertainty

Our methodManufacturer calibrationSimulated, σd=0.6

(a) Full calibrated range

0.4 0.6 0.8 1 1.2 1.4 1.6 1.80

1

2

3

4

5

6

7

Depth (m)

Err

or s

td. d

ev. (

mm

)

Depth uncertainty

(b) Zoom in

Fig. 7. Measurement uncertainty for varying depths.

applied to this disparity and the corrupted depth isobtained through Equations (4) and (6). The standarddeviation of the error between the initial and cor-rupted depths is plotted as a solid line in Figure 7.We can see that these virtual results are very close toour experimental results.

Like any stereo camera, the Kinect is expected tobe more accurate the closer the object is. Due to theinverse relation between disparity and depth, zeromean noise with constant variance on the measureddisparity will result in higher depth uncertainty asthe depth increases. This relation is shown in Figure7. Our method clearly outperforms the manufacturercalibration in ranges up to 1.5m. At 1m distance themanufacturer calibration has twice the uncertainty.After 1.5m the distortion correction has a smallerinfluence in the reconstruction and both calibrationshave similar accuracies.

It is suspected that the infrared image is not distor-tion corrected before the depth estimation algorithm isapplied, which produces the depth distortion pattern.This is why the disparity distortion has the same spa-tial distribution as a radial distortion (i.e. concentriccircles). It is unclear why the depth distortion decayswith depth. The depth estimation algorithm locatesknown point patterns in the infrared image. At fardistances, the point patterns might be closer to thefactory memorized position and the distortion in theinfrared image could have less impact.

4.3 Variability of Kinect devices

To justify the need for calibrating the Kinect weused the calibration of one data set on the validationimages of another data set. The external camera wasnot used for the validation here because its relativepose is different between the different data sets. Theresults are presented in Table 3. They show that thereconstruction using the calibration of another Kinectis highly inaccurate and increases the reprojectionerror considerably, both for color and depth camera.Thus supporting the idea that each Kinect must beindividually calibrated to achieve maximum accuracy.

8

TABLE 3Variability of Kinects. Std. dev. of residuals using

different sets. Dark cells indicate a device mismatch.

CalibA1 CalibA2 CalibB1ValidA1 0.26px, 0.77kdu 0.29px, 0.83kdu 1.35px, 1.55kduValidA2 0.45px, 0.76kdu 0.36px, 0.86kdu 1.77px, 1.80kduValidB1 1.81px, 1.52kdu 1.72px, 1.49kdu 0.56px, 0.89kdu

Fig. 8. 3D reference cube. Color and disparity images.

4.4 3D ground truthWe also compared the accuracy of calibration by re-constructing a hollow cube whose sides are known tobe at 90◦ from each other. Figure 8 shows the referencecube. A point cloud was obtained from the disparityimage and planes were fitted to the points from eachside. Table 4 shows how much the angle betweenthe obtained planes deviates from 90◦. Our methodclearly achieves a better reconstruction accuracy.

5 CONCLUSION

We have presented a calibration algorithm for a depthand color camera pair that is optimal in the senseof the postulated principles. The algorithm takes intoaccount color and depth features simultaneously toimprove calibration of the camera pair system as awhole. It requires only a planar surface and a simplecheckerboard pattern.

The results show that our algorithm achieved abetter calibration for the Kinect than that providedby the manufacturer. The disparity distortion cor-rection model considerably improved reconstructionaccuracy, better than previously proposed models.At 1m distance our calibration showed twice thereconstruction accuracy than the manufacturer’s cal-ibration. Moreover, we have released our code as aMatlab toolbox to the research community.

The extension of the calibration to several externalcolor cameras is straighforward and is already imple-mented in the released toolbox. In addition, we be-

TABLE 4Angular error between reconstructed planes. ∠ab is

the angle between planes a and b.

Manufacturer Smısek [13] Our method90◦ − ∠12 -1.4 1.2 0.690◦ − ∠13 -1.1 1.2 -0.290◦ − ∠23 1.0 -1.0 0.1

lieve that our algorithm is flexible enough to be usedwith other types of depth sensors by replacing theintrinsics model of the depth camera. The constraintsused can be applied to any type of depth sensor.Future work can include the calibration of a ToF andcolor camera pair.

AcknowledgementsThis project has been funded by the Academy ofFinland’s project #127702.

REFERENCES[1] S. G. Latta, Gesture keyboarding, uS 2010/0199228 A1 (Aug.

2010).[2] J. Heikkila, Geometric camera calibration using circular control

points. IEEE Transcations on Pattern Analysis and MachineIntelligence 22(10), 1066-1077.

[3] Z. Zhang, Flexible camera calibration by viewing a plane fromunknown orientations, in: ICCV, 1999, pp. 666–673.

[4] S. Fuchs, G. Hirzinger, Extrinsic and depth calibration of ToF-cameras, in: CVPR, 2008, pp. 1–6.

[5] Q. Zhang, R. Pless, Extrinsic calibration of a camera and laserrange finder (improves camera calibration), in: IROS, Vol. 3,2004, pp. 2301–2306.

[6] R. Unnikrishnan, M. Hebert, Fast extrinsic calibration of alaser rangefinder to a camera, Tech. Rep. CMU-RI-TR-05-09,Robotics Institute, Pittsburgh (2005).

[7] D. Scaramuzza, A. Harati, R. Siegwart, Extrinsic self calibra-tion of a camera and a 3D laser range finder from naturalscenes, in: IROS, 2007, pp. 4164–4169.

[8] M. Lindner, A. Kolb, Calibration of the intensity-related dis-tance error of the PMD TOF-Camera, in: SPIE: IntelligentRobots and Computer Vision XXV, Vol. 6764, 2007.

[9] D. Lichti, Self-calibration of a 3D range camera, ISPRS 37 (3).[10] J. Zhu, L. Wang, R. Yang, J. Davis, Fusion of time-of-flight

depth and stereo for high accuracy depth maps, in: CVPR,2008, pp. 1–8.

[11] Y. Kim, D. Chan, C. Theobalt, S. Thrun, Design and calibrationof a multi-view ToF sensor fusion system, in: IEEE CVPRWorkshop on Time-of-flight Computer Vision, 2008.

[12] Y. Cui, S. Schuon, D. Chan, S. Thrun, C. Theobalt, 3d shapescanning with a time-of-flight camera, in: Proc. of IEEE CVPR,2010, pp. 1173–1180.

[13] J. Smısek, M. Jancosek, T. Pajdla, 3D with Kinect, in: IEEEWorkshop on Consumer Depth Cameras for Computer Vision,2011.

[14] N. Burrus, Kinect calibration (Nov. 2011).URL http://nicolas.burrus.name/index.php/Research/KinectCalibration

[15] D. Herrera C., J. Kannala, J. Heikkila, Accurate and practicalcalibration of a depth and color camera pair, in: CAIP 2011,Part II, LNCS 6855, 2011, pp. 437–445.URL http://www.ee.oulu.fi/∼dherrera/kinect/

[16] C. Zhang, Z. Zhang, Calibration between depth and color sen-sors for commodity depth cameras, in: International Workhsopon Hot Topics in 3D, in conjunction with ICME, 2011.

[17] J. Shotton, A. Fitzgibbon, M. Cook, A. Blake, Real-time humanpose recognition in parts from single depth images, in: CVPR,IEEE, 2011, pp. 1297–1304.

[18] D. Brown, Close-range camera calibration, PhotogrammetricEngineering 37 (8) (1971) 855–866.

[19] D. Barry, J.-Y. Parlange, L. Li, H. Prommer, C. Cunningham,F. Stagnitti, Analytical approximations for real values of theLambert W-function, Mathematics and Computers in Simula-tion 53 (1-2) (2000) 95–103.