IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 26, NO. 10, OCTOBER 2017 4951

Inverse Rendering and Relighting From MultipleColor Plus Depth Images

Siying Liu, Member, IEEE, and Minh N. Do, Fellow, IEEE

Abstract— We propose a novel relighting approach thattakes advantage of multiple color plus depth images acquiredfrom a consumer camera. Assuming distant illumination andLambertian reflectance, we model the reflected light field interms of spherical harmonic coefficients of the bi-directionalreflectance distribution function and lighting. We make use ofthe noisy depth information together with color images takenunder different illumination conditions to refine surface normalsinferred from depth. We first perform refinement on the surfacenormals using the first order spherical harmonics. We initializethis non-linear optimization with a linear approximation togreatly reduce computation time. With surface normals refined,we formulate the recovery of albedo and lighting in a matrixfactorization setting, involving second order spherical harmonics.Albedo and lighting coefficients are recovered up to a globalscaling ambiguity. We demonstrate our method on both simulatedand real data, and show that it can successfully recover bothillumination and albedo to produce realistic relighting results.

Index Terms— Inverse rendering, illumination, relighting,reflectance.

I. INTRODUCTION

L IGHTING plays an important part in visual appearance.Similar in spirit to facial make-up, good lighting helps to

enhance the aesthetic appeal of subjects imaged. This moti-vates the problem of computationally touching up photographsto mimic desirable lighting effects. Computational relightingis of broad interest in areas such as computer graphics andcinematography. Imagine holding a video chat on Skype andbeing able to “cast” studio lightings to make one’s face moreappealing. Another application is augmented/mixed realitywhere images of objects from different environments need tobe computationally relighted to achieve an overall uniformlighting.

Image relighting refers to computationally manipulating thelighting effects given one or more image(s) of the scene,

Manuscript received October 7, 2016; revised April 6, 2017 andJune 3, 2017; accepted June 26, 2017. Date of publication July 18, 2017; dateof current version August 7, 2017. This work was supported in part by theAgency of Science, Technology and Research (A*STAR), Singapore and inpart by the US National Science Foundation under Grant CCF-1218682. Theassociate editor coordinating the review of this manuscript and approving itfor publication was Prof. Aljosa Smolic. (Corresponding author: Siying Liu.)

S. Liu is with the Institute for Infocomm Research, A*STAR,Singapore 138632, and also with the Coordinated Science Laboratory, Depart-ment of Electrical and Computer Engineering, University of Illinois atUrbana–Champaign, Urbana, IL 61801 USA (email: sliu50@illinois.edu).

M. N. Do is with the Department of Electrical and Computer Engineering,Coordinated Science Laboratory, Beckman Institute, University of Illinois atUrbana–Champaign, Urbana, IL 61801 USA (e-mail: minhdo@illinois.edu).

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TIP.2017.2728184

Fig. 1. The rendering pipeline involving BRDF, scene geometry and lighting.In the inverse rendering process, given the input images (same viewpoint,different lighting) and any two of the three mentioned scene components,the rest can be inferred. Relighting can be seen as a forward rendering processin which the lighting component is replaced by a novel one for the synthesisof a new image.

to produce desirable and photorealistic renderings. Whileimage relighting has been widely used on synthetic scenesconstructed from computer graphics models as seen in manycomputer games, or on real scenes captured under controlledlighting conditions, efficient relighting on general real scenesremain a research challenge due to its complexities involvingboth the geometric and photometric aspects underlying theimage formation process. An illustration of the relightingprocess is presented in Fig. 1. State-of-the-art relightingmethods often consist in estimation of the scene geometry,the BRDF (bi-directional reflectance distribution function,which models how light interacts with the surface) and theillumination directions. Given only the image data, the esti-mation of all these factors is ill-posed, and can only beachieved by strict assumptions of the scene geometry and/orlight sources, or by controlled experiments.

The earliest attempt to render lighting effects on a syntheticobject involved using a spherical mirror called the lightprobe [1]. A light probe captures environment illuminationsfrom all directions. To render an object in a given lightingenvironment, the light probe is placed at the location of theobject to measure the incident illumination. Lighting effectsare added onto the object by mapping texture based on surfacenormal directions. This method is widely adopted in themovie and gaming industries, but it requires knowing both thereflectance and geometry of the object for relighting. Anothernotable approach is to capture relightable representations

1057-7149 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

4952 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 26, NO. 10, OCTOBER 2017

of a scene. Debevec et al. [2] proposed a method to denselysample the reflectance field of the human face with highangular resolution using a dome-shaped setup called LightStage. The method is able to render realistic lighting resultsbut requires highly controlled data capturing. More recentworks on computational relighting have been focusing on theincorporation of 3D information of scenes to parametricallymodel the lighting and surface reflectance such as [3], but theyassume homogeneous surface albedos to simplify the problem.

Recently, consumer grade depth cameras, e.g., MicrosoftKinect and Intel RealSense, have gained increased popularitydue to its low cost and real-time depth acquisition capabil-ity. Seminal works on shape refinement by fusing multipledepth maps include [4] for rigid and [5] for dynamic scenes.In this work, we explore the incorporation of geometry (depth)information obtained through a depth sensor, in a relightingproblem of a convex object, under general, arbitrary distantillumination. Applying Lambertian reflectance and modelingillumination in terms of spherical harmonics (SH) [6], [7],we formulate an efficient relighting algorithm for practicalapplications such as video conferencing and stylistic light-ing transfer that can be readily used by lay users, withoutexpensive data processing. The input to our system includemultiple RGB images and a corresponding depth map. TheRGB images are taken under different lighting conditions,while the depth map is acquired by a depth sensor, wheresurface normals can be derived from. However, the surfacenormals are often of poor quality since the acquired depthcould be noisy. We first show that through combining intensityand depth information, the noisy normals can be refined toa satisfactory level to enable solving of albedo and lightingparameters for relighting. This work is an extension of ourprevious work [8]. Instead of assuming the given surfacenormals were clean as in [8], we account for noisy surfacenormals from the depth sensor. We refine them in a shape-from-shading setting through a nonlinear optimization thatseeks to resolve the linear ambiguity arising from matrixfactorization. We further show that this nonlinear optimizationcan be significantly sped up when initialized by a linearoptimization. We solve for surface normal and albedo in aprogressive manner: using the first-order spherical harmonicsto refine noisy normal, followed by the second-order sphericalharmonics for albedo recovery.

The outline of our paper is as follows. The backgroundand related works are presented in Section II. Details ofthe proposed formulation is presented in Section III toSection VII-A.2. Experimental results and evaluations aregiven in Section VII and lastly, Section VIII concludes thepaper.

II. BACKGROUND AND RELATED WORK

Strictly speaking, the physically correct way to relight ascene is to follow the rendering equation [9]. One way ofdoing so is to measure the scene’s light transport, a discreterepresentation of the rendering equation which encodes theinteraction between the input illumination and the scene, aswell as the lighting interactions within the scene itself, knownas interreflections. However, since light transport is very

expensive to measure or pre-compute, it is often precludedin practical relighting systems. For efficiency, many relight-ing methods approach the problem by either interpolatingilluminations from examples or by parametric modeling ofdirect illumination (illumination due solely to light sources)and reflectance using sparse samples of the scene. The effectsof interreflections are neglected. The proposed method alsomakes an assumption on the convexity of the scene toavoid handling interreflections. In the following subsections,we review major computational relighting methods as well asworks on depth/shape estimation through RGB-D fusion.

A. Illumination Interpolation

These approaches use images captured under differentknown and controlled illumination conditions, without priorknowledge of the scene geometry and surface properties.Taking advantage of the linearity of the rendering operatorwith respect to illumination, novel lighting can be synthesizedas a linear combination of a set of basis images. The idea wasfirst introduced by Nimeroff et al. [10] and later popularizedby Debevec et al. [2] who pioneered the dense capturing ofreflectance field of the human face with high angular resolutionwith a setup called Light Stage. The setup is a sphericalstructure mounted with 156 white LED light sources. Lightingdirections are changed thousands of times per second and thecorresponding appearances of the subject is captured by ahigh-speed camera. While this method is able to synthesizerealistic lighting effects, it is data-expensive as it often requiresdense sampling of viewpoints and illumination directionsfor the interpolation of novel lighting. A highly controlledenvironment is also required to allow for precise measurementof illumination. In contrast, our work only requires a sparse setof samples for the estimation of both illumination and diffusealbedo.

B. Environment Mapping

Compared to interpolation-based relighting techniques thatrequire dense sampling, the data requirement is greatly reducedwith a light based model. In computer graphics, syntheticobjects can be rendered using a pre-computed environmentmap called the light probe [1]. A light probe captures envi-ronment illuminations from all directions (omnidirectionalirradiance map) by imaging a mirror sphere with high-dynamic-range capturing. It can be viewed as a look-up tablefor environment lighting indexed by surface normal vectors.The captured lighting environment can be used together withsurface reflectance properties, i.e., the BRDF, to generatephoto-realistic rendering. To render a synthetic object in agiven lighting environment, the light probe is used to measurethe incident illumination at the location of the syntheticobject. Lighting effects are added onto the synthetic object bymapping texture from the light probe based on surface normaldirections.

Wen et al. [11] introduced Radiance EnvironmentMap (REM) for face relighting. Pre-integrated with the sur-face’s BRDF, radiance environment map avoids the time-consuming integration of illumination with surface reflectance,

LIU AND DO: INVERSE RENDERING AND RELIGHTING FROM MULTIPLE COLOR PLUS DEPTH IMAGES 4953

and is particularly useful for rendering surfaces with com-plex reflectance properties. Assuming diffuse reflectance andknown 3D geometry, ratio-image technique was used to relightfaces when the environment lighting rotates and to transferlighting from one person’s face to another. However, thismethod assumes that the two faces have similar skin albedos.

C. Illumination, Shape and Reflectance ModelingFrom Multiple Views

Illumination, surface reflectance of scene points can alsobe inferred by varying viewpoints. The central idea ofsuch methods is to relax the requirement for controlledillumination. Using known surface geometry, Yu et al. [12]proposed to gather images under different illumination con-ditions using multi-view images captured under a single,unknown illumination setting. They assume a dichromaticreflection model, which includes both Lambertian and specu-lar reflections. The directional specular reflection is used toresolve the texture-illumination ambiguity of diffuse scenecomponents, and to estimate the illumination. Estimationof Lambertian reflectance then proceeds after illumina-tion is recovered. Since Lambertian reflection is viewpoint-independent, the scene needs to contain specular reflectionfor illumination to be recovered. To overcome this limitation,Joshi and Kriegman [13] recovered shape under varying view-point and lighting by combining multi-view and photometricstereo to derive a multi-ocular photometric matching cost.However, these methods require dense correspondence acrossmultiple views for initial depth map recovery, a computation-ally expensive step that is prone to errors in matching low-texture regions.

D. Parametric Relighting With Measured 3D Model

With depth cameras, scene geometry can now be efficientlymeasured, thus enabling parametric modelling of illumination.This frees us from the highly controlled data acquisitionprocess as seen in interpolation based methods. In [3], relight-ing results were first demonstrated using the RGBZ videodata where the depth acquired from a time-of-flight sensoris augmented by spatiotemporal upsampling and denoising.The environment lighting was modeled in terms of sphericalharmonic basis functions and estimated in an offline processfrom a single video frame, assuming white light source anddiffuse albedo. In [14], the relighting problem was formulatedas a factorization of the image matrix into a product of adiagonal albedo and a lighting matrix with a known SH matrix(surface normals represented in terms of spherical harmonics)in the middle. This special matrix factorization is unique ifand only if the SH matrix associated with the object hasnonseparable full rank. Our proposed method is most relatedto [3], in that we acquire the 3D model in real-time usinga depth sensor, but we make use of multiple images underdifferent lighting conditions to estimate surface normal fromphotometric data. We use the noisy depth map to resolvethe linear ambiguity in an uncalibrated photometric stereoproblem.

E. Shape/Depth Enhancement Through RGB-D Fusion

Having accurate shape and depth estimate is an importantintermediate goal towards relighting. Fusing intensity data andnoisy depth measurement helps to enhance the accuracy ofthe depth maps. The intensity data could be in the form of asingle image or multiple images under different illuminationconditions. Recent techniques such as [15] estimate shapefrom a single image under natural illumination, but it dealswith uniform albedo objects and requires a calibration objectto measure lighting. Yu et al. [16] used a single intensityimage. Noisy depth information was used in to recover relativealbedo, which helps to separate shading from albedo. Theestimated shading was then used to enhance the quality ofsurface normals. Han et al. [17] modeled natural lighting asa combination of global and local components, where theglobal component was solved using the noisy depth. Next,local lighting model was estimated to account for spatiallyvarying illumination. Surface normals were refined using theestimated lighting and later fused with depth information usingthe algorithm by Nehab et al. [18]. Wu et al. [19] presenteda real-time shading-based depth refinement by reformulatingthe inverse rendering optimization into a highly paralleliz-able algorithm that can be efficiently solved on the GPU.Or-El et al. [20] improved the accuracy of this framework byincorporating the Retinex constraint [21] to handle spatiallyvarying albedo. Hudon et al. [22] further improved the shaperefinement by capturing with sequential illumination, to recorda pair of images with and without flash. The effect of ambientillumination (contained in the non-flashed image) can thusbe cancelled out. Barron and Malik [23] recovered intrinsicproperties of a scene by using a learned mixture model ofshapes and illuminations. They used the noisy depth map fromthe RGB-D data to improve low-frequency shape estimation.

There are also works that used images under multiple illu-mination conditions. In [24], the authors combined noisy depthacquired from Kinect and surface normals evaluated usingcalibrated light sources. They extended the formulation of [18]to handle degenerate cases due to insufficient light sources.In [25], the authors assumed uniform albedo and used and theinfrared channel from the depth sensor to guide multi-viewreconstruction. Similar to [24], our proposed formulation usesmultiple illuminations and depth images. However, we assumeunknown, arbitrary light sources and spatially varying albedo.Instead of having depth and surface normal from independentsources, we work with surface normals derived from noisydepth and show that they can be refined using the first-orderspherical harmonic approximation.

An overview of the discussed techniques on shape (depth)recovery/enhancement is presented in Table I, where we listevery method’s input, output and assumptions. For complete-ness, we also included [13] and [12] that made use of multi-view reconstruction to obtain initial depth estimates.

III. MODELLING GENERAL ILLUMINATION

Ramamoorthi et al. [26] showed that lighting, BRDF andreflectance can be expressed in a signal processing framework.Neglecting the effects of cast shadow and interreflection,

4954 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 26, NO. 10, OCTOBER 2017

TABLE I

AN OVERVIEW OF VARIOUS SHAPE (DEPTH) RECOVERY/ENHANCEMENT TECHNIQUES

the irradiance E from a convex object is a function ofthe surface normal n, given by an integral over the upperhemisphere �(n)

E(n) =∫

�(n)L(ω)(n · ω)dω, (1)

where L denotes the distant lighting distribution and ω denotesthe incident angle between the surface normal and L. ForLambertian reflectance, E is scaled by the surface albedo,which may be dependent on position p. The radiosity I , whichcorresponds directly to the image intensity1 is given by

I (p, n) = ρ(p)E(n). (2)

Furthermore, E can be expressed in terms of sphericalharmonics basis functions Ylm as follows

E(n) =∞∑

l=0

l∑m=−l

El,mYl,m(n), (3)

where l ≥ 0, −l ≤ m ≤ l. Ylm are the analogue onthe sphere to the Fourier basis on the line or circle [6].Elm is the coefficient of the spherical harmonic expansionof the irradiance E , and n = (nx , ny, nz) is represented inglobal Cartesian coordinates. There are 2l + 1 basis functions,−l ≤ m ≤ l, for a given order l ≥ 0.

For a Lambertian scene illuminated by any lighting con-ditions, it can be shown that the reflected light field iswell approximated up to order l = 2 using only the first9 terms of its spherical harmonics expansion, where theaccuracy of the approximation exceeds 98% [6]. A formalanalysis by Basri et al. [7] showed that under a point sourceillumination, the accuracies of the first and second order

1This means that the camera is linear or it is radiometrically calibrated tohave a linear mapping from scene radiance to image intensity.

harmonics approximations are 87.5% and 99.2%, respectively.order 1 approximation with four terms.

Following the derivation in [26], (3) can be simplified usingthe spherical harmonic representation

E(n) ≈2∑

l=0

l∑m=−l

�l Al Ll,m Yl,m(n), (4)

where �l =√

4π2l+1 is a normalization constant, Al is the

spherical harmonic coefficient of the transfer function (n · ω),and Ll,m is the coefficients of spherical harmonic expansionof the illumination L, which is unknown. The specific formsof the first 9 terms of Ylm [26] are given as follows:

Y0,0(n) = 1√4π

, Y1,−1(n)=√

3

4πny, Y1,0(n)=

√3

4πnz,

Y1,1(n) =√

3

4πnx , Y2,−2(n) =

√15

4πnx ny,

Y2,−1(n) =√

15

4πnynz, Y2,0(n) =

√5

16π(3n2

z − 1),

Y2,1(n) =√

15

4πnznx , Y2,2(n) =

√15

16π(n2

x − n2y). (5)

The specific forms of Al up to order 2 are listed as follows:

A0 = π4 , A1 = π

3 , A2 =√

5π8 . (6)

Substituting (4) into (2), the acquired image intensity canbe expressed as a single summation with only 9 terms,

I (p, n) = ρ(p)E(n) ≈ ρ(p)

8∑i=0

hi (n)Li , (7)

where

hi (nx , ny, nz) ↔ �l AlYl,m (nx , ny, nz), (8)

LIU AND DO: INVERSE RENDERING AND RELIGHTING FROM MULTIPLE COLOR PLUS DEPTH IMAGES 4955

with i = l2 + l + m. Here, hi and Li represent the sphericalharmonic function and its corresponding lighting coefficient ateach combination of (l, m), respectively.

Putting all scene points in matrix form, an image i can beexpressed as

i1×N ≈ l1×9H9×N �N×N , (9)

where l = [L1, · · · , L9

]. Hi j corresponds to the

i th harmonic component of the j th pixel. H can be computedexplicitly given the surface normals of all scene points. � isa diagonal matrix whose diagonals are ρ j , 1 ≤ j ≤ N , andN is the number of pixels.

We define the shape matrix S = H�, whose rows areharmonics images, i.e, the shape harmonics (expressed interms of surface normals) scaled by the albedo at each pixel.The ambient illumination is accounted for by simply scalingthe zeroth order spherical harmonic basis function (since itrepresents equal distribution in all directions).

IV. INVERSE RENDERING

The goal of relighting consists in recovering both the diffusealbedos and the lighting coefficients, followed by manipulatingthe lighting coefficients to create novel lighting effects.

Assuming we have K images of the same viewpoint cap-tured under different lighting conditions, they can be assem-bled into an K × N data matrix M, where K denotes thenumber of frames and N denotes the number of pixels in eachimage. Stacking (9) for multiple images, the data matrix Mcan be approximated by the linear combination of harmonicimages

M ≈ LS, (10)

where L(K × r) contains the lighting coefficients correspond-ing to the low order harmonics, e.g., r = 9 as in (9), andshape matrix S(r × N) contains the corresponding harmonicimages. Hence, M has low rank, up to r . The above is aclassical shape-from-shading set-up, also termed PhotometricStereo [27], where S is to be estimated given M and L (maybe unknown in the case of photometric stereo with unknownlight sources).

Using Singular Value Decomposition (SVD), we candecompose M into

M = UK×K �K×N V�N×N , (11)

where U and V are orthonormal while � is a diagonal matrixcontaining the singular values of M.

The bulk of the energy is contained in the first r com-ponents. Assuming K , N ≥ r , we seek an r -dimensionalsubspace for the shape and lighting matrices,

M = LS, (12)

where L = U√

�K r and S = √�r N V� and �K r denotes

the first r columns and �r N denotes the first r rows of �,respectively.

√· denotes the non-negative square root. Effec-tively, (12) denoises the data matrix M by best approximatingit (in the Frobenius norm) with a low-rank r matrix. Thisdecomposition is based on the finding in [28] that the low order

spherical harmonics, which form an orthonormal basis, lievery close to the subspace constructed by SVD. The analysisshowed that for a convex, Lambertian object under arbitrary,distant illumination, the principal components or eigenvectorsare identical to the spherical harmonic basis functions evalu-ated at the surface normal vectors. However, the decomposi-tion (10) is unique only up to an r × r linear ambiguity Q.If Q is any invertible matrix, M = (LQ−1)(QS) is also a validdecomposition.

Lemma 1 (Linear Ambiguity): Suppose M = LS whereL ∈ R

K×r , S ∈ Rr×N and K > r , and L has full column

rank, then the SVD-based decomposition M = LS returns asolution that S = Q−1S, where Q is an invertible r ×r matrix.

Proof: Since L, L have full column rank (= r ), thenthere exists an invertible Q ∈ R

r×r such that L = LQ−1.From M = LS = LS, we get

LQ−1S = LS (13)

Since L has full column rank, there exists a left inverse LL

such that LLL = Ir×r . Pre-multiplying both sides of (13)by LL, we get

Q−1S = S.

Hence, S = QS.We seek to find a linear transformation Qr×r that best

recovers S. However, as we are given the surface normalsderived from the depth data, we only know H [as in (9)]without the albedo information. Each column of the matrix Scan be written as the corresponding column of H scaled byalbedo, s j = ρ j h j , where ρ j is the j th element of the N × Ndiagonal matrix � containing the albedo values,

S = H� = QS, (14)

where H is a 9 × N matrix.Remark 1 (Relation to Photometric Stereo Under

Unknown, General Lighting): In an uncalibrated photometricstereo problem, lighting, surface normal and albedo areall unknown. However, with first-order approximation ofspherical harmonics, Basri et al. [29] showed that byexploiting the geometric meaning of the first 4 sphericalharmonics, the harmonic images S in (10) can be recoveredup to a scaled Lorentz transformation with seven degreesof freedom. The ambiguity can be further removed withadditional constraints such as knowing the albedo andsurface normals of two points in the scene, or by enforcingintegrability as in [30], which would be prone to errorsif the scene contains depth discontinuities. This ambiguityalso means that lighting L can only be recovered up to ascaled Lorentz transformation. This is clearly insufficientfor our purpose of relighting where the directions ofdesirable light sources are to be specified without ambiguity.Hence, we propose to incorporate depth information in ourformulation to resolve the mentioned ambiguity. Incorporatingdepth information also enables us to efficiently solve forlighting and albedo quantities, which would otherwise be atime-consuming process inhibitive to any real-time relightingsystems.

4956 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 26, NO. 10, OCTOBER 2017

Fig. 2. Flow-chart of the proposed inverse rendering and relighting system.See text for details.

An overview of the proposed relighting system is shownin Fig. 2. The input consists of multiple images taken underdifferent illumination conditions and an acquired depth map,which is often noisy. Surface normals are derived from thenoisy depth. We first perform refinement on the surfacenormals using the first order spherical harmonics. With surfacenormals refined, we formulate the recovery of albedo andlighting, involving second order spherical harmonics. Finally,we generate novel relit images following the forward renderingprocess. Details of each step are elaborated in the subsequentsections.

V. NORMAL REFINEMENT FROM NOISY

DEPTH AND INTENSITY DATA

Since the depth map from consumer devices like Kinectis often noisy and may contain holes, the surface normalsderived from such depth measurement would be too noisy forthe purpose of lighting and albedo recovery. However, theycan be used as a shape prior to constrain the solution space ofshape-from-shading, leading to a more accurate surface normalrecovery.

We have shown previously that there exists a non-singularmatrix Q such that S = QS. In the case of 4 harmonics(r = 4), the first four harmonics span approximately the samespace as the first four principal components. As discussedin [29], the scaled surface normals lie in the row space of S.Our goal is to find the linear ambiguity Q. We first discussa linear solution to estimate Q, followed by a non-linearrefinement.

Let Z denote the depth values along a surface, the compo-nents of the surface normals can be computed by taking thepartial derivatives of the surface using filter convolution:

nx = Z ∗ Kx

B, ny = Z ∗ Ky

B, nz = 1

B(15)

where Kx , Ky and B are given by

Kx = 1

8

⎡⎣ 1 0 −1

2 0 −21 0 −1

⎤⎦, Ky = 1

8

⎡⎣ 1 2 1

0 0 0−1 −2 −1

⎤⎦

and B =√

1 + (Z ∗ Kx )2 + (Z ∗ Ky)2.Consider a column corresponding to a pixel of the equation

S = QS as

si = Qsi . (16)

By construction of the first 4 spherical harmonics,from (5) and (7), we have

si = [ρi ρi nix ρi niy ρi niz ]�.

Hence, given depth information which infer normal vector n,we know s up to a scale factor.

A. Linear Estimation of Q

We note that Q is a Projective Transformation relating eachpair of P3 points {si , si } in homogeneous coordinates,

Qsi = ρi

⎡⎢⎢⎣

1nix

niy

niz

⎤⎥⎥⎦ = ρi n′

i , (17)

where ρi is the albedo value of pixel i , and n′i =[

1 nix niy niz]�

denotes the surface normal vector inhomogeneous form.

Equation (17) bears much resemblance to the case of planarhomography in the 2D projective space P2 , except that “1” isnow in first element of n′

i . Kanatani [31] proposed a method tocompute the homography H from P2 point correspondences,λi v = Hu, where both u and v are normalized to norm 1,and H is normalized so that det (H) = 1. This computationof projective transform in P2 can be generalized to the casein P3 as follows.

Our problem from (17) is, given si and n′i , find Q such that

the vector Qsi is parallel to n′i . Hence, (17) is equivalent to

n′�i Qsi

‖n′i‖‖Qsi‖ = ±1 ⇔ (n

′�i Qsi )

2 = ‖n′i‖2‖Qsi‖2

However, we have from Cauchy-Schwarz inequality

(n′�i Qsi )

2 ≤ ‖n′i‖2‖Qsi‖2.

Without loss of generality, we normalize all si and n′i such

that ‖si‖ = 1 and∥∥n′

i

∥∥ = 1, therefore, we can set the errorfor each pair {n′

i , si } to be

ei =‖ Qsi ‖2 −(n′�i Qsi )

2. (18)

The problem is to compute Q by minimizing the followingobjective function:

J (Q) =N∑

i=1

ei =N∑

i=1

(‖ Qsi ‖2 −(n

′�i Qsi )

2)

=N∑

i=1

(s�

i Q�Qsi − (n′�i Qsi )

2)

(19)

Let

Q =

⎡⎢⎢⎣

q�1

q�2

q�3

q�4

⎤⎥⎥⎦

4×4

,

where qi denotes the i th column in Q, and x =[ q�

1 q�2 q�

3 q�4 ]� be the 16 parameter vector to be com-

puted, we can show that the cost function in (19) takes thefollowing form,

J (Q) = J (x) = x�N∑

i=1

Ai x, (20)

LIU AND DO: INVERSE RENDERING AND RELIGHTING FROM MULTIPLE COLOR PLUS DEPTH IMAGES 4957

where Ai is a symmetric matrix given by

Ai =

⎡⎢⎢⎣

si

si

si

si

⎤⎥⎥⎦

16×4

(I4 − n′i n

′�i )

⎡⎢⎢⎣

s�i

s�i

s�i

s�i

⎤⎥⎥⎦

4×16(21)

and I4 is a 4 × 4 identity matrix.Let A = ∑N

i=1 Ai , as Q is to be recovered up to an unknownscale, we can assume that tr(Q�Q) = 1, hence, ‖x‖ = 1. Thesolution is the eigenvector of A corresponding to the smallesteigenvalue. Since we only have noisy surface normals derivedfrom depth, we use the noisy normals n′

i in place of n′i in (21)

when solving for Q.

B. Non-Linear Optimization

Having recovered an initial estimate for Q by the linearmethod, we move on to the non-linear optimization of Q tofurther refine it. From (14), we know that

QS = β

⎡⎢⎢⎣

ρ1 ρ2 · · · ρN

ρ1n1 x ρ2 n2x · · · ρN nN x

ρ1n1 y ρ2 n2 y · · · ρN nN y

ρ1 n1z ρ2n2z · · · ρN nN z

⎤⎥⎥⎦, (22)

where β is a non-zero scalar ambiguity.Let N4×N be the noisy surface normals derived from the

depth map, we seek a Q such that

Q∗ = arg minQ

‖ (QS)(2:4,:) � D − N ‖2F , (23)

where

D =⎡⎣ (QS)(1,:)

(QS)(1,:)(QS)(1,:)

⎤⎦

is formed by stacking the first row of QS. “�” is an element-wise division operator, i.e., if x = u � v, x(i) = u(i)/v(i).

The above optimization is non-linear. To speed up itsconvergence, we approximate Q by the linear method dis-cussed earlier and use it as an initialization in the non-linearoptimization. Note that taking the first row of the product QSgives us the albedo for every pixel, up to a global scalingambiguity. Hence, we obtain an approximation for the albedomatrix � using the first-order harmonics. In practice, we caniterate the process in (23) to further speed up convergence. Letthe Q∗t be the optimal solution from the t th iteration, we canobtain Nt = (Q∗t S)(2:4,:) � D, Using this in the next iteration,we have

Q∗(t+1) = arg minQ

‖ (QS)(2:4,:) � D − Nt ‖2F . (24)

To investigate the time saving contributed by the lin-ear initialization, we simulated noisy depth maps using theMIT-Berkeley dataset which contains 20 real scenes withground truth depth maps (refer to VII-A.1 for details in experi-ment setting and evaluation on normal refinement). The typicalimage size of this dataset is 467 ×367. In our experiment, forthe non-linear optimization part, we iterate the process (24)twice. Comparison of average run time with and without the

TABLE II

COMPARISON OF AVERAGE RUN TIME FOR NORMAL REFINEMENT WITHAND WITHOUT LINEAR INITIALIZATION. THE LAST COLUMN SHOWS

THE AVERAGE RUN TIME FOR LINEAR INITIALIZATION ITSELF

linear initialization is presented in Table II, where we canobserve that the linear initialization speeds up the optimizationby nearly an order of magnitude.

VI. ALBEDO AND LIGHTING RECOVERY

Upon refinement of noisy surface normals, we proceed torecover lighting and albedo of the scene. Although albedocan be recovered through finding Q in the previous normalrefinement step (Section V), it only involves the first orderspherical harmonics and is therefore less accurate. Usingthe refined normals, we can compute H using (8). We thenjointly solve for the linear ambiguity Q and albedo �, takinginto account second order spherical harmonics. When thezeroth through second order spherical harmonics are used,the parameter r in the SVD based factorization (12) is 9. Q isnow a 9 × 9 linear ambiguity.

We applied an iterative method by solving for Q and � inan alternating fashion, fixing one while solving for the otherunknown. The cost function is defined as

J (Q,�) �‖ QS − H� ‖2F (25)

We use the albedo values estimated from the first-orderapproximation to initialize � and estimate Q as

Q = arg minQ

‖ QS − H� ‖2F , (26)

With Q fixed, we can estimate S as S = QS and solve for � as

� = arg min�

‖ S − H� ‖2F (27)

Since � is a diagonal matrix, � = diag{ρ1, · · · , ρN },the optimization in (27) becomes

minN∑

j=1

‖ s j − ρ j h j ‖2, (28)

where s j and h j are the j th column in S and H, respectively,and ρ j is the albedo of the j th pixel. (28) is minimizedterm-wise as

arg minρ j

‖ s j − ρ j h j ‖2,

and the optimal solution is the projection of s j onto h j

ρ j = 〈s j , h j 〉‖h j‖2 . (29)

Thus, � = diag{ρ1, · · · , ρN }.The above alternating minimization repeats until the cost

J (Q,�) is sufficiently small. Once Q is known, we can find Las L = LQ−1. New lighting effects can be rendered byplugging in novel lighting coefficients lnew into (9). For colorimages, the surface albedo and lighting coefficients of each ofthe R, G and B channels are estimated separately.

4958 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 26, NO. 10, OCTOBER 2017

Fig. 3. Normal estimation using simulated noisy depth for 6 selected scenesin the MIT-Berkeley data set. Mean angular errors are indicated below thefigures.

VII. EXPERIMENT

To verify the correctness of the proposed method,we present our experimental results on both simulated andreal data. First, we test our proposed normal refinement usingreal images with ground truth depth that is added with Kinect-like noise. Next, we present experiment results and evaluationof our proposed albedo recovery and relighting using standardphotometric stereo dataset. We also test our method on realdata captured using our proposed set-up. Since our modelingof image formation involves the second order approximationof spherical harmonics, certain error may persist even in anideal case, it is desirable to verify some basic properties ofour method.

A. Test With Simulated Noise

1) Normal Refinement: To verify the proposed normalrefinement, we make use of the MIT-Berkeley data set,2

where ground-truth depth is provided. The dataset contains20 real scenes, each captured under 10 different, arbitraryillumination. We use its image data as color image input andsimulated noisy depth maps based on [32]. Distance-dependentGaussian noise is added to simulate the noisy data producedby a Kinect sensor, which is computed as follows:

Zn = Z0 + g(Z0) (30)

The depth-dependent noise follows a Gaussian distribution,g(Z0) ∼ N (0, cZ2

0), where Z0 denotes the noiseless depthvalue and c = 1.43 × 10−5 is a Kinect-oriented constant [32].The input surface normal is computed from the noisy depthusing (15). Instead of using all pixels in the scene, we firstavoid pixels on the boundary of the object, as normal estima-tion in these places are noisy due to incomplete data in the

2http://www.cs.berkeley.edu/~barron/SIRFS_release1.5.zip

TABLE III

AVERAGE SIGNAL-TO-NOISE RATIO (SNR) IN d B FOR ALBEDORECOVERY UNDER DIFFERENT CONDITIONS

computation of partial derivative. We select pixels by lookingat their median intensity values over all illumination condi-tions. Typically, we select pixels whose median intensities fallin the range of [0.2, 0.8]. This selection process reduces thenumber of samples and helps speed up computation. Further-more, we apply bilateral filtering on the noisy normals, �n′

i ’s,prior to normal refinement. We compare our method againsttwo depth (normal) refinement algorithms, namely, the methodby Or-El et al. [20] and that by Barron and Malik [23].Results are shown in Fig. 3. Mean angular errors (computedusing ground truth surface normals) are shown below thefigures. The errors of both [20] and [23] are shown to putour performance into perspective. It would be unfair to simplycompare the angular errors, as these two methods used onlyone RGB image as input. Qualitatively, we can observe thatthe proposed normal refinement algorithm compares favorablyto the two reference methods. The results of [20] still containa small amount of noise, while [23] tends to overly smooththe surface normals. The proposed method, especially its non-linear refinement step, greatly improves the quality of surfacenormals with much reduced angular errors and less texturecopying artifacts.

2) Albedo Recovery: In this experiment, we validate ourfactorization for reflectance recovery (Section VII-A.2) andstudy the effects due to normal refinement. We assume noisyinput normals (simulated by adding noise to the depth maps asin (30)). For this purpose, we use the MIT-Berkeley data setcaptured under general illumination conditions. We recoveralbedo for each of the R, G and B channels based on themethod described in Section . We use the provided groundtruth albedo maps for quantitative evaluation, and computethe average signal-to-noise ratio (SNR) of albedo recoveryfor each channel, across all 20 scenes. Results are presentedin Table III. We also verify the albedo recovery when cleannormals are used. As we are testing albedo recovery as a stand-alone module, we initialize � in (26) to identity.

Fig. 4 shows a qualitative comparison of reflectance recov-ery on the “teabag2” scene. Pixel-wise absolute errors fromthe ground truth are shown. Observe the impact of normalrefinement on the final recovered reflectance map. Withoutnormal refinement, we can still see residual shading on thereflectance image; while the result after normal refinement isvisually close to the ground truth.

3) Relighting: Upon recovery of both albedo � and light-ing L, novel lighting effects can be rendered by pluggingin new lighting coefficients in (9). We assume white light

LIU AND DO: INVERSE RENDERING AND RELIGHTING FROM MULTIPLE COLOR PLUS DEPTH IMAGES 4959

Fig. 4. (a) Visualization of absolute error in reflectance recovery with andwithout normal refinement, for each of the R, G, B channels. (b) Comparisonof recovered reflectance with and without normal refinement.

Fig. 5. Three newly relit images of the “Raccoon” scene with randomly gen-erated lighting coefficients. Refer to the fifth row of Fig. 3 for correspondingnormal refinement result.

illumination (equal illumination intensity for each channel).As we recover the albedos channel-wise for each of theRGB channels and the recovery is up to a scaling ambiguity,we computed the relative channel contribution from the meanintensity of each channel, over all illumination conditions.Finally, we rescale the recovered albedos for each channelaccording to the estimated ratio to generate colored relitresults. Fig. 5 shows three newly relit images of the “Raccoon”scene with randomly generated lighting coefficients.

B. Test on Real Data

1) Experiment Setup: We used Kinect sensor for depth dataacquisition. Since it is difficult to radiometrically calibrate itsbuilt-in RGB camera, we mounted a PointGrey ChameleonCMLN-1632C camera directly above its built-in RGB cameraand calibrated the cameras intrinsically and extrinsically. ThisPointGrey camera is set to capture linear color images. Thesetup for our experiment is shown in Fig. 6.

The depth map from Kinect camera has a resolutionof 480 × 640, while the PointGrey RGB camera has a higherresolution of 960×1280. To align the depth data to an externalRGB image, we performed intrinsic calibration of both theIR camera (responsible for depth sensing) and the PointGreycamera using Zhang’s method [33] with a checkerboard target.

Fig. 6. The setup for color and depth images acquisition. An external RGBcamera is mounted on top of a Kinect sensor.

Fig. 7. Images of a scene under 12 different illumination conditions.

The relative pose of these two cameras are also calibratedusing known correspondences from the checkerboard. Fig. 8shows an example of depth to RGB mapping. To validate thecorrectness of the calibration, we shaded the depth map withits corresponding RGB values from the external camera. Theresultant image is visually consistent with the scene.

2) Relighting From RGB-D Data: At this stage, we capturedimages of the same scene by varying the light sources. TheRGB images are captured in an indoor environment, whoselight sources consist of both point source and area lights. Ourscene is illuminated under 12 different lighting conditions.Examples of the lighting conditions are shown in Fig. 7.

When the scene contains purely Lambertian reflection, ourimage formation model predicts that the data matrix M is ofrank 9. However, real world data are often affected by noise,specularities and other non-Lambertian effects, the rank willbe typically larger than 9. We perform a pre-processing stepto recover a low-rank intensity matrix [34]. This is basedon a recent convex optimization approach [35] that recoversthe low-rank intensity matrix without having any informationabout what entries are corrupted. This is used ensure a robustsolution even in the presence of minor specularities andshadow.

We applied the proposed RGB and depth normal refinementas described in Section V to estimate surface normals of areal scene. We used a manually drawn mask to segment outthe object of interest. We performed bilateral filtering on the

4960 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 26, NO. 10, OCTOBER 2017

Fig. 8. (a) Depth map; (b) Depth map aligned to the external RGB image;(c) Depth map shaded with RGB values from the external camera, showingthe correctness of the calibration.

Fig. 9. Comparison on normal refinement. (a) Noisy normals from depthmap. (b) Normal refinement and reflectance recovery by the proposed method.(c) Normal refinement and reflectance recovery by Barron and Malik [23].(d) Normal refinement by Or-El et al. [20]. Run time for each algorithm isindicated below the results.

noisy depth map and up-sampled it to match the resolutionof the RGB image. Results of the surface normals from thenoisy depth map, the resultant normals after refinement andrecovered reflectance are shown in Fig. 9. We compare ourmethod against Or-El et al. [20] and Barron and Malik [23].Run time for each algorithm is indicated below the results.All algorithms are run on a 6-core i7 machine without GPUacceleration.3 While strictly not an apples-to-apples com-parison, our un-optimized MATLAB implementation has acomparable run time (combing time taken for both normalrefinement and reflectance recovery) to that of [20], whoseGPU implementation version was reported to have real-timeperformance, indicating the potential real-time feasibility ofour method. On visual comparison, our proposed method

3We used the source code from https://cs.technion.ac.il/royorel/projectpage/rgbd_fusion.zip, a MATLAB demo version that outputsonly a refined depth map.

Fig. 10. Relighting results using randomly generated lighting parameters.

has less texture copying artifacts as compared to [20] (seemagnified insets of the coffee box and cookie bag regions).Our method’s output normals are of higher quality suitable foruse in the estimation of scene albedo. In addition, we note thatour normal refinement only involves the first-order sphericalharmonics, while the other methods used high-order harmon-ics.

Lastly, Fig. 10 shows relighting results of the same scenewith randomly generated lighting parameters. These resultslook photo-realistic.

VIII. CONCLUSION AND FUTURE WORK

In this work, we explore the incorporation of raw3D depth from Kinect into a computational relighting problem.Assuming distant illumination and Lambertian reflectance,we model the reflected light field in terms of spherical har-monics. We first use the noisy depth information to resolvethe linear ambiguity in photometric stereo, leading to moreaccurate surface normals that can be used in the later estima-tion of albedo and lighting coefficients (up to a global scalingambiguity). As a result, we propose an effective method torecover object geometry by combining noisy acquired depthwith shape-from-shading. We solve for surface normal andalbedo in a progressive manner: using the first-order sphericalharmonics to refine noisy normal, followed by the second-order spherical harmonics for albedo recovery. We formulatethe normal refinement problem as a nonlinear optimization,and further show that it can be significantly sped up wheninitialized by a linear optimization, leading to much reducedcomputation time. We demonstrate our method on both simu-lated and real data, and show that it can successfully recoverboth illumination and albedo to produce realistic relightingresults. Looking forward, we aim to jointly optimize depth,lighting and reflectance for better relighting results. We alsohope to extend the current method to handle more complexreflection models beyond Lambertian reflectance.

REFERENCES

[1] P. Debevec, “Rendering synthetic objects into real scenes: Bridgingtraditional and image-based graphics with global illumination and highdynamic range photography,” in Proc. ACM SIGGRAPH, Aug. 1998,Art. no. 32.

[2] P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, andM. Sagar, “Acquiring the reflectance field of a human face,” in Proc.ACM SIGGRAPH, 2000, pp. 145–156.

[3] C. Richardt, C. Stoll, N. A. Dodgson, H.-P. Seidel, and C. Theobalt,“Coherent spatiotemporal filtering, upsampling and rendering of RGBZvideos,” Comput. Graph. Forum, vol. 31, no. 2, pp. 247–256, May 2012.

LIU AND DO: INVERSE RENDERING AND RELIGHTING FROM MULTIPLE COLOR PLUS DEPTH IMAGES 4961

[4] S. Izadi et al., “KinectFusion: Real-time 3D reconstruction and inter-action using a moving depth camera,” in Proc. 24th Annu. ACM Symp.User Interface Softw. Technol., Oct. 2011, pp. 559–568.

[5] R. A. Newcombe, D. Fox, and S. M. Seitz, “DynamicFusion: Recon-struction and tracking of non-rigid scenes in real-time,” in Proc. Comput.Vis. Pattern Recognit., Jun. 2015, pp. 343–352.

[6] R. Ramamoorthi and P. Hanrahan, “A signal-processing frameworkfor reflection,” ACM Trans. Graph., vol. 23, no. 4, pp. 1004–1042,Oct. 2004.

[7] R. Basri and D. W. Jacobs, “Lambertian reflectance and linear sub-spaces,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 25, pp. 218–233,Feb. 2003.

[8] S. Liu and M. N. Do, “Relighting from multiple color and depthimages using matrix factorization,” in Proc. Int. Conf. Image Process.,Oct. 2014, pp. 4627–4631.

[9] J. T. Kajiya, “The rendering equation,” SIGGRAPH Comput. Graph.,vol. 20. no. 4, pp. 143–150, Aug. 1986.

[10] N. Jeffry, S. Eero, and D. Julie, “Efficient re-rendering of naturallyilluminated environments,” in Proc. Eurograph. Workshop Rendering,1994, pp. 359–373.

[11] Z. Wen, Z. Liu, and T. S. Huang, “Face relighting with radianceenvironment maps,” in Proc. Comput. Vis. Pattern Recognit., Jun. 2003,pp. 158–165.

[12] T. Yu, H. Wang, N. Ahuja, and W.-C. Chen, “Sparse lumigraph relightingby illumination and reflectance estimation from multi-view images,” inProc. ACM SIGGRAPH Sketches, 2006, Art. no. 175.

[13] N. Joshi and D. J. Kriegman, “Shape from varying illumination andviewpoint,” in Proc. Int. Conf. Comput. Vis., Oct. 2007, pp. 1–7.

[14] H. Q. Nguyen, S. Liu, and M. N. Do, “Subspace methods for compu-tational relighting,” Proc. SPIE, vol. 8657, p. 865703, Feb. 2013.

[15] G. Oxholm and K. Nishino, “Shape and reflectance from naturalillumination,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 528–541.

[16] L.-F. Yu, S.-K. Yeung, Y.-W. Tai, and S. Lin, “Shading-based shaperefinement of RGB-D images,” in Proc. Comput. Vis. Pattern Recognit.,Jun. 2013, pp. 1415–1422.

[17] Y. Han, J.-Y. Lee, and I. S. Kweon, “High quality shape from a singleRGB-D image under uncalibrated natural illumination,” in Proc. IEEEInt. Conf. Comput. Vis. (ICCV), Dec. 2013, pp. 1617–1624.

[18] D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, “Efficientlycombining positions and normals for precise 3D geometry,” ACM Trans.Graph., vol. 24, no. 3, pp. 536–543, 2005.

[19] C. Wu, M. Zollhöfer, M. Nießner, M. Stamminger, S. Izadi, andC. Theobalt, “Real-time shading-based refinement for consumer depthcameras,” ACM Trans. Graph., vol. 33, no. 6, 2014, Art. no. 200.

[20] R. Or-El, G. Rosman, A. Wetzler, R. Kimmel, and A. M. Bruckstein,“RGBD-fusion: Real-time high precision depth recovery,” in Proc.Comput. Vis. Pattern Recognit., Jun. 2015, pp. 5407–5416.

[21] E. H. Land and J. J. McCann, “Lightness and Retinex theory,” J. Opt.Soc. Amer., vol. 61, no. 1, pp. 1–11, 1971.

[22] M. Hudon, A. Gruson, P. Kerbiriou, R. Cozot, and K. Bouatouch, “Shapeand reflectance from RGB-D images using time sequential illumination,”in Proc. Int. Joint Conf. Comput. Vis., Imag. Comput. Graph. TheoryAppl. (VISIGRAPP), vol. 64. 2016, pp. 65–66.

[23] J. T. Barron and J. Malik, “Intrinsic scene properties from a singleRGB-D image,” in Proc. Comput. Vis. Pattern Recognit., Jun. 2013,pp. 17–24.

[24] Q. Zhang, M. Ye, R. Yang, Y. Matsushita, B. Wilburn, and H. Yu, “Edge-preserving photometric stereo via depth fusion,” in Proc. Comput. Vis.Pattern Recognit., Jun. 2012, pp. 2472–2479.

[25] S. M. Haque, A. Chatterjee, and V. M. Govindu, “High quality pho-tometric reconstruction using a depth camera,” in Proc. Comput. Vis.Pattern Recognit., Jun. 2014, pp. 2283–2290.

[26] R. Ramamoorthi, Modeling Illumination Variation With SphericalHarmonics, in Face Processing: Advanced Modeling and Methods.San Diego, CA, USA: Academic, 2005.

[27] R. J. Woodham, “Photometric stereo: A reflectance map techniquefor determining surface orientation from image intensity,” Proc. SPIE,vol. 0155, pp. 136–143, Jan. 1979.

[28] R. Ramamoorthi, “Analytic PCA construction for theoretical analysisof lighting variability in images of a Lambertian object,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 24, no. 10, pp. 1322–1333, Oct. 2002.

[29] R. Basri, D. W. Jacobs, and I. Kemelmacher, “Photometric stereowith general, unknown lighting,” Int. J. Comput. Vis., vol. 72, no. 3,pp. 239–257, May 2007.

[30] A. L. Yuille, D. Snow, R. Epstein, and P. N. Belhumeur, “Determininggenerative models of objects under varying illumination: Shape andalbedo from multiple images using SVD and integrability,” Int. J.Comput. Vis., vol. 35, no. 3, pp. 203–222, Dec. 1999.

[31] K. Kanatani, Geometric Computation for Machine Vision. London, U.K.:Oxford Univ. Press, 1993.

[32] K. Khoshelham and S. O. Elberink, “Accuracy and resolution of Kinectdepth data for indoor mapping applications,” Sensors, vol. 12, no. 2,pp. 1437–1454, Feb. 2012.

[33] Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 22, no. 11, pp. 1330–1334, Nov. 2000.

[34] L. Wu, A. Ganesh, B. Shi, Y. Matsushita, Y. Wang, and Y. Ma, “Robustphotometric stereo via low-rank matrix completion and recovery,” inProc. Asian Conf. Comput. Vis., 2011, pp. 703–717.

[35] Z. Lin, M. Chen, and Y. Ma. (2010). “The augmented lagrange multipliermethod for exact recovery of corrupted low-rank matrices.” [Online].Available: https://arxiv.org/abs/1009.5055

Siying Liu received the B. Eng. and M. Eng. degreesin electrical engineering from the National Univer-sity of Singapore, in 2006 and 2009, respectively, thePh.D. degree in electrical and computer engineeringfrom the University of Illinois at Urbana–Champaignin 2016, supervised by Prof. Minh N. Do. Sheis currently a Research Scientist with the Institutefor Infocomm Research, Singapore. Her researchinterest includes computational photography and3D computer vision.

Minh N. Do (M’01–SM’07–F’14) was born inVietnam in 1974. He received the B.Eng. degreein computer engineering from the University ofCanberra, Australia, in 1997, and the Dr.Sci. degreein communication systems from the Swiss Fed-eral Institute of Technology Lausanne (EPFL),Switzerland, in 2001. Since 2002, he has beenwith the Faculty, University of Illinois at Urbana–Champaign (UIUC), where he is currently a Profes-sor with the Department of Electrical and ComputerEngineering, and holds joint appointments with the

Coordinated Science Laboratory, the Beckman Institute for Advanced Scienceand Technology, the Advanced Digital Sciences Center, the Department ofBioengineering, and the Department of Computer Science. His researchinterests include signal processing, computational imaging, geometric vision,and data science. He was a member of the IEEE Signal Processing Theoryand Methods Technical Committee, Image, Video, and MultidimensionalSignal Processing Technical Committee, and an Associate Editor of the IEEETransactions on Image Processing. He received a Silver Medal from the 32ndInternational Mathematical Olympiad in 1991, a University Medal from theUniversity of Canberra in 1997, a Doctorate Award from the EPFL in 2001, aCAREER Award from the National Science Foundation in 2003, and a YoungAuthor Best Paper Award from IEEE in 2008. He was named a BeckmanFellow at the Center for Advanced Study, UIUC, in 2006, and received of aXerox Award for Faculty Research from the College of Engineering, UIUC,in 2007. He is a Fellow of the IEEE for contributions to image representationand computational imaging. He was a Co-Founder and CTO of Personify Inc.,a spin-off from UIUC to commercialize depth-based visual communication.