DIST: Rendering Deep Implicit Signed Distance Functionwith Differentiable Sphere Tracing

Shaohui Liu1,3 ∗ Yinda Zhang2 Songyou Peng1,6 Boxin Shi4,7

Marc Pollefeys1,5,6 Zhaopeng Cui1†1ETH Zurich 2Google 3Tsinghua University 4Peking University 5Microsoft

6Max Planck ETH Center for Learing Systems 7Peng Cheng Laboratory

Abstract

We propose a differentiable sphere tracing algorithm tobridge the gap between inverse graphics methods and therecently proposed deep learning based implicit signed dis-tance function. Due to the nature of the implicit function,the rendering process requires tremendous function queries,which is particularly problematic when the function is rep-resented as a neural network. We optimize both the forwardand backward passes of our rendering layer to make it runefficiently with affordable memory consumption on a com-modity graphics card. Our rendering method is fully differ-entiable such that losses can be directly computed on therendered 2D observations, and the gradients can be propa-gated backwards to optimize the 3D geometry. We show thatour rendering method can effectively reconstruct accurate3D shapes from various inputs, such as sparse depth andmulti-view images, through inverse optimization. With thegeometry based reasoning, our 3D shape prediction meth-ods show excellent generalization capability and robustnessagainst various noises.

1. IntroductionSolving vision problem as an inverse graphics process is

one of the most fundamental approaches, where the solutionis the visual structure that best explains the given observa-tions. In the realm of 3D geometry understanding, this ap-proach has been used since the very early age [1, 36, 55]. Asa critical component to the inverse graphics based 3D geo-metric reasoning process, an efficient renderer is requiredto accurately simulate the observations, e.g., depth maps,from an optimizable 3D structure, and also be differentiableto back-propagate the error from the partial observation.

As a natural fit to the deep learning framework, differ-entiable rendering techniques have drawn great interests re-

∗Work done while Shaohui Liu was an academic guest at ETH Zurich.†Corresponding author.

De

co

de

r

Dif

fere

nti

able

Ren

der

er

Silhouette

Normal

Depth

Optimization

SDF

Loss

𝒛𝟎

𝒛

Figure 1. Illustration of our proposed differentiable renderer forcontinuous signed distance function. Our method enables geomet-ric reasoning with strong generalization capability. With a ran-dom shape code z0 initialized in the learned shape space, we canacquire high-quality 3D shape prediction by performing iterativeoptimization with various 2D supervisions.

cently. Various solutions for different 3D representations,e.g., voxels, point clouds, meshes, have been proposed.However, these 3D representations are all discretized up toa certain resolution, leading to the loss of geometric detailsand breaking the differentiable properties [24]. Recently,the continuous implicit function has been used to representthe signed distance field [35], which has premium capac-ity to encode accurate geometry when combined with thedeep learning techniques. Given a latent code as the shaperepresentation, the function can produce a signed distancevalue for any arbitrary point, and thus enable unlimited res-olution and better preserved geometric details for render-ing purpose. However, a differentiable rendering solutionfor learning-based continuous signed distance function doesnot exist yet.

In this paper, we propose a differentiable renderer forcontinuous implicit signed distance functions (SDF) to fa-cilitate the 3D shape understanding via geometric reason-ing in a deep learning framework (Fig. 1). Our methodcan render an implicit SDF represented by a neural networkfrom a latent code into various 2D observations, e.g., depthimages, surface normals, silhouettes, and other propertiesencoded, from arbitrary camera viewpoints. The render-ing process is fully differentiable, such that loss functions

arX

iv:1

911.

1322

5v2

[cs

.CV

] 1

1 Ju

n 20

20

can be conveniently defined on the rendered images and theobservations, and the gradients can be propagated back tothe shape generator. As major applications, our differen-tiable renderer can be applied to infer the 3D shape fromvarious inputs, e.g., multi-view images and single depthimage, through an inverse graphics process. Specifically,given a pre-trained generative model, e.g., DeepSDF [35],we search within the latent code space for the 3D shapethat produces the rendered images mostly consistent withthe observation. Extensive experiments show that our geo-metric reasoning based approach exhibits significantly bet-ter generalization capability than previous purely learningbased approaches, and consistently produce accurate 3Dshapes across datasets without finetuning.

Nevertheless, it is challenging to make differentiable ren-dering work on a learning-based implicit SDF with compu-tationally affordable resources. The main obstacle is that animplicit function provides neither the exact location nor anybound of the surface geometry as in other representationslike meshes, voxels, and point clouds.

Inspired by traditional ray-tracing based approaches, weadopt the sphere tracing algorithm [14], which marchesalong each pixel’s ray direction with the queried signed dis-tance until the ray hits the surface, i.e., the signed distanceequals to zero (Fig. 2). However, this is not feasible in theneural network based scenario where each query on the raywould require a forward pass and recursive computationalgraph for back-propagation, which is prohibitive in termsof computation and memory.

To make it work efficiently on a commodity level GPU,we optimize the full life-time of the rendering process forboth forward and backward propagations. In the forwardpass, i.e., the rendering process, we adopt a coarse-to-fineapproach to save computation at initial steps, an aggressivestrategy to speed up the ray marching, and a safe conver-gence criteria to prevent unnecessary queries and maintainresolution. In the backward propagation, we propose a gra-dient approximation which empirically has negligible im-pact on the training performance but dramatically reducesthe computation and memory consumption. By making therendering tractable, we show how producing 2D observa-tions with the sphere tracing and interacting with cameraextrinsics can be done in differentiable ways.

To sum up, our major contribution is to enable ef-ficient differentiable rendering on the implicit signeddistance function represented as a neural network. Itenables accurate 3D shape prediction via geometric rea-soning in deep learning frameworks and exhibits promis-ing generalization capability. The differentiable ren-derer could also potentially benefit various vision prob-lems thanks to the marriage of implicit SDF and inversegraphics techniques. The code and data are available athttps://github.com/B1ueber2y/DIST-Renderer.

2. Related Work3D Representation for Shape Learning. The study of 3Drepresentations for shape learning is one of the main fo-cuses in 3D deep learning community. Early work quan-tizes shapes into 3D voxels, where each voxel contains ei-ther a binary occupancy status (occupied / not occupied)[53, 6, 47, 40, 13] or a signed distance value [56, 9, 46].While voxels are the most straightforward extension fromthe 2D image domain into the 3D geometry domain for neu-ral network operations, they normally require huge memoryoverhead and result in relatively low resolutions. Meshesare also proposed as a more memory efficient representationfor 3D shape learning [48, 12, 23, 21], but the topology ofmeshes is normally fixed and simple. Many deep learningmethods also utilize point clouds as the 3D representation[38, 39]; however, the point-based representation lacks thetopology information and thus makes it non-trivial to gen-erate 3D meshes. Very recently, the implicit functions, e.g.,continuous SDF and occupancy function, are exploited as3D representations and show much promising performancein terms of the high-frequency detail modeling and the highresolution [35, 30, 31, 4]. Similar idea has also been used toencode other information such as textures [34, 41] and 4Ddynamics [33]. Our work aims to design an efficient anddifferentiable renderer for the implicit SDF-based represen-tation.Differentiable Rendering. With the success of deep learn-ing, the differentiable rendering starts to draw more at-tention as it is essential for end-to-end training, and solu-tions have been proposed for various 3D representations.Early works focus on 3D triangulated meshes and lever-age standard rasterization [29]. Various approaches try tosolve the discontinuity issue near triangle boundaries bysmoothing the loss function or approximating the gradient[22, 37, 26, 3]. Solutions for point clouds and 3D voxels arealso introduced [49, 18, 32] to work jointly with PointNet[38] and 3D convolutional architectures. However, the dif-ferentiable rendering for the implicit continuous functionrepresentation does not exist yet. Some ray tracing basedapproaches are related, while they are mostly proposed forexplicit representation, such as 3D voxels [28, 32, 44, 19]or meshes [24], but not implicit functions. Liu et al. [27]firstly propose to learn from 2D observations over occu-pancy networks [30]. However, their methods make severalapproximations and do not benefit from the efficiency ofrendering implicit SDF. Most related to our work, Sitzmannet al. [45] propose a LSTM-based renderer for an implicitscene representation to generate color images, while theirmodel focuses on simulating the rendering process with anLSTM without clear geometric meaning. This method canonly generate low-resolution images due to the expensivememory consumption. Alternatively, our method can di-rectly render 3D geometry represented by an implicit SDF

StartEndp(0) p(1) p(2) p(3) p(4)

f (p(0))

Surface

Figure 2. Illustration on the sphere tracing algorithm [14]. A rayis initiated at each pixel and marching along the viewing direction.The front end moves with a step size equals to the signed distancevalue of the current location. The algorithm converges when thecurrent absolute SDF is smaller than a threshold, which indicatesthat the surface has been found.

to produce high-resolution images. It can also be appliedwithout training to existing deep learning models.3D Shape Prediction. 3D shape prediction from 2D obser-vations is one of the fundamental vision problems. Earlyworks mainly focus on multi-view reconstruction usingmulti-view stereo methods [42, 15, 43]. These purelygeometry-based methods suffer from degraded performanceon texture-less regions without prior knowledge [7]. Withprogress of deep learning, 3D shapes can be recovered un-der different settings. The simplest setting is to recover 3Dshape from a single image [6, 11, 52, 20]. These systemsrely heavily on priors, and are prone to weak generalization.Deep learning based multi-view shape prediction methods[54, 16, 17, 50, 51] further involve geometric constraintsacross views in the deep learning framework, which showsbetter generalization. Another thread of works [9, 8] take asingle depth image as input, and the problem is usually re-ferred as shape completion. Given the shape prior encodedin the neural network [35], our rendering method can ef-fectively predict accurate 3D object shape from a randominitial shape code with various inputs, such as depth andmulti-view images, through geometric optimization.

3. Differentiable Sphere Tracing

In this section, we introduce our differentiable renderingmethod for the implicit signed distance function representedas a neural network, such as DeepSDF [35]. In DeepSDF, anetwork takes a latent code and a 3D location as input, andproduces the corresponding signed distance value. Eventhough such a network can deliver high quality geometry,the explicit surface cannot be directly obtained and requiresdense sampling in the 3D space.

Our method is inspired by Sphere Tracing [14] designedfor rendering SDF volumes, where rays are shot from thecamera pinhole along the direction of each pixel to searchfor the surface level set according to the signed distancevalue. However, it is prohibitive to apply this method di-rectly on the implicit signed distance function represented

Algorithm 1 Naive sphere tracing algorithm for a cameraray L : c + dv over a signed distance fields f : N3 → R.

1: Initialize n = 0, d(0) = 0, p(0) = c.2: while not converged do:3: Take the corresponding SDF value b(n) = f(p(n))

of the location p(n) and make update: d(n+1) = d(n) +b(n).

4: p(n+1) = c + d(n+1)v, n = n+ 1.5: Check convergence.6: end while

as a neural network, since each tracing step needs a feed-forward neural network and the whole algorithm requiresunaffordable computational and memory resources. Tomake this idea work in deep learning framework for in-verse graphics, we optimize both the forward and backwardpropagations for efficient training and test-time optimiza-tion. The sphere traced results, i.e., the distance along theray, can be converted into many desired outputs, e.g., depth,surface normal, silhouette, and hence losses can be conve-niently applied in an end-to-end manner.

3.1. Preliminaries - Sphere Tracing

To be self-contained, we first briefly introduce the tra-ditional sphere tracing algorithm [14]. Sphere tracing is aconventional method specifically designed to render depthfrom volumetric signed distance fields. For each pixel onthe image plane, as shown in Figure 2, a ray (L) is shot fromthe camera center (c) and marches along the direction (v)with a step size that is equal to the queried signed distancevalue (b). The ray marches iteratively until it hits or getssufficiently close to the surface (i.e. abs(SDF) < threshold).A more detailed algorithm can be found in Algorithm 1.

3.2. Efficient Forward Propagation

Directly applying sphere tracing to an implicit SDF func-tion represented by a neural network is prohibitively com-putational expensive, because each query of f requires aforward pass of a neural network with considerable capac-ity. Naive parallelization is not sufficient since essentiallymillions of network queries are required for a single render-ing with VGA resolution (640 × 480). Therefore, we needto cut off unnecessary marching steps and safely speed upthe marching process.Initialization. Because all the 3D shapes represented byDeepSDF are bounded within the unit sphere, we initializep(0) to be the intersection between the camera ray and theunit sphere for each pixel. Pixels with the camera rays thatdo not intersect with the unit sphere are set as background(i.e., infinite depth).Coarse-to-fine Strategy. At the beginning of sphere trac-ing, rays for different pixels are fairly close to each other,which indicates that they will likely march in a similar way.

(a) Coarse-to-fine Strategy (b) Aggressive Marching (c) Convergence CriteriaFigure 3. Strategies for our efficient forward propagation. (a) 1D illustration of our coarse-to-fine strategy, and for 2D cases, one ray willbe spitted into 4 rays; (b) Comparison of standard marching and our aggressive marching; (c) We stop the marching once the SDF value issmaller than ε, where 2ε is the estimated minimal distance between the corresponding 3D points of two neighboring pixels.

To leverage this nice property, we propose a coarse-to-finesphere tracing strategy as shown in Fig. 3 (a). We start thesphere tracing from an image with 1

4 of its original resolu-tion, and split each ray into four after every three marchingsteps, which is equivalent to doubling the resolution. Aftersix steps, each pixel in the full resolution has a correspond-ing ray, which keeps marching until convergence.Aggressive Marching. After the ray marching begins, weapply an aggressive strategy (Fig. 3 (b)) to speed up themarching process by updating the ray with α times of thequeried signed distance value, where α = 1.5 in our imple-mentation. This aggressive sampling has several benefits.First, it makes the ray march faster towards the surface, es-pecially when it is far from surface. Second, it acceleratesthe convergence for the ill-posed condition, where the anglebetween the surface normal and the ray direction is small.Third, the ray can pass through the surface such that spacein the back (i.e., SDF < 0) could be sampled. This is cru-cially important to apply supervision on both sides of thesurface during optimization.Dynamic Synchronized Inference. A naive parallelizationfor speeding up sphere tracing is to batch rays together andsynchronously update the front end positions. However, de-pending on the 3D shape, some rays may converge earlierthan others, thus leading to wasted computation. We main-tain a dynamic unfinished mask indicating which rays re-quire further marching to prevent unnecessary computation.Convergence Criteria. Even with aggressive marching, theray movement can be extremely slow when close to the sur-face since f is close to zero. We define a convergence cri-teria to stop the marching when the accuracy is sufficientlygood and the gain is marginal (Fig. 3(c)). To fully main-tain details supported by the 2D rendering resolution, it issufficiently safe to stop when the sampled signed distancevalue does not confuse one pixel with its neighbors. Foran object with a smallest distance of 100mm captured by acamera with 60mm focal length, 32mm sensor width, and aresolution of 512× 512, the approximate minimal distancebetween the corresponding 3D points of two neighboringpixels is 10−4m (0.1mm). In practice, we set the conver-

gence threshold ε as 5× 10−5 for most of our experiments.

3.3. Rendering 2D Observations

After all rays converge, we can compute the distancealong each ray as the following:

d = α

N−1∑n=0

f(p(n)) + (1− α)f(p(N−1)) = d′ + e, (1)

where e = (1 − α)f(p(N−1)) is the residual term on thelast query. In the following part we will show how this com-puted ray distance is converted into 2D observations.Depth and Surface Normal. Suppose that we find the 3Dsurface point p = c+dv for a pixel (x, y) in the image, wecan directly get the depth for each pixel as the following:

zc =d√

x2 + y2 + 1, (2)

where (x, y, 1)> = K−1(x, y, 1)> is the normalized homo-geneous coordinate.

The surface normal of the point p(x, y, z) can be com-puted as the normalized gradient of the function f . Since fis an implicit signed distance function, we take the approx-imation of the gradient by sampling neighboring locations:

n =1

2δ

f(x+ δ, y, z)− f(x− δ, y, z)f(x, y + δ, z)− f(x, y − δ, z)f(x, y, z + δ)− f(x, y, z − δ)

, n =n

|n|. (3)

Silhouette. The silhouette is a commonly used supervisionfor 3D shape prediction. To make the rendering of silhou-ettes differentiable, we get the minimum absolute signeddistance value for each pixel along its ray and subtract it bythe convergence threshold ε. This produces a tight approx-imation of the silhouette, where pixels with positive valuesbelong to the background, and vice versa. Note that directlychecking if ray marching stops at infinity can also generatethe silhouette but it is not differentiable.Color and Semantics. Recently, it has been shown that tex-ture can also be represented as an implicit function param-eterized with a neural network [34]. Not only color, other

Method size #step #query timeNaive sphere tracing 5122 50 N/A N/A

+ practical grad. 5122 50 6.06M 1.6h+ parallel 5122 50 6.06M 3.39s+ dynamic 5122 50 1.99M 1.23s

+ aggressive 5122 50 1.43M 1.08s+ coarse-to-fine 5122 50 887K 0.99s+ coarse-to-fine 5122 100 898K 1.24s

Table 1. Ablation studies on the cost-efficient feedforward de-sign of our method. The average time for each optimization stepwas tested on a single NVIDIA GTX-1080Ti over the architec-ture of DeepSDF [35]. Note that the number of initialized rays isquadratic to the image size, and the numbers are reported for theresolution of 512× 512.

spatially varying properties, like semantics, material, etc,can all be potentially learned by implicit functions. Theseinformation can be rendered jointly with the implicit SDFto produce corresponding 2D observations, and some ex-amples are depicted in Fig. 8.

3.4. Approximated Gradient Back-Propagation

DeepSDF [35] uses the conditional implicit function torepresent a 3D shape as fθ(p, z), where θ is the networkparameters, and z is the latent code representing a certainshape. As a result, each queried point p in the sphere tracingprocess is determined by θ and the shape code z, whichrequires to unroll the network for multiple times and costshuge memory for back-propagation with respect to z:

∂d′

∂z|z0 = α

N−1∑i=0

∂fθ(p(i)(z), z)

∂z|z0

= α

N−1∑i=0

(∂fθ(p

(i)(z0), z)

∂z+∂fθ(p

(i)(z), z0)

∂p(i)(z)

∂p(i)(z0)

∂z).

(4)Practically, we ignore the gradients from the residual

term e in Equation (1). In order to make back-propagationfeasible, we define a loss for K samples with the minimumabsolute SDF value on the ray to encourage more signalsnear the surface. For each sample, we calculate the gradi-ent with only the first term in Equation (4) as the high-ordergradients empirically have less impact on the optimizationprocess. In this way, our differentiable renderer is particu-larly useful to bridge the gap between this strong shape priorand some partial observations. Given a certain observation,we can search for the code that minimizes the differencebetween the rendering from our network and the observa-tion. This allows a number of applications which will beintroduced in the next section.

4. Experiments and Results

In this section, we first verify the efficacy of our dif-ferentiable sphere tracing algorithm, and then show that

parallel + dynamic + aggressive + coarse-to-fine

Figure 4. The surface normal rendered with different speed upstrategies turned on. Note that adding up these components doesnot deteriorate the rendering quality.

Figure 5. Loss curves for 3D prediction from partial depth. Ouraccelerated rendering does not impair the back-propagation. Theloss on the depth image is tightly correlated with the Chamfer dis-tance on 3D shapes, which indicates effective back-propagation.

3D shape understanding can be achieved through geometrybased reasoning by our method.

4.1. Rendering Efficiency and Quality

Run-time Efficiency. In this section, we evaluate the run-time efficiency promoted by each design in our differen-tiable sphere tracing algorithm. The number of queries andruntime for both forward and backward passes at a resolu-tion of 512 × 512 on a single NVIDIA GTX-1080Ti arereported in Tab. 1, and the corresponding rendered surfacenormal are shown in Fig. 4. We can see that the proposedback-propagation prunes the graph and reduces the mem-ory usage significantly, making the rendering tractable witha standard graphics card. The dynamic synchronized in-ference, aggressive marching and coarse-to-fine strategy allspeed up rendering. With all these designs, we can renderan image with only 887K query steps within 0.99s whenthe maximum tracing step is set to 50. The number of querysteps only increases slightly when the maximum step is setto 100, indicating that most of the pixels converge safelywithin 50 steps. Note that related works usually render at amuch lower resolution [45].Back-Propagation Effectiveness. We conduct sanitychecks to verify the effectiveness of the back-propagationwith our approximated gradient. We take a pre-trainedDeepSDF [35] model and run geometry based optimizationto recover the 3D shape and camera extrinsics separately us-ing our differentiable renderer. We first assume camera poseis known and optimize the latent code for 3D shape w.r.t thegiven ground truth depth map, surface normal and silhou-ette. As can be seen in Fig. 5 (left), the loss drops quickly,and using acceleration strategies does not hurt the optimiza-tion. Fig. 5 (right) shows the total loss on the 2D imageplane is highly correlated with the Chamfer distance on the

initial optimized

Figure 6. Illustration of the optimization process over the cameraextrinsic parameters. Our differentiable renderer is able to propa-gate the error from the image plane to the camera. Top row: ren-dered surface normal. Bottom row: error map on the silhouette.

ε = 5× 10−2 ε = 5× 10−4 ε = 5× 10−6 ε = 5× 10−8

Figure 7. Effects on choices of different convergence thresholds.Under the same marching step, a very large threshold can incurdilation around boundaries while a small threshold may lead toerosion. We pick 5× 10−5 for all of our experiments.

predicted 3D shape, indicating that the gradients originatedfrom the 2D observation are successfully back-propagatedto the shape. We then assume a known shape (fixed latentcode) and optimize the camera pose using a depth imageand a binary silhouette. Fig. 6 shows that a random initialcamera pose can be effectively optimized toward the groundtruth pose by minimizing the gradients on 2D observations.

Convergence Criteria. The convergence criteria, i.e., thethreshold on signed distance to stop the ray tracing, has adirect impact on the rendering quality. Fig. 7 shows the ren-dering result under different thresholds. As can be seen,rendering with a large threshold will dilate the shape, whichlost boundary details. Using a small threshold, on the otherhand, may produces incomplete geometry. This parame-ter can be tuned according to applications, but in practicewe found our threshold is effective in producing completeshape with details up to the image resolution.

Rendering Other Properties. Not only the signed dis-tance function for 3D shape, implicit functions can also en-code other spatially variant information. As an example,we train a network to predict both signed distance and colorfor each 3D location, and this grants us the capability ofrendering color images. In Fig. 8, we show that with a512-dim latent code learned from textured meshes as theground truth, color images can be rendered in arbitrary res-olution, camera viewpoints, and illumination. Note that thelatent code size is significantly smaller than the mesh (ver-tices+triangles+texture map), and thus can be potentiallyused for model compression. Other per-vertex properties,such as semantic segmentation and material, can also berendered in the same differentiable way.

LR texture 32x HR texture HR Relighting HR 2nd View

Figure 8. Our method can render information encoded in the im-plict function other than depth. With a pre-trained network encod-ing textured meshes, we can render high resolution color imagesunder various resolution, camera viewpoints, and illumination.

4.2. 3D Shape Prediction

Our differentiable implicit SDF renderer builds up theconnection between 3D shape and 2D observations and en-ables geometry based reasoning. In this section, we showresults of 3D shape prediction from a single depth image,or multi-view color images using DeepSDF as the shapegenerator. On a high-level, we take a pre-trained DeepSDFand fixed the decoder parameters. When given 2D observa-tions, we define proper loss functions and propagate the gra-dient back to the latent code, as introduced in Section 3.4,to generate 3D shape. This method does not require anyadditional training and only need to run optimization at testtime, which is intuitively less vulnerable to overfitting ordomain gap issues in pure learning based approach. In thissection, we specifically focus on evaluating the generaliza-tion capability while maintaining high shape quality.

4.2.1 3D Shape Prediction from Single Depth Image

With the development of commodity range sensors, thedense or sparse depth images can be easily acquired, andseveral methods have been proposed to solve the problem of3D shape prediction from a single depth image. DeepSDF[35] has shown state-of-the-art performance for this task,however requires an offline pre-processing to lift the input2D depth map into 3D space in order to sample the SDFvalues with the assistance of the surface normal. Our differ-entiable render makes 3D shape prediction from a depth im-age more convenient by directly rendering the depth imagegiven a latent code and comparing it with the given depth.Moreover, with the silhouette calculated from the depth mapor provided from the rendering, our renderer can also lever-age it as an additional supervision. Formally, we obtain thecomplete 3D shape by solving the following optimization:

arg minzLd(Rd(f(z)), Id) + Ls(Rs(f(z)), Is), (5)

where f(z) is the pre-trained neural network encodingshape priors, Rd and Rs represent the rendering function

dense 50% 10% 100pts 50pts 20ptssofaDeepSDF 5.37 5.56 5.50 5.93 6.03 7.63Ours 4.12 5.75 5.49 5.72 5.57 6.95Ours (mask) 4.12 3.98 4.31 3.98 4.30 4.94planeDeepSDF 3.71 3.73 4.29 4.44 4.40 5.39Ours 2.18 4.08 4.81 4.44 4.51 5.30Ours (mask) 2.18 2.08 2.62 2.26 2.55 3.60tableDeepSDF 12.93 12.78 11.67 12.87 13.76 15.77Ours 5.37 12.05 11.42 11.70 13.76 15.83Ours (mask) 5.37 5.15 5.16 5.26 6.33 7.62

Table 2. Quantitative comparison between our geometric optimiza-tion with DeepSDF [35] for shape completion over partial denseand sparse depth observation on ShapeNet dataset [2]. We re-port the median Chamfer Distance on the first 200 instances ofthe dataset of [6]. We give DeepSDF [35] the groundtruth normalotherwise they could not be applied on the sparse depth.

for the depth and silhouette respectively, Ld is the L1 lossof depth observation, and Ls is the loss defined based onthe differentiably rendered silhouette. In our experiment,the initial latent shape z0 is chosen as the mean shape.

We test our method and DeepSDF [35] on 200 models onplane, sofa and table category respectively from ShapeNetCore [2]. Specifically, for each model, we use the first cam-era in the dataset of Choy et al. [6] to generate dense depthimages for testing. The comparison between DeepSDF andour method is listed in Tab. 2. We can see that our methodwith only depth supervision performs better than DeepSDF[35] when dense depth image is given. This is probably be-cause that DeepSDF samples the 3D space with pre-definedrule (at fixed distances along the normal direction), whichmay not necessarily sample correct location especially nearobject boundary or thin structures. In contrast, our differen-tiable sphere tracing algorithm samples the space adaptivelywith the current estimation of shape.

Robustness against sparsity. The depth from laser scan-ners can be very sparse, so we also study the robustnessof our method and DeepSDF against sparse depth. The re-sults are shown in Tab. 2. Specifically, we randomly sam-ple different percentages or fixed numbers of points fromthe original dense depth for testing. To make a competi-tive baseline, we provide DeepSDF ground truth normalsto sample SDF, since it cannot be reliably estimated fromsparse depth. From the table, we can see that even withvery sparse depth observations, our method still recoversaccurate shapes and gets consistently better performancethan DeepSDF with additional normal information. Whenthe silhouette is available, our method achieves significantlybetter performance and robustness against the sparsity, in-dicating that our rendering method can back-propagate gra-dients effectively from the silhouette loss.

Video sequence Optimization process

Figure 9. Illustration of the optimization process under multi-viewsetup. Our differentiable renderer is able to successfully recover3D geometry from a random code with only the photometric loss.

Method car planePMO (original) 0.661 1.129PMO (rand init) 1.187 6.124Ours (rand init) 0.919 1.595

Table 3. Quantitative results on 3D shape prediction from multi-view images under the metric of Chamfer Distance (only in the di-rection of gt→pred for fair comparison). We randomly picked 50instances from the PMO test set to perform the evaluation. 10000points are sampled from meshes for evaluation.

4.2.2 3D Shape Prediction from Multiple Images

Our differentiable renderer can also enable geometry basedreasoning for shape prediction from multi-view color im-ages by leveraging cross-view photometric consistency.

Specifically, we first initialize the latent code with a ran-dom vector and render a depth image for each of the inputviews. We then warp each color image to other input viewsusing the rendered depth image and the known camera pose.The difference between the warped and input images arethen defined as the photometric loss, and the shape can bepredicted by minimizing this loss. To sum up, the optimiza-tion problem is formulated as follows,

arg minz

N−1∑i=0

∑j∈Ni

‖Ii − Ij→i(Rid(f(z))‖, (6)

whereRid represents the rendered depth image at view i,Niare the neighboring images of Ii, and Ij→i is the warpedimage from view j to view i using the rendered depth.Note that no mask is required under the multi-view setup.Fig. 9 shows an example of the optimization process of ourmethod. As can be seen, the shape is gradually improvedwhile the loss is being optimized.

We take PMO [25] as a competitive baseline, since theyalso perform deep learning based geometric reasoning viaoptimization over a pre-trained decoder, but use the triangu-lar mesh representation. Their model first predicts an initialmesh from a selected input view and improve the qualityvia cross-view photo-consistency. Both the synthetic andreal datasets provided in [25] are used for evaluation.

In Tab. 3, we show quantitative comparison to PMO ontheir synthetic test set. It can be seen that our methodachieves comparable results with PMO [25] from only ran-dom initializations. Note that while PMO uses both theencoder and decoder trained on the PMO training set, ourDeepSDF decoder was neither trained nor finetuned on it.

0.6 0.8 1.0 1.5 2.0Focal Length Change

2

4

6

8

Cham

fer D

istan

ce

PMOOurs

0.01 0.02 0.03 0.04Noise Level

2

3

4

5

6

Cham

fer D

istan

ce

PMOOurs

(a) (b)

Figure 10. Robustness of geometric reasoning via multi-view pho-tometric optimization. (a) Performance w.r.t changes on camerafocal length. (b) Performance w.r.t noise in the initialization code.Our model is robust against focal length change and not affectedby noise in the latent code since we start from random initializa-tion. In contrast, PMO is very sensitive to both factors, and theperformance drops significantly when the testing images are dif-ferent from the training set.

Besides, if the shape code for PMO, instead of being pre-dicted from their trained image encoder, is also initializedrandomly, their performance decreases dramatically, whichindicates that with our rendering method, our geometric rea-soning becomes more effective. Our method can be furtherimproved with good initialization.Generalization Capability To further evaluate the gener-alization capability, we compare to PMO on some unseendata and initialization. We first evaluate both methods ona testing set generated using different camera focal lengths,and the quantitative comparison is in Fig. 10 (a). It clearlyshows that our method generalizes well to the new images,while PMO suffers from overfitting or domain gap. To fur-ther test the effectiveness of the geometric reasoning, wealso directly add random noise to the initial latent code.The performance of PMO again drops significantly, whileour method is not affected since the initialization is ran-domized (Fig. 10 (b)). Some qualitative results are shown inFig. 11. Our method produces accurate shapes with detailedsurfaces. In contrast, PMO suffers from two main issues: 1)the low resolution mesh is not capable of maintaining geo-metric details; 2) their geometric reasoning struggles withthe initialization from image encoder.

We further show comparison on real data in Fig. 12.Following PMO, since the provided initial similarity trans-formation is not accurate in some cases, we also optimizeover the similarity transformation in addition to the shapecode. As can be seen, both methods perform worse onthis challenging dataset. In comparison, our method pro-duces shapes with higher quality and correct structures,while PMO only produce very rough shapes. Overall, ourmethod shows better generalization capability and robust-ness against domain change.

5. Conclusion

We propose a differentiable sphere tracing algorithm torender 2D observations such as depth maps, normals, sil-

Video sequence PMO (rand init) PMO Ours

Figure 11. Comparison on 3D shape prediction from multi-viewimages on the PMO test set. Our method maintains good surfacedetails, while PMO suffers from the mesh representation and maynot effectively optimize the shape.

Video sequence PMO Ours

Figure 12. Comparison on 3D shape prediction from multi-viewimages on real-world dataset [5]. It is in general challenging forshape prediction on real image. Comparatively, our method pro-duces more reasonable results with correct structure.

houettes, from implicit signed distance functions parameter-ized as a neural network. This enables geometric reasoningin 3D shape prediction from both single and multiple viewsin conjunction with the high capacity 3D neural representa-tion. Extensive experiments show that our geometry basedoptimization algorithm produces 3D shapes that are moreaccurate than SOTA, generalizes well to new datasets, and isrobust to imperfect or partial observations. Promising direc-tions to explore using our renderer include self-supervisedlearning, recovering other properties jointly with geometry,and neural image rendering.

AcknowledgementsThis work is partly supported by National Natural Sci-

ence Foundation of China under Grant No. 61872012, Na-tional Key R&D Program of China (2019YFF0302902),and Beijing Academy of Artificial Intelligence (BAAI).

References[1] Bruce Guenther Baumgart. Geometric modeling for com-

puter vision. Technical report, STANFORD UNIV CADEPT OF COMPUTER SCIENCE, 1974. 1

[2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas,Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese,Manolis Savva, Shuran Song, Hao Su, et al. Shapenet:An information-rich 3d model repository. arXiv preprintarXiv:1512.03012, 2015. 7

[3] Wenzheng Chen, Jun Gao, Huan Ling, Edward J Smith,Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learn-ing to predict 3d objects with an interpolation-based differ-entiable renderer. In Proc. of Advances in Neural Informa-tion Processing Systems (NeurIPS), 2019. 2

[4] Zhiqin Chen and Hao Zhang. Learning implicit fields forgenerative shape modeling. In Proc. of Computer Vision andPattern Recognition (CVPR), 2019. 2

[5] Sungjoon Choi, Qian-Yi Zhou, Stephen Miller, and VladlenKoltun. A large dataset of object scans. arXiv preprintarXiv:1602.02481, 2016. 8, 20

[6] Christopher B Choy, Danfei Xu, JunYoung Gwak, KevinChen, and Silvio Savarese. 3d-r2n2: A unified approachfor single and multi-view 3d object reconstruction. InProc. of European Conference on Computer Vision (ECCV).Springer, 2016. 2, 3, 7

[7] Zhaopeng Cui, Jinwei Gu, Boxin Shi, Ping Tan, and JanKautz. Polarimetric multi-view stereo. In Proc. of ComputerVision and Pattern Recognition (CVPR), 2017. 3

[8] Angela Dai and Matthias Nießner. Scan2mesh: From un-structured range scans to 3d meshes. In Proc. of ComputerVision and Pattern Recognition (CVPR), pages 5574–5583,2019. 3

[9] Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner.Shape completion using 3d-encoder-predictor cnns andshape synthesis. In Proc. of Computer Vision and PatternRecognition (CVPR), 2017. 2, 3

[10] David Eigen and Rob Fergus. Predicting depth, surface nor-mals and semantic labels with a common multi-scale con-volutional architecture. In Proceedings of the IEEE inter-national conference on computer vision, pages 2650–2658,2015. 13

[11] Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Ab-hinav Gupta. Learning a predictable and generative vectorrepresentation for objects. In Proc. of European Conferenceon Computer Vision (ECCV). Springer, 2016. 3

[12] Thibault Groueix, Matthew Fisher, Vladimir G Kim,Bryan C Russell, and Mathieu Aubry. Atlasnet: A papier-mache approach to learning 3d surface generation. In Proc.of Computer Vision and Pattern Recognition (CVPR), 2018.2

[13] Christian Hane, Shubham Tulsiani, and Jitendra Malik. Hi-erarchical surface prediction for 3d object reconstruction. InProc. of International Conference on 3D Vision (3DV). IEEE,2017. 2

[14] John C Hart. Sphere tracing: A geometric method for theantialiased ray tracing of implicit surfaces. The Visual Com-puter, 12(10), 1996. 2, 3

[15] Carlos Hernandez, George Vogiatzis, and Roberto Cipolla.Multiview photometric stereo. IEEE Transactions on PatternAnalysis and Machine Intelligence, 30(3):548–554, 2008. 3

[16] Po-Han Huang, Kevin Matzen, Johannes Kopf, NarendraAhuja, and Jia-Bin Huang. Deepmvs: Learning multi-viewstereopsis. In Proc. of Computer Vision and Pattern Recog-nition (CVPR), pages 2821–2830, 2018. 3

[17] Sunghoon Im, Hae-Gon Jeon, Stephen Lin, and In SoKweon. Dpsnet: end-to-end deep plane sweep stereo. InProc. of International Conference on Learning Representa-tions (ICLR), 2019. 3

[18] Eldar Insafutdinov and Alexey Dosovitskiy. Unsupervisedlearning of shape and pose with differentiable point clouds.In Proc. of Advances in Neural Information Processing Sys-tems (NeurIPS), 2018. 2

[19] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker.Sdfdiff: Differentiable rendering of signed distance fields for3d shape optimization. In Proc. of Computer Vision and Pat-tern Recognition (CVPR), 2020. 2

[20] Adrian Johnston, Ravi Garg, Gustavo Carneiro, Ian Reid,and Anton van den Hengel. Scaling cnns for high resolutionvolumetric reconstruction from a single image. In Proc. ofInternational Conference on Computer Vision (ICCV), pages939–948, 2017. 3

[21] Angjoo Kanazawa, Michael J Black, David W Jacobs, andJitendra Malik. End-to-end recovery of human shape andpose. In Proc. of Computer Vision and Pattern Recognition(CVPR), 2018. 2

[22] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neu-ral 3d mesh renderer. In Proc. of Computer Vision and Pat-tern Recognition (CVPR), 2018. 2

[23] Chen Kong, Chen-Hsuan Lin, and Simon Lucey. Using lo-cally corresponding cad models for dense 3d reconstructionsfrom a single image. In Proc. of Computer Vision and Pat-tern Recognition (CVPR), 2017. 2

[24] Tzu-Mao Li, Miika Aittala, Fredo Durand, and Jaakko Lehti-nen. Differentiable monte carlo ray tracing through edgesampling. In Proc. of ACM SIGGRAPH, page 222. ACM,2018. 1, 2

[25] Chen-Hsuan Lin, Oliver Wang, Bryan C Russell, Eli Shecht-man, Vladimir G Kim, Matthew Fisher, and Simon Lucey.Photometric mesh optimization for video-aligned 3d objectreconstruction. In Proc. of Computer Vision and PatternRecognition (CVPR), 2019. 7, 14

[26] Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. Softrasterizer: Differentiable rendering for unsupervised single-view mesh reconstruction. In Proc. of International Confer-ence on Computer Vision (ICCV), 2019. 2

[27] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li.Learning to infer implicit surfaces without 3d supervision.

In Proc. of Advances in Neural Information Processing Sys-tems (NeurIPS), 2019. 2

[28] Stephen Lombardi, Tomas Simon, Jason Saragih, GabrielSchwartz, Andreas Lehrmann, and Yaser Sheikh. Neural vol-umes: Learning dynamic renderable volumes from images.Proc. of ACM SIGGRAPH, 2019. 2

[29] Matthew M Loper and Michael J Black. Opendr: An approx-imate differentiable renderer. In Proc. of European Confer-ence on Computer Vision (ECCV), pages 154–169. Springer,2014. 2

[30] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Se-bastian Nowozin, and Andreas Geiger. Occupancy networks:Learning 3d reconstruction in function space. In Proc. ofComputer Vision and Pattern Recognition (CVPR), 2019. 2

[31] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack,Mahsa Baktashmotlagh, and Anders Eriksson. Implicit sur-face representations as layers in neural networks. In Proc. ofInternational Conference on Computer Vision (ICCV), 2019.2

[32] Thu H Nguyen-Phuoc, Chuan Li, Stephen Balaban, andYongliang Yang. Rendernet: A deep convolutional networkfor differentiable rendering from 3d shapes. In Proc. of Ad-vances in Neural Information Processing Systems (NeurIPS),2018. 2

[33] Michael Niemeyer, Lars Mescheder, Michael Oechsle, andAndreas Geiger. Occupancy flow: 4d reconstruction bylearning particle dynamics. In Proc. of International Con-ference on Computer Vision (ICCV), 2019. 2

[34] Michael Oechsle, Lars Mescheder, Michael Niemeyer, ThiloStrauss, and Andreas Geiger. Texture fields: Learning tex-ture representations in function space. In Proc. of Interna-tional Conference on Computer Vision (ICCV), 2019. 2, 4

[35] Jeong Joon Park, Peter Florence, Julian Straub, RichardNewcombe, and Steven Lovegrove. Deepsdf: Learning con-tinuous signed distance functions for shape representation. InProc. of Computer Vision and Pattern Recognition (CVPR),2019. 1, 2, 3, 5, 6, 7, 11, 13, 14, 16, 17, 18

[36] Gustavo Patow and Xavier Pueyo. A survey of inverse ren-dering problems. In Computer graphics forum, volume 22,pages 663–687. Wiley Online Library, 2003. 1

[37] Felix Petersen, Amit H Bermano, Oliver Deussen, andDaniel Cohen-Or. Pix2vex: Image-to-geometry reconstruc-tion using a smooth differentiable renderer. arXiv preprintarXiv:1903.11149, 2019. 2

[38] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.Pointnet: Deep learning on point sets for 3d classificationand segmentation. In Proc. of Computer Vision and PatternRecognition (CVPR), 2017. 2

[39] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas JGuibas. Pointnet++: Deep hierarchical feature learning onpoint sets in a metric space. In Proc. of Advances in NeuralInformation Processing Systems (NeurIPS), 2017. 2

[40] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger.Octnet: Learning deep 3d representations at high resolu-tions. In Proc. of Computer Vision and Pattern Recognition(CVPR), 2017. 2

[41] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Mor-ishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned

implicit function for high-resolution clothed human digiti-zation. In Proc. of International Conference on ComputerVision (ICCV), 2019. 2

[42] Steven M Seitz, Brian Curless, James Diebel, DanielScharstein, and Richard Szeliski. A comparison and evalua-tion of multi-view stereo reconstruction algorithms. In Proc.of Computer Vision and Pattern Recognition (CVPR). IEEE,2006. 3

[43] Ben Semerjian. A new variational framework for multiviewsurface reconstruction. In Proc. of European Conference onComputer Vision (ECCV). Springer, 2014. 3

[44] Vincent Sitzmann, Justus Thies, Felix Heide, MatthiasNießner, Gordon Wetzstein, and Michael Zollhofer. Deep-voxels: Learning persistent 3d feature embeddings. In Proc.of Computer Vision and Pattern Recognition (CVPR), 2019.2

[45] Vincent Sitzmann, Michael Zollhofer, and Gordon Wet-zstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Proc. of Ad-vances in Neural Information Processing Systems (NeurIPS),2019. 2, 5, 13

[46] David Stutz and Andreas Geiger. Learning 3d shape comple-tion under weak supervision. International Journal of Com-puter Vision (IJCV), pages 1–20, 2018. 2

[47] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.Octree generating networks: Efficient convolutional archi-tectures for high-resolution 3d outputs. In Proc. of Interna-tional Conference on Computer Vision (ICCV), 2017. 2

[48] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, WeiLiu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d meshmodels from single rgb images. In Proc. of European Con-ference on Computer Vision (ECCV), 2018. 2

[49] Yifan Wang, Felice Serena, Shihao Wu, Cengiz Oztireli, andOlga Sorkine-Hornung. Differentiable surface splatting forpoint-based geometry processing. Proc. of ACM SIGGRAPHAsia, 2019. 2

[50] Yi Wei, Shaohui Liu, Wang Zhao, and Jiwen Lu. Condi-tional single-view shape generation for multi-view stereo re-construction. In The IEEE Conference on Computer Visionand Pattern Recognition (CVPR), June 2019. 3

[51] Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu.Pixel2mesh++: Multi-view 3d mesh generation via defor-mation. In Proc. of International Conference on ComputerVision (ICCV), 2019. 3

[52] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, andJosh Tenenbaum. Learning a probabilistic latent space ofobject shapes via 3d generative-adversarial modeling. InProc. of Advances in Neural Information Processing Systems(NeurIPS), 2016. 3

[53] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Lin-guang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3dshapenets: A deep representation for volumetric shapes. InProc. of Computer Vision and Pattern Recognition (CVPR),2015. 2

[54] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and LongQuan. Mvsnet: Depth inference for unstructured multi-viewstereo. In Proc. of European Conference on Computer Vision(ECCV), pages 767–783, 2018. 3

[55] Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins.Inverse global illumination: Recovering reflectance modelsof real scenes from photographs. volume 99, pages 215–224,1999. 1

[56] Andy Zeng, Shuran Song, Matthias Nießner, MatthewFisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch:Learning local geometric descriptors from rgb-d reconstruc-tions. In Proc. of Computer Vision and Pattern Recognition(CVPR), 2017. 2

AppendixIn this supplementary material, we provide detailed anal-

ysis of the proposed renderer, implementation details, andmore qualitative results.

A. More Analysis on the Design of Differen-tiable Sphere Tracing

A.1. Benefits of Aggressive Marching

We use an aggressive strategy (Sec. 3.2 in main submis-sion) to speed up the sphere tracing. Instead of marchingwith the step size as the SDF of the current location, wemarch α times of it, where α is larger than 1 and set as 1.5by default. This brings two benefits - faster convergenceand stable training.

Faster Convergence As shown in Fig. 13, the sphere trac-ing algorithm can become unexpectedly slow when the an-gle between the camera ray and the surface is relativelysmall. From an initial point with a ray distance d towardsthe surface, the marching step k needs to satisfy the equa-tion below to reach convergence:

|d(1− αsinθ)k| < ε, (7)

where α equals to 1.0 in the conventional sphere tracingalgorithm. When |1− αsinθ| < 1, we can easily derive theminimum marching step needed,

k > kmin =logε− logd

log|1− αsinθ|. (8)

By taking an aggressive strategy with α greater than 1 (weset it to 1.5 by default), the convergence can be speeded upunder the ill-posed conditions. For example, suppose d =1.0, ε = 5 × 10−5, the minimum number of convergencesteps decreases from 52 to 33 when θ equals to 10 degrees.

Stable Training Besides speeding up the overall march-ing, the aggressive marching also allows more samplingsfrom the locations behind the surface, which adds supervi-sion at the interior of the shape and stabilize the training.As shown in Fig. 14, in traditional ray marching, the frontend of the ray approaches the surface from the camera side

𝜃

P𝑑

+ −

Surface

Figure 13. Illustration of ill-posed conditions for sphere tracingalgorithm. When θ is relatively small, the queried SDF is muchlower than the actual ray distance d towards the surface, makingthe sphere tracing process slower than expected.

Figure 14. Compared to the regular ray marching algorithm, theaggressive ray marching strategy can march inside the surface andbounce back-and-forth between the inside and outside areas. Thisgives more samples on the negative side of implicit signed distancefunction and benefits the optimization.

(i.e. SDF > 0) and less likely to trespass the surface. Incontrast, our marching is more likely to pass through thesurface (and for sure under the ground truth SDF when theray direction is orthogonal to the surface since the marchingstep is larger than SDF). This will not add much computa-tional overhead when working conjointly with the conver-gence criteria, but achieves ray convergence from both sidesthe surface. This gives the training more supervision withboth positive and negative SDF, compared to positive onlyusing regular marching. The aggressive marching also nat-urally samples more points near the surface, which are im-portant for network to learn surface details. Coincidentally,DeepSDF [35] also mentioned the importance of samplingmore points near surface. While they need to rely on extraground truth normal and depth to perform the sampling, ourmethod is fully automatic and sample adaptively accordingto the SDF field.

A.2. Convergence Criteria

It is important to define a proper convergence criteria asshown in Fig. 7 of the main submission. In our differen-tiable sphere tracing, a ray stops marching if the absoluteSDF is smaller than certain threshold ε. Essentially, thismeans that the true intersection on the surface is boundedby a ball with radius of ε centered at our current sampling

Figure 15. Illustration on the geometric meaning of the thresholdε of the convergence criteria.

point. There are two guidelines to select this threshold. Onone hand, the threshold should not be too large, since therendering noise is theoretically bounded by this threshold.Large threshold may result in large error in the rendereddepth. On the other hand, the threshold must not be toosmall. The rendering time will be significantly longer if itis too small since more queries would be needed. More-over, with a fixed maximum number of tracing steps, somepixels may not converge and thus are considered as back-ground, which causes erosion (as shown in Fig. 7 in ourmain submission). Based on these two observations, wepropose to define the threshold as the distance where theray front ends from neighboring pixels are clearly separa-ble. This is equivalent to finding the radius such that ballscentered at the ray front ends of neighboring pixels do notintersect with each other. Fig. 15 demonstrates how to com-pute the ε. For a camera with focal length f , sensor size S,and resolution R, we get the following equation for objectsroughly dmin away from the camera, according to similartriangles:

S/R · cos(θ)f/cos(θ)

=2ε

dmin(9)

This gives:

ε =dmin · S · cos2(θ)

2 · f ·R(10)

Taking the common set up shown in Fig. 15, where f =60mm, dmin = 10cm, S = 32mm,R = 512, we get ε ≈0.5× 10−4m (0.05mm).

A.3. Differentiable Rendering of Silhouette

Fig. 16 shows how to render the silhouette in a dif-ferentiable way. After running our deep sphere tracing(Fig. 16 (a)), we render the minimal absolute SDF on eachpixel, and get the soft silhouette by substracting it by ε(Fig. 16 (b)). In this way the binary silhouette (Fig. 16 (c))can be easily acquired by checking whether the renderedsilhouette is positive (background) or not (foreground).

Because our rendered silhouette is fully differentiablewith respect to implicit signed distance functions and cam-

(a) Sphere Tracing (b) min(abs(SDF )) (c) min(abs(SDF )) < ε

Figure 16. The differentiable rendering of silhouette. We take min-imum absolute SDF value along each ray and determine the silhou-ette by checking whether the minimum absolute value is less thanthe threshold ε or not. We also consider the rendered soft silhou-ette as the minimum absolute queries substracted by ε, which isfully differentiable and feasible for optimization.

era extrinsic parameters, we can define differentiable lossterm over the rendered silhouette Sr and the ground-truthbinary silhouette Sgt. The silhouette loss Ls can be formu-lated as below:

Ls = Sgtmax(0, Sr) + (1− Sgt)max(0,−Sr). (11)

This formulation is able to get the silhouette error dif-ferentiably back-propagate to the optimized parameters.Note that by using the minimum absolute query, we utilizethe nice individual property for signed distance functions,where the nearest surface with respect to the camera ray isoptimized (as shown in Fig. 17). This strategy makes theshape optimization smooth and effective. Intuitively, com-bining the rendered silhouette with the distance transformon the 2D image plane can further improve the efficacy ofthe term, which is left for future work.

As discussed in the main paper, we check whether theray intersects with the unit sphere to generate an initializa-tion mask, where the ray without intersection with the unitsphere is directly set to background. To make the renderedsilhouette on those background pixels differentiable, we setthe soft silhouette value on each of those pixels to be thedistance from the origin to the corresponding camera rayminus 1.0. This design shares similar spirits with the previ-ous design on differentiable rendering of the silhouette. Be-cause the distance from the origin to each of those camerarays is always greater than 1.0, we can consistently checkwhether the rendered silhouette is negative to determine itscorresponding binary silhouette.

A.4. Drawbacks

Most of our acceleration strategy does not affect the ren-dering quality, except the aggressive marching. Fig. 18shows the drawback of the aggressive tracing strategy. Thismostly happens for the case when the geometry is super thinsuch that a single step of marching may trespass two sur-faces. As a result, the ray front end is still considered out ofthe shape and will keep marching till infinite, which causesartifacts shown in the back part of the truck.

Ray (foreground pixel)Min query

Figure 17. Differentiable error propagation along the boundary ofthe foreground. We can make use of the nice property of signeddistance fields to optimize the nearest surface.

parallel + dynamic + aggressive + coarse-to-fine

Figure 18. Illustration of the small artifacts induced by the aggres-sive strategy. Small holes could occur on thin surface areas. Betterviewed when zoomed in.

Another potential drawback is the well-known aliasingeffect, since for each pixel there is only one ray shot fromthe pixel center. Many well-known anti-aliasing strategiescan be directly applied to our renderer to mitigate this issue.

B. Implementation DetailsB.1. Network Architecture

We follow the same network architecture as DeepSDF[35], which consists of 9 fully connected layers. Eachhidden layer has a dimension of 512. For the texture re-rendering applications, we concatenate the shape code andtexture code together and feed the concatenated code to thetexture network that employs the same architecture as thegeometry network. Both the shape code and texture codehas a dimension of 256. Note that the network architec-ture of DeepSDF [35] is much heavier than the backboneused in [45] (4 layers with 256 dimensions for each). How-ever, with our proposed advanced sphere tracing strategies,we can produce high-resolution images within limited timeconsumption overhead.

B.2. Implementation of Dynamic Synchronized In-ference

Here we present some details on the implementation ofdynamic synchronized inference with the off-the-shelf deeplearning framework. We maintain a binary flag over eachcamera ray during the sphere tracing process. For eachstep, we concatenate all unfinished camera rays togetherand perform feedforward in a batch-wise manner and mapthem back to the original image resolution. Then, we checkwhether the tracing on each ray converges or gets out of theunit sphere and set the corresponding binary flags to zero.

Note that because the operation is performed on a concate-nated tensor in the computational graph, trivially comput-ing the minimum absolute SDF value for each ray resultsin unaffordable memory consumption. To address this is-sue, we introduce an implementation trick that the locationof each query is saved globally, and minimization is per-formed over a detached tensor graph. After this operation,we get the minimum K queries for each ray and feedforwardthose queries again with the gradients attached. This strat-egy enables our renderer to produce much high resolutionimages (2048× 2048) on a single GTX-1080Ti.

B.3. Depth / Normal / Silhouette / Color Loss

Our method renders 2D observations in a differentiablemanner and computes the loss on the image plane. Thedepth loss and normal loss are computed over the fore-ground region determined by the rendered silhouette. Forthe multi-view shape reconstruction application, we com-pute the photometric error over the commonly visible pixelsin two views.The visibility can be determined by comput-ing the difference ∆d between the reprojected depth fromthe source view and the directly rendered depth of the targetview. In our experiments, the pixels with ∆2

d < 0.001 areconsidered as visible. L1 loss is used for depth and photo-metric error computation, and negative dot product is com-puted to measure the loss of surface normals [10]. For therendered silhouette (substracting minimum absolute querywith ε), we require that the value should be negative overthe foreground and positive over the background and usethe loss term defined in Eq. (11).

B.4. More Details and Hyperparameters

Our framework is implemented in PyTorch and code willbe made publicly available. For all experiments, we usethe Adam optimizer with the initial learning rate 1e-2. Tocompute the Chamfer Distance for evaluation, we sample30k points on depth completion and 10k points on multi-view shape reconstruction respectively. We follow the com-mon practice to report the distance scaled by 1000. In themulti-view scenario, the 3D shapes from the DeepSDF de-coder may consist of structures inside some objects but theground-truth meshes only have the structure on the surface.Therefore, we report the Chamfer Distance only in the di-rection of gt→pred for fair comparison. For the samplingstretegy when rendering depth, empirically, K = 1 alreadyworks reasonably well. We use K = 3 for shape comple-tion and K = 1 for the multi-view shape reconstruction.The maximum marching step we use is 100. Empirically, avalue choice ranging from 1.2 to 1.8 can produce an effec-tive α. Small α results in slow convergence while large αcan lead to small holes in thin areas.

For shape completion, we initialize the latent code to bezero (which denotes the mean shape) and perform optimiza-

tion for 100 iterations. The loss weights of the depth lossand silhouette loss are set to 10.0 and 1.0 respectively. Wealso follow [35] to add an `2 regularizer over the shape codeduring optimization and the loss weight of the regulariza-tion term is set to 1.0.

For multi-view shape reconstruction under the PMO [25]synthetic test set, for each iteration we sample 8 views uni-formly distributed 360◦ around the object, warp each ofthem and calculate the photometric loss to the closest nextview based on the estimated depth image. We downsampleinput images to its half size 112 × 112 in order to back-propagate the photometric and regularization losses from all8 views together. Since there are 72 views provided in thedataset for each object, we run 9 iterations for each epochand 20 epochs in total.

As for the real-world multi-view dataset, in contrast toPMO [25] which uses over 100 images of an object, weonly picked between 20 and 30 views and further downsam-ple them from 480× 640 to 96× 128. 6 views are selectedduring each iteration. Since the provided initial similaritytransformation is not accurate in some cases, we also opti-mize over this similarity transformation in addition to theshape code. We find out that it is usually enough to ac-quire an accurate similarity transformation after only 1 or 2epochs. The weight of the photometric loss is set to 5.0 forboth synthetic and real-world experiments.

C. Rendering DemosWe attach a video demo in the supplementary material to

show that our method can render high-resolution depth, sur-face normals, silhouette and RGB image with various light-ing conditions and camera viewpoints. Also, more quali-tative results on multi-view reconstruction are included inthe video. We also show a larger version on the texture re-rendering demo (Fig. 8 in the main paper) in Fig. 19.

D. Additional Experimental ResultsD.1. Qualitative Comparison for Shape Completion

We show qualitative comparison on shape completionunder different sparsity of input depth in Fig. 20, Fig. 21and Fig. 22. The visual quality of our optimized meshclearly outperforms the baseline method DeepSDF [35].By employing perspective camera model and using onlinecomputed error (compared to the fixed sampling strategyin DeepSDF [35]) to perform inverse optimization on therendered 2D observations, our method generates less holesin the predicted mesh when the input depth is sparse, par-ticularly from the original view where the input depth iscaptured. When the silhouette information is available, ourmethod can work reasonably well when the input depth isextremely sparse. Note that we give DeepSDF [35] the sur-face normal predicted by the dense ground-truth depth to

make its sampling feasible. On the contrary, we do notuse the surface normal information in the optimization ofour method, which potentially can further improve our per-formance. Our method can generate reasonable occludedpart with the assistance of the pretrained shape model prior.However, our method also could fail when the view of thecamera cannot give sufficient information and the depthcompletion problem becomes extremely ill-posed.

D.2. Qualitative Comparison for Multi-view ShapePrediction

We firstly show more qualitative comparisons on PMOtest dataset in Fig. 23. It can be noticed that our methodproduces much more visually satisfactory 3D shapes withonly random initializations. In contrast, even if PMO [25]uses a much better initialization from the encoder pretrainedon their PMO training set, their 3D shapes have low resolu-tions because of the limited number of vertices. Moreover,if the random initialized codes are applied, PMO fails togenerate reasonable 3D shapes in most of the cases. Whenchecking closely the second column in Fig. 23, the resultseven converge to rather similar shapes, especially for chairsand planes.

We also illustrate more results on the real-world multi-view chair dataset in Fig. 24. Note that similar to PMO, wenow also optimize over the similarity transformation includ-ing rotation, translation and scale, together with the shapecode. It can be noticed easily that our method with ran-dom initialization again generates much superior outputsover PMO. We also show a failure case in the last row.Our method fails when there is insufficient texture on fore-ground or background as photometric cues. Moreover, ourmethod may fail when the similarity transformation can notbe correctly estimated.

LR texture 32x HR texture HR Relighting HR 2nd View

Figure 19. Qualitative results on the applications on texture re-rendering, where we can generate high-resolution outputs under variousresolution, camera viewpoints and illumination.

dense 50% pts 10% pts 100 pts 50 pts 20 pts

Input depth (direct view)

DeepSDF [35] (direct view)

DeepSDF [35] (second view)

DeepSDF [35] (third view)

Ours (direct view)

Ours (second view)

Ours (third view)

Ours w. mask (direct view)

Ours w. mask (second view)

Ours w. mask (third view)

Figure 20. Qualitative comparisons on shape completion under different sparsity of input depth (sofa).

dense 50% pts 10% pts 100 pts 50 pts 20 pts

Input depth (direct view)

DeepSDF [35] (direct view)

DeepSDF [35] (second view)

DeepSDF [35] (third view)

Ours (direct view)

Ours (second view)

Ours (third view)

Ours w. mask (direct view)

Ours w. mask (second view)

Ours w. mask (third view)

Figure 21. Qualitative comparisons on shape completion under different sparsity of input depth (plane).

dense 50% pts 10% pts 100 pts 50 pts 20 pts

Input depth (direct view)

DeepSDF [35] (direct view)

DeepSDF [35] (second view)

DeepSDF [35] (third view)

Ours (direct view)

Ours (second view)

Ours (third view)

Ours w. mask (direct view)

Ours w. mask (second view)

Ours w. mask (third view)

Figure 22. Qualitative comparisons on shape completion under different sparsity of input depth (table).

Video sequence PMO (rand init) PMO Ours (rand init)

Figure 23. More quatitative comparisons on 3D shape prediction from multi-view images on the PMO test set.

Video sequence PMO Ours (rand init)

Figure 24. More Comparisons on 3D shape prediction from multi-view images on real-world chair dataset [5]. It is in general challengingfor shape prediction on real images. Comparatively, our method produces more reasonable results with correct structure.