Discrete Optimization of Ray Potentials for Semantic 3D Reconstruction

Nikolay Savinov1, L’ubor Ladicky1, Christian Hane and Marc PollefeysETH Zurich, Switzerland

{nikolay.savinov,lubor.ladicky,christian.haene,marc.pollefeys}@inf.ethz.ch

Abstract

Dense semantic 3D reconstruction is typically formu-lated as a discrete or continuous problem over label assign-ments in a voxel grid, combining semantic and depth like-lihoods in a Markov Random Field framework. The depthand semantic information is incorporated as a unary po-tential, smoothed by a pairwise regularizer. However, mod-elling likelihoods as a unary potential does not model theproblem correctly leading to various undesirable visibilityartifacts.

We propose to formulate an optimization problem that di-rectly optimizes the reprojection error of the 3D model withrespect to the image estimates, which corresponds to the op-timization over rays, where the cost function depends on thesemantic class and depth of the first occupied voxel alongthe ray. The 2-label formulation is made feasible by trans-forming it into a graph-representable form under QPBO re-laxation, solvable using graph cut. The multi-label prob-lem is solved by applyingα-expansion using the same re-laxation in each expansion move. Our method was indeedshown to be feasible in practice, running comparably fastto the competing methods, while not suffering from ray po-tential approximation artifacts.

1. Introduction

In this paper we are studying the problem of jointly infer-ring dense 3D geometry and semantic labels from multipleimages, formulated as an optimization over rays. The prob-lem of dense 3D reconstruction from images and semanticsegmentation is up to date still a hard problem. A particu-larly powerful approach to the problem of dense 3D recon-struction from images is to pose it as a volumetric labelingproblem. The volume is segmented into occupied and freespace (the inside and the outside of an object) and the sur-face is extracted as the boundary in between. Traditionally,the data costs are extracted from the input images either di-rectly by computing matching scores per voxel or by first

1The authors assert equal contribution and joint first authorship

computing depth maps and deriving a per pixel unary po-tential based on the depth maps. In both cases after theimage data has been converted to a per voxel unary term theinput images are discarded. Unary terms, approximatelymodelling the likelihood the depth for a given pixel agreeswith the estimate, typically encourage voxels in an inter-val just before the matched 3D point to take the free-spacelabel and voxels in an interval right after the matched 3Dpoint to take the foreground label. However, this assump-tion does not hold in general. The interval right behind thecorner of an object does not necessarily have to belong toforeground. Failures due to this problem lead to blowing upcorners, roofs of buildings or thin objects. The problem canbe partly fixed by decreasing the length of an interval, how-ever, in that case in presence of noise the matched neigh-bouring points start to compete against each other. Anotherproblem of unary approximation is that the unary potentialdoes not model, whether the voxel is visible. If there is ahole in the wall and a matched object behind, there is nopenalty associated with a data term for closing the hole,which might get filled in by regularization. This problemcan also be partially resolved by penalizing foreground forall pixels in front of the matched voxel, however such solu-tion is not robust to outliers. The standard approach is alsonot suitable for incorporating multiple candidate matchesalong the viewing ray in the optimization together.

We propose to formulate an optimization problem whichmeasures the data fidelity directly in image space while stillhaving all the benefits of a volumetric representation. Themain idea is to use a volumetric representation, but describethe data cost as a potential over rays. Traversing along aray from the camera center we observe free space until wefirst hit an occupied voxel of a certain semantic class andwe cannot assume anything about the unobserved space be-hind. The potential we introduce correctly assigns for eachray the cost, based on the depth and semantic class of thefirst occupied voxel along the ray. The key to make sucha formulation feasible is the transformation of the potentialinto a graph-representable form under QPBO relaxation [2],solvable using graphcut-based methods. Our proposed op-timization method can be directly used also for multi-label

problems by applyingα-expansion [3] with QPBO used tocalculate each expansion move. Our method runs compara-bly fast to the competing methods, while not suffering fromray potentials approximation artifacts.

1.1. Related Work

Generating dense 3D models out of multiple images isa well-studied problem in computer vision. An overviewis given in [21]. Posing the problem of dense 3D recon-struction as a volumetric segmentation was first proposedin [7]. The initial formulation does not use any regular-ization. However, often the data is contaminated by noiseand strong assumptions about the smoothness have to bemade. Regularizing the surface by penalizing the surfacearea has been proposed in the discrete graph cut formula-tion [16, 25] and also as continuous convex optimizationformulation [28]. The solution space is not restricted toregular voxel grids. [27] uses a thetrahedronization of thespace. The data cost is formulated as a pairwise potentialalong viewing rays. They put a unary prior for a tetrahedronright behind the initial estimated depth match and a penaltyfor every cut before it. The photo-consistency of faces isused as a weight for the cost to cut a ray with a given face.A Markov Random Field (MRF) formulation over rays hasbeen proposed to estimate surface geometry and appearancejointly [17, 18]. The energy is formulated as a reprojectionerror of the reconstructed voxels to the input images, whichjointly estimates voxel occupancy and color. The high qual-ity refining of approximate mesh has also been formulatedas a reprojection error minimization [26, 8].

The silhouettes of objects in the input image contain im-portant information about the geometry. They constrain thesolution, such that for every ray passing through the silhou-ette there must be at least one occupied voxel, and every rayoutside of the silhouette consists of free space voxels only.This constraint has been used in form of a convex relax-ation in [6]. [12] proposes an intelligent unary ballooningvisibility term based on the consensus from different views.In [23], the silhouettes are handled in a two-stage process,where the initial surface is reprojected into each image andthe interior is heuristically corrected using the sets of er-roneous pixels, by finding the most photo-consistent vox-els along the ray. Recently, also approaches which jointlyreason about geometry and semantic labels have been pro-posed [11]. For volumetric 3D reconstruction in indoor en-vironments a Conditional Random Field (CRF) model wasproposed in [13]. It includes higher order potentials overgroups of voxels to include priors from 2D object detectionsand 3D surface detections in the raw input depth data. Fur-thermore, potentials over rays are used to enforce visibilityof only one voxel along a ray.

2. Ray Energy Formulation

We are interested in finding the smooth solution, whoseprojection into each camera agrees with the depth and se-mantic observations. Thus, the energy will take the form:

E(x) =∑

r∈R

ψr(xr) +

∑

(i,j)∈E

ψp(xi, xj), (1)

where eachxi ∈ L is the voxel variable taking a label fromthe label setL with a special labellf ∈ L correspondingto free space;R is the set of rays,ψr(.) is the ray poten-tial over the set of voxelsxr , E is the set of local voxelneighbourhoods, andψp(·) is a smoothness enforcing pair-wise regularizer. Each rayr of lengthNr consists of voxelsxri = xri , wherei ∈ {0, 1, ..Nr − 1}. The ray potentialtakes the cost depending only on the first non-free spacevoxel along the rayKr (if there is any):

Kr =

{

min(i|xri 6= lf) if ∃xri 6= lf

Nr otherwise.(2)

The ray potential is defined as:

ψr(xr) = φr(K

r, xrKr ), (3)

wherexrNr = lf . The costs for each depth and semanticlabelφr(.) could be arbitrary and typically come from thecorresponding semantic and depth classifiers or measure-ments.

2.1. Optimization of the 2-label Ray Potentials

Let us first consider the two label case. Each variableximay take a value from the set{0, 1}, wherexi = 0 corre-sponds to the occupied voxel andxi = 1 corresponds to freespacelf . The ray potential (3) for arbitrary costs could benon-submodular even for a ray of the length2 making the2-label problem NP-hard in general. Thus, we propose a so-lution using QPBO relaxation [2], where the energyE(x)is transformed to a submodular energyE(x,x) with addi-tional variablesxi = 1 − xi, and solved by relaxing theseconstraints.

In our case, the non-submodular ray potentials are oflarger order than 2. To make our problem solvable usinggraph-cut, our goal is to transform these potentials into apairwise energy with additional auxiliary variablesz, suchthat:

ψr(xr) = min

z

ψq(xr ,xr, z), (4)

whereψq(.) is pairwise submodular. Additionally, to keepthe problem feasible, our goal is to find a transformation,for which the number of edges in the graph with auxiliaryvariables grows at most linearly with length of a ray. Weachieve this goal using these five steps:

1. Polynomial representation of the ray potential,

2. Transformation into higher order submodular potentialusing additional variablesx,

3. Pairwise graph construction of a higher order submod-ular potential using auxiliary variablesz,

4. Merging variables [20] to get the linear dependency ofthe number of edges on length,

5. Transformation into a normal form, symmetric overx

andx, suitable for QPBO [2].

Next we describe in details each one of these steps. Thetwo-label equivalent of the ray potential takes the form:

ψr(xr) :=

{

φr(min(i|xri = 0)) if ∃xri = 0

φr(Nr) otherwise,(5)

whereφr(i) := φr(i) is the cost taken, ifi is the first fore-ground pixel along the ray, andφr(Nr) is the cost for thewhole ray being free space. We would like to transformthis potential into the polynomial representation - the sumof products:

ψr(xr) = kr +

Nr−1∑

i=0

cri

i∏

j=0

xrj . (6)

Applying to equation (5), we getφr(K) = kr +∑K−1

i=0 cri ,thus kr = φr(0) and cri = φr(i + 1) − φr(i), ∀i ∈{0, ..Nr − 1}.

It is well-known that the productcri∏i

j=0 xrj is submod-

ular only if cri ≤ 0 [9]. For cri > 0 we can transform theproduct into submodular function using additional variablexri = 1− xri as:

cri

i∏

j=0

xrj = cri (1− xri )i−1∏

j=0

xrj = −crixri

i−1∏

j=0

xrj + cri

i−1∏

j=0

xrj .

(7)That means, that starting from the last term we can itera-tively check ifcri ≤ 0, and if it is not, we transform the termusing (7) and updatecri−1 := cri−1+ c

ri . The transformation

algorithm is explained in details in Algorithm 1.Ignoring the constant term we transform the potential

into:

ψr(xr) =

Nr−1∑

i=0

(

− ari

i∏

j=0

xrj − brixri

i−1∏

j=0

xrj

)

. (8)

Each product in the sum is submodular and graph-representable using one auxiliary variablezi ∈ {0, 1}. Thestandard pairwise graph constructions [9] for a negativeproduct terms are:

− ari

i∏

j=0

xrj = ari minzi

(

− zi +

i∑

j=0

zi(1 − xrj)

)

(9)

Algorithm 1 Transformation into submodular potential.Input: c

r,Nr

Output: ar, br

i = Nr − 1;while i ≥ 0 do

if cri ≤ 0 thenari = −cri , bri = 0

elseari = 0, bri = criif i > 0 then cri−1 = cri−1 + cri

end ifi = i− 1

end while

−brixri

i−1∏

j=0

xrj = bri minz′

i

(

− z′i + z′i(1− xri )

+

i−1∑

j=0

z′i(1− xrj )

)

, (10)

however, such constructions for each term in the sum wouldlead to a quadratic number of edges per ray. Instead, wefirst build a more complex graph constructions with(i +1) auxiliary variableszi ∈ {0, 1}i+1 andz′i ∈ {0, 1}i+1

respectively with the foresight, that this will lead to a graphconstruction with linear growth of the number of edges:

−ari

i∏

j=0

xrj = ari minzi

(

− zii + zii(1 − xri ) (11)

+

i−1∑

j=0

(zij+1(1− zij) + zij(1− xrj))

)

−brixri

i−1∏

j=0

xrj = bri minz′i

(

− z′i

i + z′i

i(1− xri ) (12)

+

i−1∑

j=0

(z′i

j+1(1− z′ji ) + z′

i

j(1− xrj))

)

,

where the (one of the) optimal assignment for auxiliary vari-ables is:

∀j ∈ {0, .., i} : zij =

j∏

k=0

xrk (13)

z′i

i = xri

i−1∏

k=0

xrk (14)

∀j ∈ {0, .., i− 1} : z′i

j =

j∏

k=0

xrk (15)

Both equations are structurally the same, so we just prove(11). It can be easily checked, that given the assignments ofauxiliary variables (13) every term is always going to be0except−zii, that is by definition (13) equal to the negativeproduct of allxr on the left side of (11). Next we have toshow, that ifxr 6= 1, there is no assignment of auxiliaryvariables leading to a negative cost. Let us try to constructsuchxr. The only term, that could be potentially negativeis−zii , thuszii = 1. However, to keep the whole expressionnegative, the remaining terms must be0 and thus:

(zij+1(1− zij) = 0) =⇒ ((zij+1 = 1) =⇒ (zij = 1)) (16)

(zij(1 − xri ) = 0) =⇒ ((zij = 1) =⇒ (xrj = 1)). (17)

(16) implieszi = 1 and then(17) impliesxr = 1. Thus,the assignment (13) is always optimal and (11) holds. Therecould be more possible optimal assignments, however forthe next step we only need this one to exist. The alterna-tive graph construction leads to much more complex graphsthan the standard one, however, we can decrease its growthin terms of edges by applying the merging theorem [20].The theorem states, that if for every assignment of inputvariablesx there exists at least one assignment of two ormore auxiliary variableszi, ..zj, such thatzi = .. = zj ,these variables can be replaced by a single variable withoutaltering the cost for any assignment ofx. In our graph con-structions we can see (13), that the valuezij =

∏j

k=0 xrk is

independent oni, and thus:

∀i ∈ {0, .., Nr − 1}, ∀j, k ∈ {0, .., i} : zji = zki . (18)

Furthermore, based on (15):

∀i ∈ {0, .., Nr− 1}, ∀j, k ∈ {0, .., i− 1} : z′j

i = zki . (19)

After removing the unnecessary top indexzji = zi andz′

ii = z′i, and summing up all corresponding weights, the

resulting pairwise construction takes the form:

ψr(xr) = min

z,z′

(

Nr−1∑

i=0

(

− ari zi − bri z′i (20)

+ f ri (1− zi)x

ri + bri (1− zi)x

ri

)

+

Nr−2∑

i=1

(

(f ri+1(1 − zi+1)zi + bri (1 − z′i+1)zi

)

)

,

wheref ri =

∑Nr−1i arj+

∑Nr−1i+1 brj . Unlike standard graph

construction, this one leads to a number of edges growingat most linearly with the length of the ray. The process ofmerging is visually depicted in Figure 1.

Finally, the potential is converted into a symmetric nor-

mal form, suitable for QPBO [2] as:

ψr(xr) =

1

2

(

minz

ψr(x,x, z)

+ minz

ψr(1− x,1− x,1− z)

)

, (21)

leading to the symmetric quadratic representation:

ψr(xr) =

1

2min

z,z′,z,z′

(Nr−1∑

i=0

(

− ari zi − bri z′i (22)

+ f ri (1 − zi)x

ri + bri (1 − zi)x

ri

)

+

Nr−2∑

i=1

(

f ri+1(1 − zi+1)zi + bri (1 − z′i+1)zi

)

+

Nr−1∑

i=0

(

− ari (1− zi)− bri (1 − z′i)

+ f ri zi(1 − xri ) + bri zi(1− xri )

)

+

Nr−2∑

i=1

(

f ri+1zi+1(1 − zi) + bri z

′i+1(1− zi)

)

)

.

To remove fractions, we multiply all potentials by a factorof 2. The resulting graph construction is shown in Figure 2.Our method inherits the properties of the standard QPBOfor pairwise graphs [2]. The set of variables, for whichxi = 1 − xi, is the part of the globally optimal solution.The remaining variables can be labelled using Iterated Con-ditional Modes (ICM) [1] iteratively per variable.

2.2. Optimization of the Multi-label Ray Potentials

To solve the multi-label case, we adapt theα-expansionalgorithm. The moves proposed byα-expansion [3], are en-coded by a binary transformation vectort, which encodes,how should the variablexi change after the move. Thereare two distinct cases - expansion move of the free spacelabel lf and expansion move of any other foreground labell 6= lf .

For the free space label we define the transformationfunction as:

Tlf (xi, ti) =

{

lf if ti = 1xi if ti = 0

(23)

Next we have to find the 2-label projection of multi-labelray potential (3) under this transformation functionTlf (.).Let s(r) be the ordered set of variables in a rayr, whichbefore the move do not take the labellf : s(r) = {ri|x

ri 6=

lf}. After the move, the first occupied voxel ofr will be thefirst occupied voxel ofs(r), and thus the ray potential (3)

Figure 1. The process of merging for ray of the length 3. Because the optimal assignment of auxiliary variables is the samefor everyconstruction, the variables can be merged and thus we can build the final graph just by summing up corresponding edges in each one ofthese6 graphs. The weights in the final graph will bef0 = a2 + a1 + a0 + b2 + b1, f1 = a2 + a1 + b2 andf2 = a2.

projects into 2 labels as:

ψr(tr) =

{

φr(K(t), xs(r)K(t)) if ∃ts(r)i = 0

φr(Nr, lf) otherwise,(24)

whereK(t) = min(i ∈ s(r)|ts(r)i = 0). The projec-

tion takes the form of (5), and thus can be solved using thegraph-construction (22).

For l 6= lf we define the transformation function as:

Tl(xi, ti) =

{

l if ti = 0xi if ti = 1

(25)

After the move the first non-free space label could be only infront of the first non-free space label of the current solution.Let s′(r) be the ordered set of variables in front of currentfirst occupied voxelKr : s′(r) = {ri|i ≤ Kr} (includingKr if Kr exists). The ray potential will project into:

ψr(tr) =

{

φr(K(t), l) if ∃ts′(r)

i = 0

φr(Ns′(r), xs′(r)Kr

) otherwise.(26)

The projection also takes the form of (5) and can be solvedusing (22).

3. Implementation Details

As an input our method uses semantic likelihoods, pre-dicted by a pixel-wise context-based classifier from [15],and depth likelihoods obtained by a plane sweep stereomatching algorithm using zero-mean normalized cross-correlation. In general the ray costs for a labell take theform:

φr(i, l) =

(

(λsemC(l) + λdepC(d(i))

)

d(i)2, (27)

whereC(l) is the semantic cost for labell (or sky for freespace labellf ), C(d(i)) is the cost for depthd(i) of a pixel

i, andλsem andλdep are the weights of both domains. Be-cause each pixel corresponds to patch in a volume, whichscales quadratically with distance, thus the costs have tobe weighted by a factor ofd(i)2 to keep constant ratio be-tween ray potentials and regularization terms. In theory, ourmethod allows for optimization over the whole depth costvolume, however this would require too much memory tostore. In practice, a few top matches (in our case we used atmost 3) for each pixel contain all the relevant informationand all remaining scores are typically random noise. In ourexperiments the costs for given depth took the form:

C(d(i)) =

{

wn(−1 +|d(i)−dn

top|

∆ ) if |d(i)− dntop| ≤ ∆

0 otherwise.(28)

wherewn is the weight of then-th match with the depthdntop, calculated as a ratio of confidence scores of the n-th match with respect to the best one. As a smoothnessenforcing pairwise potential we used the discrete approx-imation [14] of the continuous anisotropic pairwise regu-larizer from [11]. To deal with the high resolution of the3D we use a coarse-to-fine approximation with 3 subdivi-sion stages. As a graph-cut solver, we eventually used theIBFS algorithm [10] algorithm, which was typically5−50×faster than commonly used Boykov-Kolmogorov [3] algo-rithm optimized for lattice graphs. The run-time dependsnot only on the resolution, but also on the number of rayssampled. For example for the Castle dataset with50 millionvoxels and150 million rays the optimization took approxi-mately 40 minutes on 48 CPU cores, which is comparableor faster than other methods [11]. To extract mesh out ofvoxelized solution we used Marching cubes algorithm [19].Final models were smoothed using Laplacian smoothing toreduce discretization artifacts.

Figure 2. The graph construction for ray with 7 voxels. Each vari-able has at most 3 outgoing edges, thus the total number of themgrows linearly with the length of the ray.

4. Experiments

We tested our algorithm on6 datasets - South Build-ing [11], Catania [11], CAB [5], Castle-P30 [24], Provi-dence [11] and Vienna Opera [5]. The number of imagesranged from 30 for Castle to 271 for Opera. The seman-tic classifier [15] for 5 classes (building, tree, ground, clut-ter and sky) was trained on the CamVid [4] and MSRCdatasets [22], with additional training data from [11]. Fig-ure 3 shows the qualitative results for all datasets. Ourmethod managed to successfully reconstruct all 3D sceneswith a relatively high precision. Minor problems werecaused by insufficient amount of input data and incorrectprediction of semantic and depth estimators. The compari-son of models with the state-of-the-art volumetric 3D recon-struction algorithm is shown in the Figure 4. Our methodmanaged to fix systematic reconstruction artifacts, causedby approximations in the modelling of the true ray likeli-hoods - thin structures tend to be thickened (see columns,roof or tree trunk in the South Building datasets) or open-ings in the wall (such as arches or doors) undesirably closed,because there is no penalty associated with it.

5. Conclusion

In this paper we proposed feasible optimization methodfor volumetric 3D reconstruction by minimization of repro-jection error. Unlike several state-of-the-art methods, ouralgorithm does not suffer from the systematic errors dueto the approximations of corresponding ray potentials. Weshowed that a direct optimization of the higher order po-tentials by transformation into pairwise graph under QPBO

relaxation is indeed feasible in practice even for high reso-lution models. Further work will focus on principled incor-poration of other geometric cues in the optimization frame-work, such as estimated surface normals or planarity enforc-ing potentials. Another direction would be to investigatethe possibility of incorporating reprojection-minimizing raypotentials into other frameworks such as in continuous ormesh-based formulations.

Acknowledgements We gratefully acknowledge partialfunding by the Swiss National Science Foundation projectno. 157101 on 3D image understanding for urban scenes.

References

[1] J. Besag. On the statisical analysis of dirty pictures.Journalof the Royal Statistical Society, 1986.

[2] E. Boros and P. Hammer. Pseudo-boolean optimization.Dis-crete Applied Mathematics, 2002.

[3] Y. Boykov, O. Veksler, and R. Zabih. Fast approximate en-ergy minimization via graph cuts.Transactions on PatternAnalysis and Machine Intelligence, 2001.

[4] G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla. Seg-mentation and recognition using structure from motion pointclouds. InEuropean Conference on Computer Vision, 2008.

[5] A. Cohen, C. Zach, S. N. Sinha, and M. Pollefeys. Discover-ing and exploiting 3d symmetries in structure from motion.In Conference on Computer Vision and Pattern Recognition,2012.

[6] D. Cremers and K. Kolev. Multiview stereo and silhou-ette consistency via convex functionals over convex domains.Transactions on Pattern Analysis and Machine Intelligence,2011.

[7] B. Curless and M. Levoy. A volumetric method for build-ing complex models from range images. InConference onComputer graphics and interactive techniques, 1996.

[8] A. Delaunoy, M. Pollefeys, et al. Photometric bundle adjust-ment for dense multi-view 3d modeling. InConference onComputer Vision and Pattern Recognition, 2014.

[9] D. Freedman and P. Drineas. Energy minimization via graphcuts: Settling what is possible. InConference on ComputerVision and Pattern Recognition, 2005.

[10] A. V. Goldberg, S. Hed, H. Kaplan, R. E. Tarjan, andR. F. Werneck. Maximum flows by incremental breadth-firstsearch. InProc. ALGO ESA, 2011.

[11] C. Hane, C. Zach, A. Cohen, R. Angst, and M. Pollefeys.Joint 3D scene reconstruction and class segmentation. InConference on Computer Vision and Pattern Recognition,2013.

[12] C. Hernandez, G. Vogiatzis, and R. Cipolla. Probabilisticvisibility for multi-view stereo. InConference on ComputerVision and Pattern Recognition, 2007.

[13] B.-s. Kim, P. Kohli, and S. Savarese. 3D scene understandingby voxel-CRF. InInternational Conference on ComputerVision, 2013.

Example Image Depth Map Semantic Segmentation Reconstructed 3D model

Figure 3.Qualitative results on 6 data sets (from top to bottom): Castle [24], South Building [11], Catania [11], Providence [11], CAB [5]and Opera [5]. Our method successfully reconstructed challenging 3D data with high level of detail. Minor errors in the reconstructionswere caused by the combination of errors of the semantic classifier, insufficient amount data from certain viewpoints or errors in the depthprediction for smooth texture-less surfaces.

Joint Volumetric Fusion [11] Our Method

Figure 4. Comparison of method with the sate-of-the-art dense volumetric reconstruction [11] on South Building [11] and Castledatasets [24]. Incorporation of true semantic and depth raylikelihoods in the optimization framework led to the corrections of systematicreconstruction artifacts, due to the approximations in theformulation of the problem. As can be seen in the South Building reconstruction- state-of-the-art methods, enforcing estimated label in the range behind the depth match, typically causes thickening of the thin structures- columns, tree branches or building corners. Furthermore,as seen in the reconstruction of the Castle, the regularization, minimizing thesurface, often closes openings in the flat walls, such as arches in this case, because there is no cost associated. In our framework, suchsolution has a higher cost, because it disregards otherwisevalid matches behind the arch.

[14] V. Kolmogorov and Y. Boykov. What metrics can be approx-imated by geo-cuts, or global optimization of length/area andflux. In International Conference on Computer Vision, 2005.

[15] L. Ladicky, C. Russell, P. Kohli, and P. H. S. Torr. Associa-tive hierarchical CRFs for object class image segmentation.In International Conference on Computer Vision, 2009.

[16] V. Lempitsky and Y. Boykov. Global optimization for shapefitting. In Conference on Computer Vision and PatternRecognition, 2007.

[17] S. Liu and D. B. Cooper. Ray markov random fields forimage-based 3d modeling: Model and efficient inference.In Conference on Computer Vision and Pattern Recognition,2010.

[18] S. Liu and D. B. Cooper. A complete statistical inverse raytracing approach to multi-view stereo. InConference onComputer Vision and Pattern Recognition, 2011.

[19] W. E. Lorensen and H. E. Cline. Marching cubes: A highresolution 3d surface construction algorithm. InProceedingsof the 14th Annual Conference on Computer Graphics andInteractive Techniques, 1987.

[20] S. Ramalingam, C. Russell, L. Ladicky, and P. H. Torr.Efficient minimization of higher order submodular func-tions using monotonic boolean functions.Arxiv preprintarXiv:1109.2304, 2011.

[21] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, andR. Szeliski. A comparison and evaluation of multi-view

stereo reconstruction algorithms. InConference on Com-puter Vision and Pattern Recognition, 2006.

[22] J. Shotton, J. Winn, C. Rother, and A. Criminisi.Texton-Boost: Joint appearance, shape and context modeling formulti-class object recognition and segmentation. InEuro-pean Conference on Computer Vision, 2006.

[23] S. N. Sinha, P. Mordohai, and M. Pollefeys. Multi-viewstereo via graph cuts on the dual of an adaptive tetrahe-dral mesh. InInternational Conference on Computer Vision,2007.

[24] C. Strecha, W. von Hansen, L. Van Gool, P. Fua, andU. Thoennessen. On benchmarking camera calibration andmulti-view stereo for high resolution imagery. InConferenceon Computer Vision and Pattern Recognition, 2008.

[25] G. Vogiatzis, P. H. Torr, and R. Cipolla. Multi-view stereo viavolumetric graph-cuts. InConference on Computer Visionand Pattern Recognition, 2005.

[26] H.-H. Vu, R. Keriven, P. Labatut, and J.-P. Pons. Towardshigh-resolution large-scale multi-view stereo. InConferenceon Computer Vision and Pattern Recognition, 2009.

[27] H.-H. Vu, P. Labatut, J.-P. Pons, and R. Keriven. High accu-racy and visibility-consistent dense multiview stereo.Trans-actions on Pattern Analysis and Machine Intelligence, 2012.

[28] C. Zach. Fast and high quality fusion of depth maps. In3DData Processing, Visualization and Transmission, 2008.