Learning Low-order Models for Enforcing High-order Statistics

Patrick Pletscher Pushmeet KohliETH Zurich

Zurich, SwitzerlandMicrosoft Research

Cambridge, UK

Abstract

Models such as pairwise conditional randomfields (CRFs) are extremely popular in com-puter vision and various other machine learn-ing disciplines. However, they have limitedexpressive power and often cannot representthe posterior distribution correctly. Whilelearning the parameters of such models whichhave insufficient expressivity, researchers useloss functions to penalize certain misrepre-sentations of the solution space. Till now, re-searchers have used only simplistic loss func-tions such as the Hamming loss, to enableefficient inference. The paper shows how so-phisticated and useful higher order loss func-tions can be incorporated in the learning pro-cess. These loss functions ensure that theMAP solution does not deviate much fromthe ground truth in terms of certain higherorder statistics. We propose a learning algo-rithm which uses the recently proposed lower-envelop representation of higher order func-tions to transform them to pairwise func-tions, which allow efficient inference. We testthe efficacy of our method on the problem offoreground-background image segmentation.Experimental results show that the incorpo-ration of higher order loss functions in thelearning formulation using our method leadsto much better results compared to thoseobtained by using the traditional Hammingloss.

1 Introduction

Probabilistic models such as conditional random fields(CRFs) are extremely popular machine learning disci-

Appearing in Proceedings of the 15th International Con-ference on Artificial Intelligence and Statistics (AISTATS)2012, La Palma, Canary Islands. Volume 22 of JMLR:W&CP 22. Copyright 2012 by the authors.

plines. Pairwise CRFs, in particular, have been usedto formulate many image labeling problems in com-puter vision (Szeliski et al., 2008). However, their in-ability to handle higher order dependencies betweenrandom variables restricts their expressive power, andmakes them unable to represent the data well (Sud-derth & Jordan, 2008) i.e., the ground truth may notbe the Maximum a Posterior (MAP) solution underthe model.

Models containing higher order factors are able to en-code complex dependencies between groups of vari-ables, and can encourage solutions which match thestatistics of the ground truth solution (Potetz, 2007;Roth & Black, 2005; Woodford et al., 2009). However,the high computational cost of performing MAP infer-ence in such models has inhibited their use (Lan et al.,2006). Instead, there has been a widespread adoptionof the simpler and less powerful pairwise-CRF modelswhich allow efficient inference (Szeliski et al., 2008).

While learning the parameters of models with insuffi-cient expressivity, researchers can penalize certain mis-representations of the solution space using a ‘loss func-tion’ which specifies the deviations from ground truththat the learning algorithm should avoid (Tsochan-taridis et al., 2005; Taskar et al., 2003). Most previ-ous works on these topics have used simple choicesof the loss function, such as the Hamming loss orsquared loss, which lead to tractable learning algo-rithms (Szummer et al., 2008). However, in real worldapplications, researchers might prefer more generalloss functions which penalize deviations in some higherorder statistics.

The ability to use such higher order loss functions isparticularly important for many image labeling prob-lems in medical imaging where predictions other thanpixel labelling accuracy (Hamming loss) might be im-portant. For instance, in some diagnostic scenarios, ra-diologists/physicians are interested in the area/volumeof the segmentation of a tissue or tumor that is un-der investigation. In such cases, a loss function thatheavily penalizes solutions whose volume/area is verydifferent from that of the ground truth should be used.

Learning Low-order Models for Enforcing High-order Statistics

In this paper, we show how to learn the parameters oflow-order models such as pairwise CRFs under higher-order loss functions. These loss functions can ensurethat the MAP solution does not deviate much from theground truth in terms of certain higher order statis-tics. We propose an efficient learning algorithm whichuses the lower-envelop representation of higher orderfunctions (Kohli & Kumar, 2010) to transform themto pairwise functions. We demonstrate the power ofour method on the problem of foreground-backgroundimage segmentation. Experimental results show thatour method is able to obtain parameters which lead tobetter results compared to the traditional approach.

2 Max-margin learning

This section reviews max-margin learning (Taskaret al., 2003; Tsochantaridis et al., 2005) and introducesour notation. For a given input x ∈ X we considermodels that predict a multivariate output y ∈ Y1 bymaximizing a linearly parametrized score function (aMAP predictor):

fw(x) = argmaxy∈Y

〈w,φ(x,y)〉. (1)

Here φ(x,y) denotes a mapping of the input and out-put variables to a joint input/output feature space.In computer vision, such a feature map is generallyspecified implicitly through a graphical model. Fur-thermore, w denote the parameters of the model. Inour work we consider pairwise models G = (V, E) withenergies of the form

E(y,x,w) = −〈w,φ(x,y)〉 =∑i∈V

ψi(yi,x;wu) +∑

(i,j)∈E

ψij(yi, yj ,x;wp). (2)

Here w is separated into parameters for the unary po-tentials (wu) and pairwise potentials (wp). The max-imization problem in (1) can alternatively be writtenas an energy minimization

fw(x) = argminy∈Y

E(y,x,w). (3)

Having defined the form of the prediction func-tion, we now consider learning the parameters wof such a model. Given the training data set{(x1,y1), . . . , (xN ,yN )}, max-margin learning2 (orequivalently the structured SVM) formulates an upperbound on the empirical risk using a quadratic program(QP) with a combinatorial number of constraints. The

1Generally the dimension of the output space dependson the input x, which is neglected here.

2We consider the margin rescaled version.

exponential number of constraints can be dealt with bya cutting-plane approach (Tsochantaridis et al., 2005).The resulting QP for a regularizer weight λ reads asfollows:

minw,ξ

λ

2‖w‖2 +

N∑n=1

ξn (4)

s.t. maxy∈Y

[〈w,φ(xn,y)〉+ ∆yn(y)] (5)

− 〈w,φ(xn,yn)〉 ≥ ξn ∀nξn ≥ 0.

The slack-variable ξn measures the surrogate loss ofthe n-th example. ∆yn(y) denotes an application-specific loss function, measuring the error incurredwhen predicting y instead of the ground truth outputyn. We shall denote a generic ground truth label byy∗. The loss of an example, as given by the constraintin (5) is convex and hence the overall optimizationproblem allows for efficient optimization over w. TheQP is typically solved by variants of the cutting-planemethod shown in Algorithm 1. The algorithm oper-ates in an alternating fashion by first generating theconstraints for the current parameter estimates andthereafter solving the QP with the extended set of con-straints.

Algorithm 1 Cutting-plane algorithm as in (Finley& Joachims, 2008).

Require: (x1,y1), . . . , (xN ,yN ), λ, ε,∆y∗(· ).1: Sn ← ∅ for n = 1, . . . , N .2: repeat3: for n = 1, . . . , N do4: H(y):= ∆yn(y)+ 〈w,φ(xn,y)− φ(xn,yn)〉5: compute y = argmaxy∈Y H(y)6: compute ξn = max{0,maxy∈Sn H(y)}7: if H(y) > ξn + ε then8: Sn ← Sn ∪ {y}9: w ← optimize primal over

⋃n S

n

10: end if11: end for12: until no Sn has changed during iteration

Line 9 of Algorithm 1 corresponds to solving a stan-dard QP for the constraints in

⋃n S

n (a linear numberof constraints as in each iteration at most one addi-tional constraint is added for each example). The lossaugmented inference problem on line 5 poses the ma-jor computational bottleneck for many applications.Here, an energy minimization of the form (3) needsto be solved, with one important difference: The neg-ative loss term enters the energy. Depending on theloss term this can render the inference problem in-tractable. The loss augmented inference problem is

Patrick Pletscher, Pushmeet Kohli

investigated in detail in section 4. The next sectiondiscusses loss functions in general and introduces thelabel-count loss, which is promoted in our work.

3 Loss functions

Max-margin learning leaves the choice of the loss func-tion ∆y∗(y) unspecified. The loss allows the researcherto adjust the parameter estimation to the evaluationwhich follows the learning step. In our work we differ-entiate between low-order losses, which factorize, andhigh-order losses, which do not factorize. Factoriza-tion is considered to be a key property of a loss tomaintain computational tractability.

3.1 Low-order loss functions

For image labelling in computer vision a popular choiceis the pixelwise error, or also Hamming error. It isdefined as:

∆hammingy∗ (y) =

∑i∈V

yi 6= y∗i . (6)

For image labelling problems, it tries to prevent solu-tions with high pixel labelling error from having lowenergy under the model compared to the ground truth.If there is a natural ordering on the labels, such as inimage denoising, another common choice for the loss isthe squared pixelwise error. For the binary problemsstudied in our work, it is equivalent to the Hammingloss.

3.2 High-order loss functions

In many machine learning applications, practitionersare concerned with errors other than the simple Ham-ming loss. This is especially the case in medical imag-ing tasks involving segmentation of particular tissuesor tumors. In such problems, radiologists and physi-cians are sometimes more interested in measuring theexact volume or area of the tumor (or tissue) to ana-lyze if it is increasing or decreasing in size. This pref-erence can be handled during the learning process byusing a label-count based loss function.

More formally, consider a two-label image segmen-tation problem where we have to assign the label‘0’ (representing ‘tumour’) or ‘1’ (representing ‘non-tumour’) to every pixel/voxel in the image/volume.The area/volume based label-count loss function inthis case is defined as:

∆county∗ (y) =

∣∣∣∣∣∑i∈V

yi −∑i∈V

y∗i

∣∣∣∣∣ . (7)

Such a loss function prevents image labellings (seg-mentations) with substantially different area/volume

compared to the ground truth to be assigned a lowenergy under the model. As we will show, despite thehigh-order form of the label-count loss, learning withit in the max-margin framework is tractable.

It is easy to show that the label-count loss is a lowerbound on the Hamming loss:

∆county∗ (y) ≤ ∆hamming

y∗ (y). (8)

The work of Lempitsky & Zisserman (2010), Gould(2011) and Tarlow & Zemel (2011) are most closely re-lated to our paper. In (Lempitsky & Zisserman, 2010)a learning approach for counting is introduced. Themajor difference to our work stems from the modelthat is learned. In their work a continuous regres-sion function is trained, which predicts for each pixela positive real independent of all its neighboring pix-els. In our work a CRF is used, which includes depen-dencies among variables, only the loss term in learn-ing is changed. (Gould, 2011) discusses max-marginparameter learning in graphical models that containpotentials with a linear lower envelope representa-tion. However, the loss function used in their workis still restricted to be a simple Hamming loss. Theidea of learning with higher-order losses is also stud-ied in (Tarlow & Zemel, 2011). They discuss severalhigher-order loss functions, but only approximate al-gorithms are presented. To the best of our knowledge,our work introduces for the first time a subclass ofhigh-order loss functions, for which max-margin learn-ing remains tractable.

4 Loss augmented inference andlower-envelope representation

The loss-augmented energy minimization problem fora given input/output pair (x,y∗) is given by

minyE(y,x,w)−∆y∗(y). (9)

Even on its own, the problem of minimizing a gen-eral energy function of discrete variables is a NP-hardproblem. However, certain classes of functions havebeen identified for which the problem can be solvedexactly in polynomial time. These include pairwisefunctions that are defined over graphs that are tree-structured (Pearl, 1986) or perfect (Jebara, 2009).

Another important family of tractable functions aresubmodular functions which are discrete analogues ofconvex functions (Fujishige, 1991; Lovasz, 1983), a for-mal definition is given in the appendix. Submodularfunctions are particularly important because of theirwide use in modeling labelling problems in computervision such as 3D voxel segmentation (Snow et al.,

Learning Low-order Models for Enforcing High-order Statistics

∆county∗ (y)

∑i∈V yi

c

f2(y)

f1(y)

(a) Upper envelope for ∆county∗ (y).

−∆county∗ (y)

∑i∈V yi

c

f2(y)

f1(y)

(b) Lower envelope for −∆county∗ (y).

−∆cappedy∗ (y)

∑i∈V yi

c

f2(y)

f1(y)

f3(y)

(c) Capped loss.

Figure 1: Upper and lower envelope representations of the label-count loss and its negation. Here c :=∑

i∈V y∗i .

Interestingly, as the loss enters the loss-augmented energy with a negative sign, the resulting energy minimizationproblem miny E(y,x,w)−∆count

y∗ (y) becomes tractable. (c) shows an example of a loss which can be describedas the lower envelope of three linear functions.

2000) and foreground-background image segmentationproblems (Boykov & Jolly, 2001; Blake et al., 2004).

The presence of the loss term in the loss augmentedenergy minimization problem in (9) has the potentialto make it harder to minimize. The Hamming loss,however, has the nice property that it decomposes intounary terms which can be integrated in the energy,and thus does not make the loss-augmented energyminimization problem harder (Szummer et al., 2008).

4.1 Compact representation of higher-orderloss functions

While it is easy to incorporate the Hamming loss in thelearning formulation, this is not true for higher orderloss functions. In fact, a general n order loss functiondefined on k-state variables can require up to kn pa-rameters for just its definition. In recent years a lot ofresearch has been done on developing compact repre-sentation of higher-order functions (Kohli et al., 2007;Rother et al., 2009; Kohli & Kumar, 2010). In partic-ular, Kohli & Kumar (2010) proposed a representationbased on upper and lower envelopes of linear functionswhich enables the use of many popular classes of higherorder potentials employed in computer vision. Moreformally, they represent higher order functions as:

fh(y) = ⊗q∈Qfq(y) (10)

where ⊗ = {max,min}, and Q indexes a set of linearfunctions, defined as

fq(y) = µq +∑i∈V

∑a∈L

νqiaδ(yi = a) (11)

where the weights νqia and the constant term µq arethe parameters of the linear function fq(·), and the

function δ(yi = a) returns 1 if variable yi takes label aand returns 0 for all other labels. While the ⊗ =‘min’results in a lower envelope of the linear function, ‘max’results in the upper envelope.

The upper envelope representation, in particular, isvery powerful and is able to encode sophisticated sil-houette constraints for 3D reconstruction (Kohli &Kumar, 2010; Kolev & Cremers, 2008). It can alsobe used to compactly represent general higher orderenergy terms which encourage solutions to have a par-ticular distribution of labels. Woodford et al. (2009)had earlier shown that such terms were very useful informulations of image labelling problems such as imagedenoising and texture, and led to much better results.

Our higher order loss term defined in equation (7) canbe represented by taking the upper envelope of twolinear functions f1(·) and f2(·) that are defined as:

f1(y) =∑i∈V

yi −∑i∈V

y∗i , (12)

f2(y) =∑i∈V

y∗i −∑i∈V

yi. (13)

This is illustrated in Fig. 1a.

4.2 Minimizing loss augmented energyfunctions

Although upper envelope functions are able to repre-sent a large class of useful higher order functions, in-ference in models containing upper envelope potentialsinvolves the solution of a hard min-max optimizationproblem (Kohli & Kumar, 2010).

We made the observation that the loss term in theloss-augmented energy minimization problem (9) hasa negative coefficient, which allows us to represent the

Patrick Pletscher, Pushmeet Kohli

label-count based loss (7) by the lower envelope of thefunctions defined in equation (12) and (13) (visualizedin Fig. 1b).

Kohli and Kumar showed that the minimization ofhigher order functions that can be represented as lowerenvelopes of linear functions can be transformed tothe minimization of a pairwise energy function withthe addition of an auxiliary variable. In fact, in somecases, the resulting pairwise energy function can beshown to be submodular (Boros & Hammer, 2002; Kol-mogorov & Zabih, 2004) and hence can be minimizedby solving an minimum cost st-cut problem (Kohliet al., 2008). This is the case for all higher-order func-tions of Boolean variables which are defined as:

fh(y) = F(∑

i∈Vyi

), (14)

where F is a concave function. The worst case timecomplexity of the procedure described above is poly-nomial in the number of variables. A related familyof higher order submodular functions which can beefficiently minimized was characterized in (Stobbe &Krause, 2010). Next, we consider the loss augmentedinference for the label-count loss in more detail.

4.3 Label-count loss augmented inference

The minimization of the negative label-count basedloss (7) can be transformed to the following pairwisesubmodular function minimization problem:

miny−∆count

y∗ (y)

= miny−∣∣∣∣∣∑i∈V

yi −∑i∈V

y∗i

∣∣∣∣∣ (15)

= miny,z∈{0,1}

−z(∑

i∈Vyi −

∑i∈V

y∗i

)

−(1− z)(∑

i∈Vy∗i −

∑i∈V

yi

)

= miny,z∈{0,1}

2z

(∑i∈V

y∗i −∑i∈V

yi

)+∑i∈V

yi −∑i∈V

y∗i .

The full energy minimization for the count loss aug-mented inference reads as follows

miny,z∈{0,1}

E(y,x,w) + 2z

(∑i∈V

y∗i −∑i∈V

yi

)+∑i∈V

yi −∑i∈V

y∗i . (16)

We assume that the original energy E(y,x,w) is sub-modular. The pairwise problem above is exactly solved

by graph-cut (Boykov, 2001) on the original graph Gwhere we add one node for the variable z and |V| newedges connecting each segmentation variable yi to theauxiliary variable z. The pairwise energy constructionis visualized in Fig. 2.

z

yi0

1

z0 1

0

1

0

−1

z0 1

−c c

Figure 2: Pairwise graph used for solving the label-count loss augmented inference problem. The poten-tials of the edges connecting the segmentation nodes yito the auxiliary node z (which are shown in blue) arevisualized to the left. The unary potential of the aux-iliary variable z to the right, where c :=

∑i y∗i . Stan-

dard graph-cut solvers can be applied to this problem.

Unfortunately, we found the de-facto standard com-puter vision graph-cut algorithm by Boykov & Kol-mogorov (2004) to run fairly slowly on these probleminstances. We attribute this to the dense connectivityof the auxiliary node z. This problem is in theory, andas is turns out also in practice, solved by the recentiterative breadth-first search (IBFS) graph-cut algo-rithm introduced in (Goldberg et al., 2011). We foundthis algorithm to be roughly an order of magnitudemore efficient than the Boykov-Kolmogorov algorithm.Learning on a small subset of the data discussed in thenext section took two minutes when IBFS was usedand around 25 minutes with the Boykov-Kolmogorovalgorithm.

Alternatively, for minimizing the loss augmented en-ergy with a single Boolean z, as in (16), we can solvethe minimization efficiently by performing energy min-imization twice in the original graph (for z = 0 andz = 1). Each choice of z results in different unaries.This approach does however not scale to the case wherewe have multiple zs as the number of sub-problemsgrows exponentially. If we have a loss function with10 zs we will have to do the minimization 210 times.

5 Experiments

We implemented the max-margin learning in Matlab.For solving the QP the MOSEK solver was used. Theloss augmented inference with IBFS was implemented

Learning Low-order Models for Enforcing High-order Statistics

in C++ through a MEX wrapper. The IBFS code wasdownloaded from the authors webpage and modifiedto allow for double precision energies (as opposed tointeger precision). Submodularity of the model wasexplicitly enforced in training by ensuring that all theedge potential’s off-diagonals are larger than the di-agonals. This can be achieved by adding additionalconstraints to the QP. The loss is always normalizedby the number of pixels such that the loss is upperbounded by one.

5.1 Cell segmentation

Counting tasks naturally arise in many medical appli-cations. The estimation of the progression of cancerin a tissue or the density of cells in microscope imagesare two examples. As a first experiment we study theproblem of counting the number of mitochondria cellpixels in an image. The dataset is visualized in Fig. 3.The images have been provided by Angel Merchan andJavier de Felipe from the Cajal Blue Brain team atthe Universidad Politecnica de Madrid. Three images

Figure 3: Electroscopic image showing the mitochon-dria cells in red.

were used for learning, two images for the validationand the remaining five images for testing. The imageshave a resolution of 986×735. The pairwise CRF con-sisted of a unary term with three features (the responseof a unary classifier for mitochondria and synapse de-tection and an additional bias feature). The pairwiseterm incorporated two features (color difference be-tween neighboring pixels and a bias). The resultsare shown in a box plot in Fig. 4. As expected thelabel-count loss trained model performs better thanthe Hamming loss trained model if the label-count lossis used for the evaluation and vice-versa if evaluatedon the Hamming loss.

We also compared our lower envelope inferenceapproach to the compose max-product algo-

−1

−0.5

0

0.5

1

x 10−3

hamming count

loss d

iffe

ren

ce

Figure 4: Results for the mitochondria segmentation.We plot the normalized loss difference between theHamming loss trained model and the one trained us-ing label-count loss. The x-axis shows the loss usedfor the evaluation of the predictions. The negativevalue for the Hamming loss evaluation indicates that ifHamming loss is used for evaluation, training with theHamming loss is superior. The opposite is true whenevaluation considers the label-count loss as learningwith the label-count results in a lower loss.

rithm (Duchi et al., 2006) which is used in (Tarlow& Zemel, 2011). The latter inference approachis in general only approximate. However, for thecell segmentation problem in combination with thelabel-count loss, the solutions obtained using thetwo different loss-augmented inference algorithmswere almost identical. The running time of thetwo approaches is also comparable. Our inferencealgorithm is slightly more efficient, but also moreadapted to the count-loss.

5.2 Foreground-background segmentation

We check the effectiveness of the label-count loss forthe task of background-foreground segmentation onthe Grabcut dataset (Blake et al., 2004). We use theextended dataset from (Gulshan et al., 2010). Thedataset consists of 151 images, each comes with aground truth segmentation. Furthermore, for each im-age an initial user seed is specified by strokes mark-ing pixels belonging to the foreground or to the back-ground, respectively. As unary features we used thethree color channels together with the background andforeground posterior probabilities as computed by theGaussian mixture model algorithm used in Grabcut.Additionally we also included a constant feature tocorrect for class bias. For the pairwise features we usedthe color difference between the two pixels and again abias feature. The standard four-connected grid graphis used as the basic model. Each edge is parametrizedby the same parameter. We also experimented with ex-tensions of this basic model: In one variant we considerthe eight-connected grid, in the other variant each di-

Patrick Pletscher, Pushmeet Kohli

(a) Hamming (c: 0.077, h: 0.077). (b) Count (c: 0.037, h: 0.040). (c) Ground-truth.

(d) Hamming (c: 0.047, h: 0.047). (e) Count (c: 0.040, h: 0.043). (f) Ground-truth.

(g) Hamming (c: 0.069, h: 0.069). (h) Count (c: 0.012, h: 0.124). (i) Ground-truth.

Figure 5: Segmentations on the test set for models trained using the Hamming loss (left) and the label-count loss(middle). The image on the right shows the ground truth segmentation. We show the measured count loss andHamming loss in brackets. The bottom row shows a case where the model trained using the count loss shows amuch better count loss, however the Hamming loss substantially deteriorates due to the false positives. For thefirst two images, the label-count loss trained model even outperforms the Hamming loss trained model in termsof Hamming loss.

rection of the edge is parameterized using a differentparameter. The basic model is therefore specified byan eight dimensional parameter, the eight-connectedmodel where each direction has its own parameter bya 14-dimensional parameter. For learning 60 imageswere used, 20 for the validation of the regularizationparameter λ, the remaining images were used for test-ing.

Fig. 5 shows some of the learned segmentations andTable 1 gives a comparison of the models trained us-ing the Hamming loss and the label-count loss. Theresults were averaged over four different data splits.As expected, we observe that if the label-count loss isused for the evaluation, the model that is trained usingthis loss performs superior. More interesting is the re-sult for the case when the Hamming loss is used for theevaluation. Despite the fact that the appropriate loss

is used in training, we do not identify a statisticallysignificant advantage of the Hamming loss over thelabel-count loss. This could be explained by the max-margin objective only considering an upper bound onthe loss, and not the actual loss itself. The label-countloss might suffer less from this upper bounding thanthe Hamming loss.

6 Discussion

We have demonstrated, for the first time, how low-order models like pairwise CRFs can be encouraged topreserve higher order statistics by introducing higherorder loss functions in the learning process. The learn-ing involves the minimization of the loss augmentedenergy, which we show can be performed exactly forcertain loss functions by employing a transformation

Learning Low-order Models for Enforcing High-order Statistics

PPPPPPPPEvalTrain

Hamming better (%) Count better (%)

4/S Hamming 52.1± 7.0 47.9± 7.0

Count 33.8± 8.3 66.2± 8.3

4/D Hamming 39.4± 6.1 60.6± 6.1

Count 29.6± 8.3 70.4± 8.3

8/S Hamming 48.2± 11.9 51.8± 11.9

Count 32.0± 13.1 68.0± 13.1

8/D Hamming 50.0± 9.2 50.0± 9.2

Count 40.5± 14.3 59.5± 14.3

Table 1: Test performance of models trained using the Hamming and the label-count loss for different modelstructures. The structure of the model is shown on the far left (4 vs. 8 grid, same vs. different parameterizationof the edges). The second column shows the percentage of images for which the model trained using Hammingloss has a lower evaluation loss. The third column shows the same information for the label-count loss. The rowsshow the loss used in the evaluation. If the loss function affects training, we would expect both columns to showvalues considerably above 50% for the corresponding loss. For learning with the label-count loss this is the case,for the Hamming loss the two learned models perform roughly the same.

scheme. We demonstrate the efficacy of our methodby using a label-count loss while learning a pairwiseCRF model for binary image segmentation. The label-count loss function is useful for applications that re-quire the count of positively labeled pixels in an imageto match the count observed on a ground truth seg-mentation. Our proposed algorithm enables efficientmax-margin learning under the label-count loss, andleads to models that produces solutions with statisticsthat are closer to the ground truth, compared to solu-tions of models learned using the standard Hammingloss.

Acknowledgements

We would like to thank Pablo Marquez Neila for shar-ing the mitochondria cell segmentation data set andthe unary classifier responses. We would also like tothank D. Tarlow for helping us with getting the com-pose inference code to work in combination with thelabel-count loss.

References

Blake, A., Rother, C., Brown, M., Perez, P., and Torr,P. H. S. Interactive image segmentation using anadaptive gmmrf model. In ECCV, pp. 428–441,2004.

Boros, E. and Hammer, P.L. Pseudo-boolean opti-mization. Discrete Applied Mathematics, 2002.

Boykov, Y. Fast approximate energy minimization viagraph cuts. IEEE Transactions on Pattern Analysisand Machine Intelligence (PAMI), 8(4):413–1239,2001.

Boykov, Y. and Jolly, M.P. Interactive graph cuts for

optimal boundary and region segmentation of ob-jects in N-D images. In ICCV, 2001.

Boykov, Y. and Kolmogorov, V. An experimental com-parison of min-cut/max-flow algorithms for energyminimization in vision. IEEE Trans. Pattern Anal.Mach. Intell., 26(9):1124–1137, 2004.

Duchi, J., Tarlow, D., Elidan, G., and Koller, D. UsingCombinatorial Optimization within Max-ProductBelief Propagation. In NIPS, 2006.

Finley, T. and Joachims, T. Training structural SVMswhen exact inference is intractable. Proceedings ofthe 25th international conference on Machine learn-ing - ICML ’08, pp. 304–311, 2008.

Fujishige, S. Submodular functios and optimization.Annals of Discrete Mathematics, Amsterdam, 1991.

Goldberg, A. V., Hed, S., Kaplan, H., Tarjan, R. E.,and Werneck, R. F. Maximum flows by incrementalbreadth-first search. In ESA, pp. 457–468, 2011.

Gould, S. Max-margin learning for lower linear en-velope potentials in binary markov random fields.Proceedings of the International Conference on Ma-chine Learning (ICML), 2011.

Gulshan, V., Rother, C., Criminisi, A., Blake, A., andZisserman, A. Geodesic star convexity for interac-tive image segmentation. In Proceedings of the IEEEConference on Computer Vision and Pattern Recog-nition, 2010.

Jebara, T. MAP estimation, message passing, and per-fect graphs. In Uncertainty in Artificial Intelligence,2009.

Kohli, P. and Kumar, M. P. Energy minimization forlinear envelope MRFs. In CVPR, 2010.

Patrick Pletscher, Pushmeet Kohli

Kohli, P., Kumar, M. P., and Torr, P. H. S. P 3 andbeyond: Solving energies with higher order cliques.In CVPR, 2007.

Kohli, P., Ladicky, L., and Torr, P. Robust higherorder potentials for enforcing label consistency. InCVPR, 2008.

Kolev, K. and Cremers, D. Integration of multiviewstereo and silhouettes via convex functionals on con-vex domains. In ECCV, 2008.

Kolmogorov, V. and Zabih, R. What energy functionscan be minimized via graph cuts?. PAMI, 2004.

Lan, X., Roth, S., Huttenlocher, D. P., and Black,M. J. Efficient belief propagation with learnedhigher-order markov random fields. In ECCV (2),pp. 269–282, 2006.

Lempitsky, V. and Zisserman, A. Learning To CountObjects in Images. In NIPS, 2010.

Lovasz, L. Submodular functions and convexity. InMathematical Programming: The State of the Art,pp. 235–257, 1983.

Pearl, J. Fusion, propagation, and structuring in beliefnetworks. Artif. Intell., 29(3):241–288, 1986.

Potetz, B. Efficient belief propagation for vision usinglinear constraint nodes. In CVPR, 2007.

Roth, S. and Black, M. J. Fields of experts: A frame-work for learning image priors. In CVPR, pp. 860–867, 2005.

Rother, C., Kohli, P., Feng, W., and Jia, J. Minimiz-ing sparse higher order energy functions of discretevariables. In CVPR, 2009.

Snow, D., Viola, P., and Zabih, R. Exact voxel occu-pancy with graph cuts (470), 2000.

Stobbe, P. and Krause, A. Efficient minimization ofdecomposable submodular functions. In Proc. Neu-ral Information Processing Systems (NIPS), 2010.

Sudderth, E. and Jordan, M. Shared segmentationof natural scenes using dependent pitman-yor pro-cesses. In NIPS, pp. 1585–1592, 2008.

Szeliski, R., Zabih, R., Scharstein, D., Veksler, O.,Kolmogorov, V., Agarwala, A., Tappen, M., andRother, C. A Comparative Study of Energy Min-imization Methods for Markov Random Fields withSmoothness-Based Priors. IEEE Transactions onPattern Analysis and Machine Intelligence (PAMI),30(6):1068–1080, 2008.

Szummer, M., Kohli, P., and Hoiem, D. LearningCRFs using graph cuts. In ECCV, pp. 582–595,2008.

Tarlow, D. and Zemel, R. Big and tall: Large mar-gin learning with high order losses. In CVPR 2011

Workshop on Inference in Graphical Models withStructured Potentials, 2011.

Taskar, B., Guestrin, C., and Koller, D. Max-MarginMarkov Networks. In Advances in Neural Informa-tion Processing Systems (NIPS), 2003.

Tsochantaridis, I., Joachims, T., Hofmann, T., andAltun, Y. Large Margin Methods for Structuredand Interdependent Output Variables. Journal ofMachine Learning Research, 6:1453–1484, 2005.

Woodford, O., Rother, C., and Kolmogorov, V. Aglobal perspective on MAP inference for low-levelvision. In ICCV, 2009.

A Submodularity

For the formal definition of submodular functions, con-sider a function f(·) that is defined over the set ofvariables y = {y1, y2, ..., yn} where each yi takes val-ues from the label set L = {l1, l2, ...lL}. Then, givenan ordering over the label set L, the function f(·) issubmodular if all its projections3 on two variables sat-isfy the constraint:

fp(a, b) + fp(a+ 1, b+ 1) ≤fp(a, b+ 1) + fp(a+ 1, b), (17)

for all a, b ∈ L.

3A projection of any function f(·) is a function fp whichis obtained by fixing the values of some of the argumentsof f(·). For instance, fixing the value of k variables of thefunction f1 : Rn → R produces the projection fp

1 : Rn−k →R.