Vis ComputDOI 10.1007/s00371-010-0494-2

O R I G I NA L A RT I C L E

A comparative study of existing metrics for 3D-mesh segmentationevaluation

Halim Benhabiles · Jean-Philippe Vandeborre ·Guillaume Lavoué · Mohamed Daoudi

© Springer-Verlag 2010

Abstract In this paper, we present an extensive experimen-tal comparison of existing similarity metrics addressing thequality assessment problem of mesh segmentation. We in-troduce a new metric, named the 3D Normalized Probabilis-tic Rand Index (3D-NPRI), which outperforms the othersin terms of properties and discriminative power. This com-parative study includes a subjective experiment with humanobservers and is based on a corpus of manually segmentedmodels. This corpus is an improved version of our previ-ous one (Benhabiles et al. in IEEE International Confer-ence on Shape Modeling and Application (SMI), 2009). Itis composed of a set of 3D-mesh models grouped in dif-ferent classes associated with several manual ground-truthsegmentations. Finally the 3D-NPRI is applied to evaluatesix recent segmentation algorithms using our corpus andthe Chen et al.’s (ACM Trans. Graph. (SIGGRAPH), 28(3),2009) corpus.

Keywords 3D-mesh segmentation · Ground-truth ·Similarity metric · Subjective tests · Evaluation

H. Benhabiles (�) · J.-P. Vandeborre · M. DaoudiLIFL (UMR USTL/CNRS 8022), University of Lille, Lille,Francee-mail: halim.benhabiles@lifl.fr

J.-P. Vandeborre · M. DaoudiInstitut TELECOM; TELECOM Lille 1, Lille, France

J.-P. Vandeborree-mail: jean-philippe.vandeborre@lifl.fr

M. Daoudie-mail: mohamed.daoudi@lifl.fr

G. LavouéCNRS, INSA-Lyon, LIRIS, UMR5205, Université de Lyon,69621, Lyon, Francee-mail: glavoue@liris.cnrs.fr

1 Introduction

3D-mesh segmentation is a fundamental process in many ap-plications such as shape retrieval [1, 29], compression [29],deformation [17], texture mapping [26], etc. It consists indecomposing a polygonal surface into different regions (i.e.connected set of vertices or facets) of uniform properties,either from a geometric point of view or from a semanticpoint of view. It is a critical step toward content analysisand mesh understanding. Although some supervised meth-ods exist [12, 16], most existing techniques are fully auto-matic.

According to recent states-of-the-art [3, 24], mesh seg-mentation techniques can be classified into two categories:surface-type (or geometric) methods and part-type (or se-mantic) methods. In the first case, the algorithms are basedon low level geometric information (e.g. curvature [20])in order to define segments (i.e. regions) with respect togeometric homogeneity, while in the latter case, the algo-rithms aim at distinguishing segments that correspond torelevant features of the shape, by following higher level no-tions such as defined in human perception theory [6]. Thiskind of approach is particularly suited for object anima-tion/deformation and indexing applications, where the de-composition has to be meaningful.

Although development of mesh segmentation algorithmsfor both approaches has drawn extensive and consistentattention, relatively little research has been done on seg-mentation evaluation. For the first approach (surface-type),some tools exist depending on the end application as texturemapping [23] or medical imaging [13]. Recently, two mainworks, Benhabiles et al. [4] (our previous work) and Chen etal. [7], have been proposed to study the quality assessmentproblem of part-type 3D-mesh segmentation. Both workspropose a benchmark for segmentation evaluation which is

H. Benhabiles et al.

based on a ground-truth corpus. The corpus is composed ofa set of 3D-models grouped in different classes and associ-ated with several manual segmentations produced by humanobservers. These two benchmarks comprise the ground-truthcorpus and a set of similarity metrics, then the evaluation ofa segmentation algorithm consists in measuring the similar-ity between the reference segmentations from the corpus andthat obtained by this algorithm (on the same models). In thiskind of benchmark the quality of the evaluation depends onthe quality of the corpus but also on the quality of the seg-mentation similarity measure. This leads to conclude thatthe choice of an accurate measure is quite critical in orderto provide a strict evaluation and to reflect the real qualityof an automatic segmentation with comparison to a manualone. In this context, less efforts were investigated to proposea reliable measure of mesh segmentation similarity. Indeed,the previous works [4, 7] focused their interests on the de-sign of the ground-truth corpus and presented rather simplemetrics suffering from degeneracies and low discriminativepower.

In this context the objective of the present work is to eval-uate the existing metrics and to propose a new one which ismore reliable. This paper introduces three main contribu-tions. Firstly, we propose a thorough study and comparisonsof existing metrics addressing the assessment problem ofmesh segmentation, using a corpus of manually segmentedmodels. This corpus is an improved version of our previousone [4] and is available on-line.1 Secondly, we propose anew measure of segmentation similarity that allows one toquantify the consistency between multiple segmentations ofa model. We show that this new metric outperforms exist-ing ones in terms of properties and discriminative power. Toquantitatively compare the discriminative power of the met-rics, we have conducted subjective tests using a set of humanobservers. Thirdly, we apply this measure together with twocorpuses (our corpus and Chen et al.’s [7] corpus) to evaluatesix recent 3D-mesh segmentation algorithms.

This paper is organized as follows. In Sect. 2, we providea review of the state-of-the-art of segmentation evaluationand an analytic study of the measures that have been pro-posed in this context. In Sects. 3 and 4, we define a newobjective metric to perform a quantitative comparison be-tween a segmentation algorithm and a set of ground-truthsegmentations (of the same model). In Sect. 5, we presentour corpus which will be used for the experimental compar-ison of the metrics and for the evaluation of the segmentationalgorithms. In Sect. 6, we present an extensive experimen-tal comparison between our new metric and existing ones,then we analyze the discriminative power of this new metricusing subjective tests. In Sect. 7, we demonstrate the usabil-ity of our whole evaluation protocol through the evaluation

1http://www-rech.telecom-lille1.eu/3dsegbenchmark/.

of six recent segmentation methods. Section 8 concludes thepaper.

2 Related work

In this section, we firstly provide a review of the state-of-the-art of 2D-image and 3D-mesh segmentation evaluation.Indeed, the most significant works for the 3D-mesh segmen-tation evaluation [4, 7] are based on the same methodologyas that proposed in the 2D-image domain [21]. Secondly, wereview the measures that have been proposed in the contextof 3D-mesh segmentation evaluation, while analyzing theirproperties.

2.1 State-of-the-art of 2D-image and 3D-meshsegmentation evaluation

Several advanced works exist for the quality assessmentof 2D-image segmentation. Zhang et al. [30] offer a studyon the different methods proposed for this task. Accordingto them, the different methods can be classified into fivegroups:

– Analytical methods. They directly treat the segmentationalgorithms themselves by taking into account principles,requirements, utilities, complexity, etc. of algorithms. Us-ing analytical methods to evaluate segmentation algo-rithm avoids a concrete implementation of the algorithms.However, the real quality of these algorithms cannot beobtained by a simple analytical study.

– Subjective methods. They evaluate the segmentation al-gorithms in a subjective way in which the segmentationresults are judged by a human operator. Therefore, theevaluation scores can vary significantly from one humanevaluator to another since they do not have necessarilythe same standards for assessing the quality of a segmen-tation. Furthermore, the results can depend on the orderin which the human operator observes them. To minimizebias, such a method requires a large set of objects and alarge group of humans. Unfortunately, this kind of methodcannot be integrated in an automatic system.

– System level evaluation methods. This kind of methodsindicates if the characteristics of the results obtained bya segmentation algorithm are suited for the over-all sys-tem which uses this segmentation algorithm. However,this evaluation method is indirect. If the process whichfollows the segmentation generates better results, it doesnot necessarily mean that the segmentation results weresuperior, and vice versa.

– Empirical goodness or unsupervised methods. They eval-uate the performance of the algorithms by judging thequality of the segmented images themselves. To achievethis task, a set of quality criteria has to be defined. These

A comparative study of existing metrics for 3D-mesh segmentation evaluation

criteria are established according to human intuition aboutwhat conditions should be satisfied by an ideal segmenta-tion. However, it seems difficult to establish quantitativelythe quality of a segmentation only by using such an a pri-ori criteria.

– Empirical discrepancy or supervised methods. A set ofreference images presenting the ideal segmentation is firstof all built. This set of images, which can be manuallysegmented by experts of the domain, constitutes a ground-truth. The purpose is to measure the discrepancy betweenthese reference segmentations and that obtained by an al-gorithm to evaluate. So, these methods try to determinehow far a segmented image obtained by an algorithm isfrom one or several segmented images. A large discrep-ancy involves a large segmentation error and thus this in-dicates a low performance of the considered segmentationalgorithm.

The empirical discrepancy methods are the most popularfor 2D-image segmentation evaluation [21, 28]. Indeed theyseem to be the most suited for a quantitative evaluation asthe measures of quality can be numerically computed, andfor an objective evaluation thanks to the ground-truth.

Martin et al. [21] have proposed such a method to eval-uate image segmentation algorithms. They built a publiccorpus containing ground-truth segmentations produced byhuman volunteers for images of a wide variety of naturalscenes. They also defined a measure of segmentation simi-larity based on the computation of refinement error of a pixelbetween two segments (i.e. regions) containing this pixel.

In the 3D-domain, there exist some works proposing thequality assessment of segmentation in a specific context. Inthe MRI (Magnetic Resonance Imaging) field for example,Gerig et al. [13] propose a tool that quantifies the segmen-tation quality of 3D-images (volumetric images) includingdifferent shape distance metrics such as maximum Haus-dorff distance, and mean/median absolute distance betweenobject surfaces. For texture mapping, Sander et al. [23] in-troduce a metric based on the texture stretch induced by theparametrization of the segmented regions and allowing forthe evaluation of the segmentation quality.

More recently, Attene et al. [3] have proposed some cri-teria like the aspect of the boundaries (smoothness, length),the hierarchical/multi-scale properties, the robustness, thecomplexity and the number of parameters. However, thesecriteria rather judge some technical points than the real qual-ity of the techniques themselves, they rather fall in the em-pirical goodness methods. As raised by the authors, the mainproblem is that the objective quality of a segmentation of agiven model is quite difficult to define, since it depends onthe viewer’s point of view and knowledge.

Berretti et al. [5] have presented some experimental re-sults which are based on a ground-truth to validate theirown segmentation algorithm. However, the ground-truth is

not available on-line and according to the authors it containsvery simple 3D-models (surfaces of revolution, vases, etc.).

Lastly, we proposed a framework to study the assess-ment problem of 3D-mesh segmentation [4]. Another workproposed by Chen et al. [7] addresses the same task. Bothof these works propose a benchmark which is based ona ground-truth corpus of human segmented 3D-models, sothey both constitute empirical discrepancy methods; theevaluation of a segmentation algorithm is realized by quan-tifying the consistency between the reference segmentationsof the ground-truth corpus and those obtained by this algo-rithm on the same models using a set of similarity metricsthat we will detail in the next subsection.

2.2 Review and analytic study of mesh segmentationsimilarity metrics

In the following, we summarize the existing metrics used toevaluate 3D-mesh segmentation and check if they are reallyreliable in the context of 3D-mesh segmentation evaluation.A reliable measure of mesh segmentation similarity has topossess the following set of properties:

– No degenerative cases. The score’s measure must be pro-portional to the similarity degree between an automaticsegmentation and the ground-truth segmentations of thesame model. For example, an over-segmentation whereeach vertex (or face) is represented by a segment mustgive a very low value of similarity, since no ground-truthsegmentation can be represented in such a way.

– Tolerance to refinement. The segmentation performed bysome human observers can be coarse while the segmen-tation performed by others can be finer. However, theybasically remain consistent; the difference just lies in thelevel of refinement. Hence, a reliable segmentation mea-sure has to accommodate and to be invariant to these seg-mentation granularity differences.

– Cardinality independence. The measure must neither as-sume equal cardinality nor depend on this attribute. Thismeans that two segmentations to be compared can havedifferent numbers of segments and different sizes of seg-ments.

– Tolerance to cut boundary imprecision. The segmentboundaries are defined in a subjective way. Indeed, it ispossible that two volunteers define the same segment on amodel with a slight difference between boundaries, how-ever, from a semantic point of view, the segments remainsimilar. Hence, a reliable measure has to accommodatethis imprecision of cut boundaries.

– Multiple ground-truth. The measure has to be able tocompare one automatic segmentation with multipleground-truth (reference segmentations) for a given model,otherwise, providing multiple ground-truth in a bench-mark is useless. An alternative solution is to simply aver-age the similarity scores obtained between an automatic

H. Benhabiles et al.

segmentation and each manual segmentation (referencesegmentation), however, this may bias the result and notreally reflect how much an automatic segmentation agreeswith the multiple ground-truth.

– Meaningful comparison. The scores obtained by the mea-sure have to allow for a meaningful comparison betweendifferent segmentations of the same model and betweensegmentations of different models. For the first case (seg-mentations of the same model), the scores have to varyaccording to the segmentation quality, then, more the au-tomatic segmentation is similar to the ground-truth seg-mentations of the same model, and better the score is. Forthe second case (segmentations of different models), thescores have to indicate which kind of 3D-models is themost convenient to segment by an automatic algorithm.

Essentially, the measures used to evaluate 3D-mesh seg-mentation can be classified into three categories: bound-ary matching, region differencing and non-parametric testsbased measures.

In order to be able to formulate the above measures, weneed to define what is a mesh segmentation. We will usethis definition (according to Shamir [24]) for the remainderof this article.

Definition 1 Let M be a 3D-mesh, and R the set of meshelements which are the vertices vi or the faces fi of M .A segmentation S of M is the set of sub-meshes S ={M0, . . . ,Mk−1} induced by the partitioning of R into k dis-joint sub-sets of vertices or faces.

The three categories of measure are:

1. Boundary matching. This kind of measures compute themapping degree between the extracted region bound-aries of two segmentations. Chen et al. [7] proposedto use such a measure called Cut discrepancy. It mea-sures the distances between cuts, where each cut rep-resents an extracted region boundary. Let S1 and S2 betwo segmentations of a 3D-mesh M and C1, C2, their re-spective sets of points on the segment boundaries. LetdG(p1,C2) = min{dG(p1,p2),∀p2 ∈ C2} be the geo-desic distance from a point p1 ∈ C1 to a set of cuts C2.

The Cut discrepancy between S1 and S2 is then:

CD(S1, S2) = DCD(S1 ⇒ S2) + DCD(S2 ⇒ S1)

avgRadius

where, avgRadius is the average Euclidean distance froma point on the surface to centroid of the mesh, and DCDis a directional function defined as DCD(S1 ⇒ S2) =mean{dG(p1,C2),∀p1 ∈ C1}.

A value of 0 will indicates a perfect matching betweenS1 and S2. As observed by Chen et al. [7] the measure isundefined when the model has no cuts and decreases to

zero as more cuts are added to a segmentation. Hence itsuffers from a degenerative case (see Sect. 2.2). In addi-tion, it is not tolerant to refinement since for two segmen-tations that are perfect mutual refinements of each other,it can provide a large value. Moreover, for the unmatchedpoints, it is possible to change their locations randomlyand the measure will keep the same value. It is also nottolerant to imprecision of cut boundaries since it is basedon a geodesic distance. Finally, it allows one to comparean automatic segmentation to only one ground-truth seg-mentation.

2. Region differencing. These measures compute the consis-tency degree between the regions produced by two seg-mentations S1 and S2. Berretti et al. [5] have proposedan overlap index representing the extent to which a re-gion Ri of an automatic segmentation overlaps to closestregion Rj of a ground-truth segmentation. The overlapindex Oindex of Ri is defined as:

Oindex = maxj

A(Ri ∩ Rj)

A(Ri)

with A(.) the operator that returns the area of a region. Ifwe suppose that S1 is the automatic segmentation and S2

is the ground-truth segmentation, then the distance be-tween them is the average of the Overlap index over-all regions of S1. This measure falls in a degenerativecase when Ri is represented by one face. Then the over-partitioning is not captured and it also does not allow fora comparison to multiple ground-truth.

We (Benhabiles et al. [4]) and Chen et al. [7] proposedto use the consistency error measure. It is based on thecomputing of a local refinement error L3D of a vertex (orface) vi between S1 and S2 and is defined as:

L3D(S1, S2, vi) = |R(S1, vi)\R(S2, vi)||R(S1, vi)|

where the operator\denotes the set differencing, |x| thecardinality of the set x, and R(S, vi) the region in seg-mentation S that contains the vertex vi , i.e. the subset ofvertices corresponding to a sub-mesh Mj of S contain-ing vi .

This local refinement error produces a positive realvalued output that presents the ratio of the number of ver-tices not shared between the first segment and the secondone.

Given this L3D , there exist two ways to combine itfor all vertices into a global measure for the entire 3D-mesh: the Global Consistency Error (GCE) and the LocalConsistency Error (LCE).

A comparative study of existing metrics for 3D-mesh segmentation evaluation

The Global Consistency Error (GCE) forces all localrefinements to be in the same direction and is defined as:

GCE(S1, S2)

= 1

Nmin

{∑i

L3D(S1, S2, vi),∑

i

L3D(S2, S1, vi)

}

The Local Consistency Error (LCE) allows for differentdirections of refinement in different segments of the 3D-mesh:

LCE(S1, S2)

= 1

N

∑i

min{L3D(S1, S2, vi),L3D(S2, S1, vi)

}

where N is the number of vertices. For both the GCEand the LCE, a value of 0 indicates a complete similar-ity, whereas a value of 1 indicates a maximum deviationbetween the two segmentations being compared. Thereare two degenerative segmentations that achieve a GCEand a LCE score of zero: one vertex per segment, andone segment for the entire mesh. We can also notice thatthe measure does not allow for a comparison to multipleground-truth.

Chen et al. [7] proposed to use another measurenamely Hamming distance. The Hamming distance be-tween two segmentations S1 and S2 measures the regiondifferencing between their respective set of segments.The directional Hamming distance is defined as:

DH (S1 ⇒ S2) =∑

i

∥∥Ri2\Rit

1

∥∥

where the operator\denotes the set differencing, ‖x‖ thecardinality of the set x, and it = argmaxk‖Ri

2 ∩ Rk1‖

which allows one to find the closest segment in S1 to theregion (or segment) Ri

2 in S2.Given this DH , and considering S2 as the ground-

truth, the authors of [7] defined the missing rate Mr andthe false alarm rate Fr as follows:

Mr(S1, S2) = DH (S1 ⇒ S2)

‖S‖Fr(S1, S2) = DH (S2 ⇒ S1)

‖S‖and the Hamming distance as the average of missing rateand false alarm rate:

HD(S1, S2) = 1

2

(Mr(S1, S2) + Fr(S1, S2)

)

As observed by the authors [7] the measure has a goodbehavior when the correspondences between segmentsare correct but it fails when they are not. Another limitis the comparison to only one ground-truth.

3. Non-parametric tests. In the statistical literature there ex-ists a lot of non-parametric measures. We can cite for ex-ample Cohen’s Kappa [8], Jaccard’s index [11], Fowlkesand Mallow’s index [11]. The latter two are variants ofRand index [22]. Chen et al. [7] proposed to use Randindex for 3D-mesh segmentation evaluation. This indexconverts the problem of comparing two segmentations S1

and S2 with different numbers of segments into a prob-lem of computing pairwise label relationships. If we de-note liS1

the corresponding label of all elements (vertices

or faces) contained in region Ri of S1 and similarly liS2the corresponding label of all elements contained in re-gion Ri of S2, the Rand index can be computed as theratio of the number of pairs of vertices or faces havingthe compatible label relationship in S1 and S2 and can bedefined as:

RI(S1, S2) = 1(N

2

) ∑i,ji<j

I(liS1

= ljS1

)(liS2

= ljS2

)

+ I(liS1

�= ljS1

)(liS2

�= ljS2

)

where I is the identity function, and the denominator isthe number of possible unique pairs among N vertices orfaces. This gives a measure of similarity ranging from 1,when the two segmentations are identical, to 0 otherwise.This measure does not allow for comparison to multipleground-truth segmentations.

We can notice that all existing measures suffer fromeither degenerative cases and/or sensitivity to refinementand/or sensitivity to cut boundaries imprecision and/or lim-itation in term of comparison to multiple reference (i.e.ground-truth) segmentations. Therefore none of these mea-sures satisfies the whole set of defined criteria.

3 The 3D probabilistic rand index (3D-PRI)

The goal of this measure is to perform a quantitative com-parison between a mesh segmentation algorithm and a set ofground-truth segmentations (of the same model). In the fieldof 2D-image, Unnikrishnan et al. [28] proposed a probabilis-tic interpretation of Rand Index to evaluate the performanceof 2D-image segmentation algorithms and shown the rele-vance of the obtained results. Hence we have generalizedthis measure for 3D-mesh segmentation evaluation.

Let Sa be the automatic segmentation to be compared to aset of manual segmentations (ground-truth) {S1, S2, . . . , SK}of a 3D-mesh M . We denote the corresponding label of avertex vi (label of the segment to which belongs vertex vi )by liSa

in segmentation Sa and by liSkin the ground-truth seg-

mentation Sk . It is assumed that the label liSktakes a value

ranged between 1 and the number of segments of Sk and

H. Benhabiles et al.

similarly liSatakes a value ranged between 1 and the num-

ber of segments of Sa . The label relationships for each ver-tex pair is modeled by an unknown underlying distribution.This can be considered as a process where each human seg-menter provides information about the segmentation Sk ofthe 3D-mesh in the form of binary numbers I(lisk = l

jsk ) for

each pair of vertices (xi, xj ). The set of all perceptually cor-rect segmentations defines a Bernoulli distribution over thisnumber, giving a random variable with expected value de-noted as pij . Hence, the set {pij } for all unordered pairs(i, j) defines a generative model of correct segmentationsfor the 3D-mesh M . The 3D Probabilistic Rand Index is thendefined as:

3DPRI(Sa, {Sk}

) = 1(N

2

) ∑i,ji<j

eijpij + (1 − eij )(1 − pij ) (1)

where eij denotes the event of a pair of vertices i and j

belonging to the same segment (or region) in the automaticsegmentation:

eij = I(liSa

= ljSa

)

and pij denotes the probability of the vertices i and j be-longing to the same segment in the ground-truth set {Sk} andis given by the sample mean of the corresponding Bernoullidistribution as suggested by Unnikrishnan et al. [28]:

pij = 1

K

∑k

I(liSk

= ljSk

)

The 3D-PRI takes a value ranged between 0 and 1, where0 indicates no similarity between Sa and {S1, S2, . . . , Sk},and 1 indicates a perfect similarity.

Note that with this formulation for pij , computing the3D-PRI is equivalent to averaging the RI over the multipleground-truths. However, the 3D-PRI formulation is genericand we can imagine a different and more efficient way tocompute the pij . The main advantage of the simple meanestimator is its fast computation.

We have noticed in practice, however, that the 3D-PRIsuffers from lack of discriminative power in its values. In-deed, the values obtained by the index do not allow one toclearly decide if a segmentation obtained by an automatic al-gorithm is relevant or not. This is due to the limited effectiverange of 3D-PRI in term of maximum and minimum value.To address this drawback, we present in the next section, the3D normalized probabilistic Rand index (3D-NPRI).

4 3D normalized probabilistic rand index (3D-NPRI)

Our objective is to normalize the 3D-PRI, in order to in-crease its dynamic range and thus its discriminative power.

Hence we need to define a baseline to which the index canbe expressed. For 3D-mesh segmentations, the baseline maybe interpreted as the expected value of the index under someparticular segmentations of the input 3D-model. A popularstrategy [11, 28] of index normalization with respect to itsbaseline is:

Normalized index

= Index − Expected index

Maximum index − Expected index(2)

As observed by Unnikrishnan et al. [28] there is a littleagreement in the statistics community regarding whether thevalue of “Maximum index” should be estimated from thedata or set constant. We choose to follow what was doneby Unnikrishnan et al. [28] and set the value to be 1 (themaximum possible value of the 3D-PRI). Thus, we avoid thepractical difficulty of estimating this quantity for complexdata sets.

Another parameter to define is the expected probabilisticRand index E(3D-PRI). One may draw an analogy betweenthe E(3D-PRI) and the 3D-PRI in equation 1 as follows:

E[3DPRI

(Sa, {Sk}

)] = 1(N

2

) ∑i,ji<j

e′ijpij + (1 − e′

ij )(1 − pij )

(3)

where e′ij = E[I(liSa

= ljSa

)]. This latter quantity has to becomputed in a meaningful way. Unnikrishnan et al. [28] pro-posed to estimate it from segmentations of all images of thedatabase for all unordered pairs (i, j). Let Φ be a numberof images in a data set and Kφ the number of ground-truthsegmentations of image φ. Then, e′

ij is expressed as:

e′ij = 1

Φ

∑φ

1

Kφ

Kφ∑k=1

I(liSk

φ

= lj

Skφ

)

However, this estimation can only be used in a data-baseof 2D-images having equal sizes (where each pixel has itscorrespondent over all the other segmented images). In the3D case, it is not possible, since the different models ofthe corpus have different number of vertices and differentconnectivities. One possible way to compute the E(3D-PRI)while keeping a correct baseline and without having anyconstraint on the corpus, is to use random segmentations Sr :

E[3DPRI

(Sa, {Sk}

)] = 1

N

N∑r=1

3DPRI(Sr, {SKr }

)(4)

where N is the number of 3D-models in our corpus and {Skr }are ground-truths of the model concerned by Sr . We then

A comparative study of existing metrics for 3D-mesh segmentation evaluation

Fig. 1 Random segmentations of some 3D-models of the corpus

define the 3D-NPRI of an automatic segmentation of a given3D-model as follows:

3DNPRI(Sa) = 3DPRI(Sa, {SK}) − E[3DPRI(Sa, {Sk})]1 − E[3DPRI(Sa, {Sk})]

(5)

The random segmentations were generated using a simplealgorithm: L seed vertices were randomly chosen on the ob-ject, then L connected regions were obtained by a simpleregion growing mechanism. The number of segments (or re-gions) takes a value ranged between 2 and the number ofvertices of the concerned model. Figure 1 shows some 3D-models of the corpus on which the random segmentation al-gorithm was applied. We have to precise here that the 3D-NPRI is not affected by the choice of these random segmen-tations. Indeed we will show later (see Fig. 3) that the 3D-PRI provides very stable values when comparing ground-truth segmentations to random segmentations (even withvery different granularities) hence the normalization con-stant E(3D-PRI) (see (4)) is almost invariant to the choiceof the random segmentations Sr .

Hence, the 3D-NPRI will take a value with a lower boundof −1 and an upper bound of 1, where −1 indicates no sim-ilarity between the automatic segmentation and the ground-truth segmentations of the same model, and 1 indicates aperfect match. The lower bound of −1 is explained by thefact that the expected Index cannot exceed 0.5 since we com-pare a set of random segmentations to a set of ground-truthsegmentations (see Sect. 6.1). Therefore, the worst case willbe:

3DNPRI(Sa) = 0 − 0.5

1 − 0.5= −1

where the automatic segmentation has no similarity with itscorresponding ground-truths.

Note that the metric’s definition does not take into ac-count model with different sampling. Moreover, the score ofthe metric changes by changing the order of vertices on theautomatic segmentation and the ground-truths of the samemodel. However, in our case, it is not really a drawbacksince we compare segmentations of the same model whilekeeping the same sampling and the same order of vertices.

5 Ground-truth corpus

The current version of our corpus is an improved ver-sion of [4] in term of number of models and ground-truthsegmentations per model. The corpus is available on-line2

and contains twenty-eight 3D-models (as triangle meshes)grouped in five classes, namely animal, furniture, hand, hu-man and bust. Each 3D-model of the corpus is associatedwith 4 manual segmentations which give a total of 112ground-truth segmentations done by 36 volunteers. Figure 2illustrates the models of the corpus with one manual seg-mentation per model. We have selected a small number ofvaried models with respect to a set of properties. All theselected models are manifold, connected, and do not haveintersecting faces. Hence they are supported as an input byany segmentation algorithm. In order to collect precise man-ual segmentations, we have assisted the volunteers in trac-ing the vertex-boundaries through the different models. Notethat the volunteers have freely segmented the models and nocondition was imposed on the manner with which they havesegmented them. For this task, we used MeshLab3 applica-tion allowing for an explicit vertex-per-vertex segmentationof models using colors.

Chen et al. [7] proposed another corpus that seems com-plementary to ours: they present more objects (380 3D-models of the Watertight Track of the 2007 SHREC Shape-based Retrieval Contest [14]) when we selected a smallrepresentative set (it allows one to rapidly evaluate a seg-mentation algorithm without running it on 380 objects).They chose to use the web application Amazon’s Me-chanical Turk4 to collect the manual—i.e. ground-truth—segmentations without any supervision when we chose tosupervise our volunteers to obtain more precise manual seg-mentations. Finally, their ground-truth presents face-basedsegmentations whereas ours contains vertex-based segmen-tations.

6 Experimental comparison of properties of existingsegmentation similarity metrics

In what follows, we provide an experimental study of the3D-PRI/3D-NPRI properties and we compare them to theexisting metrics for assessing 3D-mesh segmentation qual-ity. For this end, we use our corpus models and their corre-sponding ground-truths.

Most of the measures introduced in Sect. 2.2 quantifydissimilarity (the lower is the number, the better is the seg-mentation result) between segmentations rather than simi-larity. In order to have a meaningful comparison between

2http://www-rech.telecom-lille1.eu/3dsegbenchmark/.3http://meshlab.sourceforge.net/.4http://www.mturk.com/.

H. Benhabiles et al.

Fig. 2 Models of our corpus associated with one ground-truth

these measures and the 3D-PRI/3D-NPRI, we define thequantities CDI(S1, S2) = 1 − CD(S1, S2), GCI(S1, S2) =1 − GCE(S1, S2), LCI(S1, S2) = 1 − LCE(S1, S2), andHDI(S1, S2) = 1 − HD(S1, S2). The “I” in the acronymsstands for “Index”, complying with the popular usage of theterm in statistics when quantifying similarity. Hence, exceptthe CDI, all of the other indexes are in the range [0,1] with avalue of 0 indicating no similarity between segmentations ofthe same model and a value of 1 indicating a perfect match.The CDI is in the range ]−∞,1].

6.1 Sensitivity to degenerative cases

The first property to study is the sensitivity of each index re-garding degenerative cases. For this end, we compare ourProbabilistic Rand Index (3D-PRI) with the Cut Discrep-ancy Index (CDI), the Hamming Distance Index (HDI), theGlobal and Local Consistency Index (GCI/LCI), and theOverlap Index (OI) for three kinds of random segmentationsnamely extreme-low segmentation (segmentation composedof a 2 or 3 segments), middle-segmentation (segmentationcomposed of a number of segments which is similar to thatof ground-truths of the corresponding model), and extreme-high segmentation (segmentation composed of more than 50

Fig. 3 Comparison of three levels of random segmentation (ex-treme-low, middle, and extreme-high) to the ground-truths for thewhole corpus using different indexes

segments). They were generated using a random segmen-tation algorithm. Figure 3 presents the results obtained bythe comparison of these random segmentations to the set ofthe ground-truths for each model of the corpus. Each indexof the figure is computed for the three kinds of segmenta-tion (extreme-high segmentation, middle-segmentation, andextreme-low segmentation) and averaged across the entiredata set. Since the segmentations are random, the scores ob-tained by the metrics are expected to be low for the threekinds of segmentation, and it is the case for the 3D-PRI.We can notice, however, that although the random segmenta-tions are totally different from the ground-truths, the scoresof the other metrics are very high (very good) for certainsegmentations with degenerative granularity (extreme-highand/or extreme-low). Hence the 3D-PRI is the most stableregarding degenerative cases considering its scores, whichare less than 0.32.

6.2 Tolerance to refinement

The second property to study is the tolerance of each in-dex to refinement. For this end, we perform two kinds ofexperiments. The first one uses segmentations with mutualrefinements, and the second one uses segmentations with hi-erarchical refinements. The obtained results for the first ex-periment are presented in Fig. 4.

It shows two segmentations of the dinopet model whichare perfect mutual refinements of each other, and a plot inwhich is computed the similarity between the two segmen-tations using different metrics. The plot of Fig. 4 clearlyshows that the CDI fails to capture the similarity betweenthe two segmentations (a) and (b). Although the two seg-mentations are similar (the difference just lies in the levelof refinement). However, the other metrics have a good be-havior toward this kind of refinement since all of them givescores which are close to 1.

A comparative study of existing metrics for 3D-mesh segmentation evaluation

Fig. 4 Tolerance to mutual refinement of different indexes, by com-paring two segmentations (a), (b) with perfect mutual refinement forthe dinopet model

The second experiment was performed using the hierar-chical segmentation algorithm of Attene et al. [2]. We gen-erated several levels of segmentation (from 4 segments to 15segments) on the horse model of our corpus then we com-pared these 12 versions to the ground-truths. Figure 5 illus-trates the obtained results using different indexes. The OIand the GCI does not appear on the figure since they havethe same behavior as the LCI. The figure clearly shows thatthe CDI is less stable toward hierarchical refinement than theother indexes. The LCI seems completely invariant while the3D-PRI and the HDI present a slight variation; they are notfully invariant but present a good tolerance to refinement.

6.3 Independence from cardinality

The third property to study is the independence of each in-dex toward segmentation cardinality. According to the pre-vious performed experiments about the first two properties(degenerative cases and refinement), the CDI seems to bethe only metric which depends on the cardinality, in a criti-cal way. Indeed, the comparison between two segmentationswith different number of segments will give a bad score us-ing this metric whatever the quality.

Fig. 5 Tolerance to hierarchical refinement of different indexes, bycomparing several levels of segmentation of the horse model to its cor-responding ground-truths

6.4 Tolerance to imprecision of cut boundaries

The fourth property to study is the tolerance of each indexto the imprecision of cut boundaries. For this end, we manu-ally segmented a simple model (bitorus) into 2 segments. Weproposed 5 segmentations (Fig. 6(a) to (e)) where each oneof them has a slight difference in the boundary position withcomparison to the others, then we computed the similaritybetween segmentation (c) and the other segmentations. Theplot in Fig. 6 shows the obtained results using different in-dexes. Contrary to the other indexes, the CDI gives low val-ues of similarity between segmentations. Although the CDIis not in the same range as the other metrics, the plot still al-lows us to illustrate the qualitative behavior of this latter in-dex toward the imprecision of cut boundaries. We can noticealso that except the 3D-PRI which presents a slight variationbut a good tolerance, the other indexes are almost invariant.

At this point, we have shown that the 3D-PRI satisfies thefive properties: ability to compare one automatic segmenta-tion with multiple ground-truth segmentations, no degener-ative cases, tolerance to refinement, independence from seg-mentation cardinality, and tolerance to imprecision of cutboundaries. We also have shown that the 3D-PRI outper-forms the other indexes in terms of the first two properties.We show in the next experiments that the normalization ofthis index (into 3D-NPRI) improves its discriminative powerand give better results in term of meaningful comparison.

6.5 Meaningful comparison

The main advantage of the 3D-NPRI is the ability to providevalues that allow a meaningful comparison between seg-mentations of different 3D-models. Figure 7 demonstratesthis behavior. The top two rows show different 3D-modelsof our corpus segmented at different granularity using the

H. Benhabiles et al.

Fig. 6 Tolerance to imprecision of cut boundaries of different indexes,by comparing segmentation (c) to segmentations (a) to (e) for thebitorus model

hierarchical algorithm of Tierny et al. [27]. These automaticsegmentations are compared to the ground-truth corpus (seeFig. 2) using the previous indexes and our 3D-NPRI. Visu-ally, regarding the ground-truth, segmentations (a) and (b)(Fig. 7) seem very poor, segmentations (c), (d), and (f) arecorrect, and segmentation e is perfect. One can notice thatthe OI similarity is high for all of the 3D-models. Hence, itcannot indicate which segmentation is the best. Note that al-though the HDI gives lower scores than the OI, it also failsto distinguish between correct and poor segmentations sinceit gives high values for poor ones (Fig. 7(a) and (b) and lowvalues for correct ones (Fig. 7(c) and (d). The GCI/LCI doesnot appear in the plot in order to keep a clear display. Thislatter metric has the same behavior than HDI. The CDI hasslightly a better behavior than HDI but still to fail distin-guishing between correct and poor segmentations. The 3D-PRI reflects the correct relationship among the segmenta-tions. However, its range is small, and the expected value isunknown, hence it is difficult to determine which segmen-tation is really good. The 3D-NPRI fixes all of these draw-backs. It reflects the desired relationship among the segmen-tations with no degenerate cases. Besides, any segmentationwhich gives a score significantly above 0 can be consideredas relevant (since it provides results significantly better thanrandom segmentations).

Fig. 7 Example of comparing segmentations of different models:From (a) to (f) segmentations using algorithm from [27]. The plot (g)shows the scores of different indexes for each segmentation (a) to (f)

6.6 Discriminative power

The best way to attest the discriminative power of our 3D-NPRI is to show that its values are well correlated with therates given by users for a set of segmentations.

In the following experiment, we study the correlation be-tween the metrics’ values and the observers’ rates of a setof segmentations obtained from our corpus models. For thisend, we used the following algorithms: Attene et al. [2],Lavoué et al. [20], Shapira et al. [25], and Tierny et al [27].We collected 250 segmentations including 28 ground-truthsegmentations and 28 random segmentations. Except for thealgorithm of Lavoué et al. [20], the others are hierarchical.Hence, we generated for each one of them two levels ofsegmentation, namely coarse and fine, which gives 28 × 2segmentations per algorithm and 28 segmentations fromLavoué et al.’s [20] algorithm. We computed the quality in-dex of these 250 segmentations (using our ground-truth) us-ing the different metrics. We then asked several observers togive a rate reflecting the perceived quality of each segmen-

A comparative study of existing metrics for 3D-mesh segmentation evaluation

Fig. 8 Subjective MOS vs metric values for the whole corpus models and for different metrics. Each circle represents a segmentation. TheGaussian fitted curve is displayed in red

tation between 1 (bad segmentation) and 10 (perfect seg-mentation). Hence each segmentation was associated withquality index values from the different metrics and a sub-jective Mean Opinion Score (MOS) from human observers.This latter score reflects the opinion of observers toward thequality of a segmentation. The MOS of the segmentation i

is computed as follows:

MOSi = 1

N

N∑j=1

mij

where N is the number of observers (10 in our experiment),and mij is the rate (in the range [1,10], 10 for a very goodsegmentation) of the j th observer given to the ith segmenta-tion. For the correlation, we considered a statistical indicatornamely the Pearson Product Moment Correlation [10]. Thisindicator measures the linear dependence between two vari-ables X and Y. In order to optimize the matching between thevalues of the different metrics and the MOS of observers, we

Table 1 Pearson correlation values (%) between the Mean OpinionScores and the values of different metrics for each model category ofour corpus

CDI OI GCI LCI HDI 3D-NPRI

Animal 2.6 2.3 9.3 8.3 16.9 58.7

Bust 10.9 0 45.9 61.1 54.8 77.4

Furniture 5.8 14.8 49.9 50.5 63 73.2

Hand 21.2 1 54.1 54.4 57.5 70.2

Human 1.5 5.5 32.1 32.6 39 51.6

Whole 7.1 2.6 23.7 20.9 32.9 66.1

performed a psychometric curve fitting using the Gaussianpsychometric function (recommended by [9]).

Table 1 shows the results of correlation between the val-ues of different metrics and the MOS of observers for Pear-son indicator. The results in the table clearly shows that the3D-NPRI outperforms the other metrics in term of correla-

H. Benhabiles et al.

Table 2 Properties of existingsimilarity metrics CDI OI GCI LCI HDI 3D-NPRI

Degenerative cases Yes Yes Yes Yes Yes No

Tolerance to refinement No Yes Yes Yes Yes Yes

Cardinality independence No Yes Yes Yes Yes Yes

Tolerance to cut imprecision No Yes Yes Yes Yes Yes

Multiple ground-truth No No No No No Yes

Meaningful comparison No No No No No Yes

Strong discriminative power No No No No No Yes

tion for each category and for the whole corpus. Moreover,the Pearson correlation value of the 3D-NPRI for the wholecorpus is high (66.1%), when those of the other metrics arequite bad (less than 33%). This means that except the 3D-NPRI, the other metrics fail to distinguish between goodand bad segmentations. Figure 8 presents the psychomet-ric curve fitting between the objective and subjective scoresfor 3D-NPRI, HDI, LCI and CDI for 250 segmentationsof the corpus models. It visually illustrates the superiorityof the 3D-NPRI for predicting the subjective opinion, andleads to conclude that the 3D-NPRI has the best discrim-inative power. These results clearly validate the 3D-NPRI,since they are in agreement with the human opinion.

The properties of each metric are summarized in Table 2according to the performed experiments in this section.

7 Application for the evaluation of recent segmentationalgorithms

In this section, we apply the 3D-NPRI together with theChen et al.’s [7] corpus and our corpus (described in Sect. 5)to evaluate a set of recent automatic segmentation algo-rithms, then we compare the obtained results by the twocorpuses. We have considered the six recent automatic seg-mentation algorithms used in Chen et al. [7]: Attene etal. [2], Lai et al. [19], Golovinskiy et al. [15], Katz etal. [18], and Shapira et al. [25]. The six algorithms are re-spectively based on: fitting primitives, random walks, nor-malized cuts/randomized cuts, core extraction, and shape di-ameter function. The segmentations using these algorithmsfor the Chen’s corpus are available on-line. On the otherhand, we used Attene et al.’s [2], and Shapira et al.’s [25]algorithms (the only algorithms available on-line among theprevious six) to generate automatic segmentations on ourcorpus models. The reader can refer to the original papersfor more details about the six algorithms.

Note that all the algorithms cited above are part-typehierarchical segmentation methods. Hence for each one ofthem we can generate several levels of segmentation. Chenet al. [7] provided only one level of segmentation for eachalgorithm applied on their corpus. For this end, they used

Fig. 9 Scores of 3D-NPRI sorted in increasing order over all the twocorpus models

the parameter settings recommended by the authors of thealgorithms. To keep a valid comparison between the two cor-puses, we also used the parameter settings recommended bythe authors of the algorithms to generate segmentations onour corpus models. Note that the level of segmentation willnot influence the evaluation results since we proved that the3D-NPRI is tolerant to hierarchical refinement (see Fig. 5).

A comparative study of existing metrics for 3D-mesh segmentation evaluation

Table 3 Algorithms ranking applied on respectively the Chen’s corpusand our corpus

Algorithm 3D-NPRI mean Rank

Fitting primitives 0.49/0.49 5/2

Random walks 0.50/– 4/–

Normalized cuts 0.59/– 2/–

Randomized cuts 0.63/– 1/–

Core extraction 0.46/– 6/–

Shape diameter function 0.56/0.55 3/1

To ensure a relevant comparison between the algorithms,we compute the 3D-NPRI for every 3D-model of the Chen’scorpus and of our corpus. Figure 9 shows the 3D-NPRI foreach model of the two corpuses and for each algorithm.The values are sorted in increasing order for each algorithm,hence the j th model may not be the same across algorithms.This kind of graph was already applied for segmentationevaluation in the field of 2D-image [28].

Table 3 presents the rank of each algorithm together withthe 3D-NPRI mean value over all the two corpuses.

Table 3 and Fig. 9 demonstrate, as expected, that the seg-mentations obtained by the six algorithms are relevant sincemost of the values of the 3D-NPRI are greater than zero.The Randomized Cut algorithm seems to provide the bestresults. It is very interesting to notice that the Fitting Primi-tives and Shape Diameter keep similar behavior for the twocorpuses although these two corpuses are very different: theprofiles of the 3D-NPRI distribution (see Fig. 9) and themean 3D-NPRI values (see Table 3) for these algorithms arealmost exactly the same for both corpuses. Hence it validatesthe fact that our corpus, since it presents high quality manualsegmentation and heterogeneous models, is clearly efficientfor segmentation evaluation despite its small size. Anotherinteresting experiment is to see which category models thealgorithms fail to segment well. For this end, we average the3D-NPRI for each category of the two corpuses. Figures 10and 11 illustrate the obtained results for the six algorithms.One can notice that whatever the corpus is, there is no algo-rithm that is reaching the highest scores for all categories.Moreover, each algorithm has at least one category inade-quately segmented since its mean 3D-NPRI value is verylow (close to 0 or less). The core extraction algorithm forinstance fails to adequately segment the Bearing and Mechcategories (see Fig. 11(e)). This result is straight since theconcerned algorithm is a part-based one. Indeed, it tries todetect the core of a model which from a semantic point ofview is hard to define in such categories. As observed byChen et al. [7], some algorithms do not necessarily segmentthe best (with comparison to others) categories for whichthey were designed. We can notice this behavior on our cor-pus too. For instance, the algorithm based on Fitting Primi-

Fig. 10 Scores of 3D-NPRI averaged for each category models of ourcorpus

tives gives greater 3D-NPRI score (better) for the hand cat-egory than the algorithm based on Shape Diameter Functionand vice versa for the furniture category. As raised by Chenet al. [7], this means that either the human observers do notsegment models in the expected way, or the part structuresof these models are revealed by other properties.

Our results and those of Chen et al. [7] are coherent. Thisis straight since our metric is a probabilistic interpretation ofthe Rand Index (metric used by Chen et al. [7] to analyze andevaluate the algorithms) to which we added a normalizationallowing a better results analysis.

8 Conclusion

This paper presents a thorough comparison between exist-ing similarity metrics and a new one addressing the assess-ment problem of mesh segmentation. For this end we usea corpus of manually segmented models. This corpus is animproved version of our previous one [4] and is available on-line. The new 3D-NPRI metric is a probabilistic interpreta-tion of the Rand Index which allows to quantify the consis-

H. Benhabiles et al.

Fig. 11 Scores of 3D-NPRI averaged for each category models of the Chen’s corpus

tency between multiple segmentations of a 3D-mesh model.The paper shows that this new metric outperforms existingones in terms of properties and discriminative power. The re-sults are validated by comparing subjective scores of humanobservers to the objective metric scores. Finally, the mea-sure is applied together with the Chen et al.’s [7] corpus andour corpus to evaluate six recent 3D-mesh segmentation al-

gorithms. This evaluation allowed to compare the obtainedresults depending on the corpus and showed their coherence.

For future work, we plan to explore other kind of esti-mator to compute the pij (see (1) in Sect. 3) in order toimprove the correlation between metric’s scores and the ob-serves’ scores, we also plan to enrich our subjective testsby integrating more experiments allowing to compare algo-

A comparative study of existing metrics for 3D-mesh segmentation evaluation

rithms. Finally, we plan to exploit our ground-truths to de-sign a learning segmentation algorithm.

Acknowledgements We would like to thank Marco Attene, ArielShamir, Shy Shalom, and Julien Tierny for providing us the sourcecode or the binary of the segmentation algorithms and Fatan Souhaibfor implementing the 3dsegbenchmark site. We also would like tothank Xiaobai Chen for providing on-line the manual and auto-matic segmentations of different algorithms. We thank AIM@SHAPE,GAMMA-INRIA, and Princeton Shape Benchmark databases for pro-viding 3D-mesh models.

This work is supported by the ANR (Agence Nationale de laRecherche, France) through MADRAS project (ANR-07-MDCO-015).

References

1. Antini, G., Berretti, S., Pala, P.: 3d mesh partitioning for retrievalby parts application. In: IEEE International Conference on Multi-media & Expo (ICMEÆ05) (2005)

2. Attene, M., Falcidieno, B., Spagnuolo, M.: Hierarchical mesh seg-mentation based on fitting primitives. Vis. Comput. 22(3), 181–193 (2006)

3. Attene, M., Katz, S., Mortara, M., Patané, G., Spagnuolo, M.,Tal, A.: Mesh segmentation, a comparative study. In: IEEE In-ternational Conference on Shape Modeling and Applications, p. 7(2006)

4. Benhabiles, H., Vandeborre, J.P., Lavoué, G., Daoudi, M.:A framework for the objective evaluation of segmentation algo-rithms using a ground-truth of human segmented 3d-models. In:IEEE International Conference on Shape Modeling and Applica-tion (SMI) (2009)

5. Berretti, S., Bimbo, A.D., Pala, P.: 3d mesh decomposition usingReeb graphs. Image Vis. Comput. 27(10), 1540–1554 (2009)

6. Biederman, I.: Recognition-by-components: a theory of humanimage understanding. Psychol. Rev. 94, 115–147 (1987)

7. Chen, X., Golovinskiy, A., Funkhouser, T.: A benchmark for3d mesh segmentation. ACM Trans. Graph. (SIGGRAPH) 28(3)(2009)

8. Cohen, J.: A coefficient of agreement for nominal scales. Educ.Psychol. Meas. 37–46 (1960)

9. Corsini, M., Gelasca, E.D., Ebrahimi, T., Barni, M.: Watermarked3d mesh quality assessment. IEEE Trans. Multim. 9, 247–256(2007)

10. Daniel, W.W.: A Foundation for Analysis in the Health SciencesBooks, 7th edn. Wiley, New York (1999)

11. Fowlkes, E.B., Mallows, C.L.: A method for comparing two hier-archical clusterings. J. Am. Stat. Assoc. 78(383), 553–569 (1983)

12. Funkhouser, T., Kazhdan, M., Shilane, P., Min, P., Kiefer, W., Tal,A., Rusinkiewicz, S., Dobkin, D.: Modeling by example. ACMTrans. Graph. (Proc. SIGGRAPH) (2004)

13. Gerig, G., Jomier, M., Chakos, A.: Valmet: A new validation toolfor assessing and improving 3d object segmentation. In: MIC-CAI 2001: Fourth International Conference on Medical ImageComputing and Computer-Assisted Intervention, pp. 516–523.Springer, Berlin (2001)

14. Giorgi, D., Biasotti, S., Paraboschi, L.: Shrec: shape retrievalcontest: watertight models track. http://watertight.ge.imati.cnr.it/(2007)

15. Golovinskiy, A., Funkhouser, T.: Randomized cuts for 3d meshanalysis. ACM Trans. Graph. 27(5) (2008)

16. Ji, Z., Liu, L., Chen, Z., Wang, G.: Easy mesh cutting. Comput.Graph. Forum 25(3), 283–291 (2006)

17. Katz, S., Tal, A.: Hierarchical mesh decomposition using fuzzyclustering and cuts. ACM Trans. Graph. (SIGGRAPH) 22(3),954–961 (2003)

18. Katz, S., Leifman, G., Tal, A.: Mesh segmentation using fea-ture point and core extraction. Vis. Comput. 21(8–10), 649–658(2005)

19. Lai, Y.K., Hu, S.M., Martin, R.R., Rosin, P.L.: Fast mesh segmen-tation using random walks. In: SPM ’08: Proceedings of the 2008ACM Symposium on Solid and Physical Modeling (2008)

20. Lavoué, G., Dupont, F., Baskurt, A.: A new cad mesh segmenta-tion method, based on curvature tensor analysis. Comput. AidedDes. 37(10), 975–987 (2005)

21. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of humansegmented natural images and its application to evaluating algo-rithms and measuring ecological statistics. Int. Conf. Comput. Vis.2, 416–423 (2001)

22. Rand, W.M.: Objective criteria for the evaluation of clusteringmethods. J. Am. Stat. Assoc. 66(336), 846–850 (1971)

23. Sander, P.V., Snyder, J., Gortler, S.J., Hoppe, H.: Texture mappingprogressive meshes. In: SIGGRAPH 2001, pp. 409–416 (2001)

24. Shamir, A.: A survey on mesh segmentation techniques. Comput.Graph. Forum 27(6), 1539–1556 (2008)

25. Shapira, L., Shamir, A., Cohen-Or, D.: Consistent mesh partition-ing and skeletonisation using the shape diameter function. Vis.Comput. 24(4), 249–259 (2008)

26. Sheffer, A., Praun, E., Rose, K.: Mesh parameterization meth-ods and their applications. Found. Trends Comput. Graph. Vis.(FTCGV) 2(2), 64 (2007)

27. Tierny, J., Vandeborre, J.P., Daoudi, M.: Topology driven 3D meshhierarchical segmentation. In: IEEE International Conference onShape Modeling and Application (SMI) (2007)

28. Unnikrishnan, R., Pantofaru, C., Hebert, M.: Toward objectiveevaluation of image segmentation algorithms. IEEE Trans. PatternAnal. Mach. Intell. 29(6), 929–944 (2007)

29. Zeckerberger, E., Tal, A., Shlafman, S.: Polyhedral surface de-composition with applications. Comput. Graph. 26(5), 733–743(2002)

30. Zhang, H., Fritts, J., Goldman, S.: Image segmentation evaluation:A survey of unsupervised methods. Comput. Vis. Image Underst.110, 260–280 (2008)

Halim Benhabiles received theengineer degree from Universityof Sciences and Technologies ofHouari Boumediene (USTHB), Al-giers, Algeria, in 2006, and the M.S.degree from the University of Sci-ences and Technologies of Lille(USTL), Lille, France, in 2008,both in computer science. He iscurrently a Ph.D. candidate withinthe Computer Science laboratoryof the University of Lille (UMRUSTL/CNRS 8022). He is alsoa teaching assistant in TelecomLille1, School of Engineering. His

research interests include shape modeling, shape similarity estimation,geometry processing and their applications.

H. Benhabiles et al.

Jean-Philippe Vandeborre receivedthe M.Sc. degree in 1997 and thePh.D. degree in Computer Sciencein 2002, both from Lille 1 Univer-sity, France. Currently, he is an as-sociate professor in Institut Tele-com/Telecom Lille 1. He is also amember of the Computer ScienceLaboratory in Lille 1 University(LIFL UMR CNRS 8022). His re-search interests are mainly focusedon three-dimensional model analy-sis, and include indexing and re-trieval from content, 3D-mesh seg-mentation methods and automatic

evaluation of segmentation quality. He is co-author of several papers inrefereed journals and proceedings of international conferences. He hasbeen involved in French and European projects and has served as pro-gram committee member and reviewer for international journals andconferences.

Guillaume Lavoué received the en-gineering degree in signal process-ing and computer science from CPELyon (2002), the MSc degree in im-age processing from the UniversityJean Monnet, St-Etienne (2002) andthe PhD degree in computer sci-ence from the University ClaudeBernard, Lyon, France (2005). Af-ter a postdoctoral fellowship at theSignal Processing Institute (EPFL)in Switzerland, since September2006 he has been an associate pro-fessor at the French engineeringuniversity INSA of Lyon, in the

LIRIS Laboratory (UMR 5205 CNRS). His research interests include3D model analysis and processing, including compression, watermark-ing, perception, and 2D/3D recognition.

Mohamed Daoudi is a Full Pro-fessor of Computer Science in theInstitut TELECOM; TELECOMLille 1. He received the Ph.D. de-gree in Computer Engineering fromthe University of Lille 1 (USTL),France, in 1993 and Habilitationà Diriger des Recherches (HDR)from the University of Littoral,France, in 2000. He was the founderand the head of the MIIRE re-search group of LIFL (UMR USTL-CNRS 8022). His research interestsinclude pattern recognition, imageprocessing, invariant representation

of images and shapes, three-dimensional analysis and retrieval andmore recently 3D face recognition. He has published more than 80 pa-pers in refereed journals and proceedings of international conferences.He is the author the book 3D processing: Compression, Indexing andWatermarking (Wiley, 2008). He has served as a Program Commit-tee member for the International Conference on Pattern Recognition(ICPR) in 2004 and the International Conference on Multimedia andExpo (ICME) in 2004 and 2005. He is a frequent reviewer for IEEETransactions on Pattern Analysis and Machine Intelligence and for Pat-tern Recognition Letters. His research has been funded by ANR, RNRTand European Commission grants. He is Senior Member of IEEE.