On Symmetry, Perspectivity, and Level-Set-Based Segmentation

On Symmetry, Perspectivity,and Level-Set-Based Segmentation

Tammy Riklin-Raviv, Member, IEEE, Nir Sochen, Member, IEEE, and

Nahum Kiryati, Senior Member, IEEE

Abstract—We introduce a novel variational method for the extraction of objects with either bilateral or rotational symmetry in the

presence of perspective distortion. Information on the symmetry axis of the object and the distorting transformation is obtained as a

by-product of the segmentation process. The key idea is the use of a flip or a rotation of the image to segment as if it were another view

of the object. We call this generated image the symmetrical counterpart image. We show that the symmetrical counterpart image and

the source image are related by planar projective homography. This homography is determined by the unknown planar projective

transformation that distorts the object symmetry. The proposed segmentation method uses a level-set-based curve evolution

technique. The extraction of the object boundaries is based on the symmetry constraint and the image data. The symmetrical

counterpart of the evolving level-set function provides a dynamic shape prior. It supports the segmentation by resolving possible

ambiguities due to noise, clutter, occlusions, and assimilation with the background. The homography that aligns the symmetrical

counterpart to the source level-set is recovered via a registration process carried out concurrently with the segmentation. Promising

segmentation results of various images of approximately symmetrical objects are shown.

Index Terms—Symmetry, segmentation, level-sets, homography.

Ç

1 INTRODUCTION

SHAPE symmetry is a useful visual feature for imageunderstanding [48]. This research employs symmetry for

segmentation.1 In the presence of noise, clutter, shadows,occlusions, or assimilation with the background, segmenta-tion becomes challenging. In these cases, object boundariesdo not fully correspond to edges in the image and may notdelimit homogeneous image regions. Hence, classicalregion-based (RB) and edge-based segmentation techniquesare not sufficient. We therefore suggest a novel approach tofacilitate segmentation of objects that are known to besymmetrical by using their symmetry property as a shapeconstraint. The model presented is applicable to objectswith either rotational or bilateral (reflection) symmetrydistorted by projectivity.

The proposed segmentation method partitions the imageinto foreground and background domains, where theforeground region is known to be approximately symme-trical up to planar projective transformation. The boundaryof the symmetrical region (or object) is inferred by

minimizing a cost functional. This functional imposes thesmoothness of the segmenting contour, its alignment withthe image edges, and the homogeneity of the regions itdelimits. The assumed symmetry property of the object toextract provides an essential additional constraint.

There has been intensive research on symmetry relatedto human and computer vision. We mention a few of theclassical and of the most recent papers. Most naturalstructures are only approximately symmetrical, thereforethere is a great interest, pioneered by the work in [56] andfollowed by [18], in defining a measure of symmetry. Thereis a mass of work on symmetry and symmetry-axisdetection [1], [5], [19], [23], [29], [31], [34], [44], [49].Recovery of 3D structure from symmetry is explored in[12], [42], [45], [46], [50], [53], [57]. There are a few worksthat use symmetry for grouping [43] and segmentation [13],[14], [25], [27], [54]. The majority of the symmetry-relatedpapers consider either bilateral symmetry [12], [42], [57] orrotational symmetry [5], [16], [55], [34]. Some studies areeven more specific, for example, Milner et al. [30] suggest asymmetry measure for bifurcating structures, Ishikawaet al. [17] handle tree structures, and Hong et al. [16] andYang et al. [53] demonstrate the relation between symmetryand perspectivity on simple geometrical shapes.

Only a few algorithms treat the symmetrical object shapeas a single entity and not as a collection of landmarks orfeature points. Refer for example to [44], which suggests theedge strength function to determine the axes of localsymmetry in shapes, using variational framework. In thesuggested framework, the symmetrical object contour isrepresented as a single entity by the zero-level of a level-setfunction. We assign, without loss of generality, the positivelevels to the object domain. Taking the role of objectindicator functions, level-sets are most adequate for

1458 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 8, AUGUST 2009

. T. Riklin-Raviv is with the Department of Electrical Engineering andComputer Science, Massachusetts Institute of Technology, Cambridge, MA02139. E-mail: [email protected].

. N. Sochen is with the Department of Applied Mathematics, School ofMathematical Sciences, Tel Aviv University, Tel-Aviv 69978, Israel.E-mail: [email protected], [email protected].

. N. Kiryati is with the School of Electrical Engineering, Tel AvivUniversity, Tel-Aviv 69978, Israel. E-mail: [email protected].

Manuscript received 20 June 2007; revised 24 Mar. 2008; accepted 22 May2008; published online 6 June 2008.Recommended for acceptance by J. Weickert.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log NumberTPAMI-2007-06-0373.Digital Object Identifier no. 10.1109/TPAMI.2008.160.

1. A preliminary version of this manuscript appeared in [37].

0162-8828/09/$25.00 � 2009 IEEE Published by the IEEE Computer Society

dynamic shape representation. A dissimilarity measurebetween objects is defined as a function of the imageregions with contradicting labeling. Transformation of theregion bounded by the zero level is applied by a coordinatetransformation of the domain of the embedding level-setfunction [38], as illustrated in Fig. 1.

We define the concept of symmetrical counterpart in thecontext of image analysis. When the imaged object has abilateral symmetry, the symmetrical counterpart image isobtained by a vertical (or horizontal) flip of the imagedomain. When the imaged object has a rotational symmetry,the symmetrical counterpart image is provided by arotation of the image domain. In the same manner, wedefine the symmetrical counterpart of a level-set (or alabeling) function. The symmetry constraint is imposed byminimizing the dissimilarity measure between the evolvinglevel-set function and its symmetrical counterpart. Theproposed segmentation approach is thus fundamentallydifferent from other methods that support segmentation bysymmetry [13], [14], [25], [27], [54].

When symmetry is distorted by perspectivity, thedetection of the underlying symmetry becomes nontrivial,thus complicating symmetry-aided segmentation. In thiscase, even a perfectly symmetrical image is not identical toits symmetrical counterpart. We approach this difficulty byperforming registration between the symmetrical counter-parts. The registration process is justified by showing thatan image of a symmetrical object, distorted by a projectivetransformation, relates to its symmetrical counterpart by aplanar projective homography. A key result presented inthis manuscript is the structure of this homography, whichis determined, up to well-defined limits, by the distortingprojective transformation. This result allows us to bypassthe phase of symmetry axis detection.

Figs. 2 and 3 illustrate the main idea of the proposedframework. Fig. 2a shows an approximately symmetricalobject (its upper left part is corrupted) that underwent aperspective distortion. Fig. 2b is a reflection of Fig. 2a withrespect to the vertical symmetry axis of the image domain.Note, however, that this is not the symmetry axis of theobject, which is unknown. We call Fig. 2b the symmetricalcounterpart of Fig. 2a. Figs. 2a and 2b can be considered astwo views of the same object. Fig. 2b can be aligned toFig. 2a by applying a perspective transformation differentfrom the counter reflection, as shown in Fig. 2c. Super-position of Figs. 2a and 2c yields the complete noncorruptedobject view as shown in Fig. 2d. In the course of the iterativesegmentation process, the symmetrical counterpart of the

object delineated provides a dynamic shape prior2 and thusfacilitates the recovery of the hidden or vague objectboundaries.

Fig. 3 demonstrates the detection of symmetrical objects.In Fig. 3a, only one of the flowers imaged has rotationalsymmetry (up to an affine transformation). The symmetricalcounterpart image (Fig. 3b) is generated by rotation of theimage domain. Fig. 3c shows the superposition of theimages displayed in Figs. 3a and 3b. Fig. 3d presents thesuperposition of the original image (Fig. 3a) and itssymmetrical counterpart aligned to it. Note that thealignment between the two images was not obtained bythe counter rotation but by an affine transformation.

This paper contains two fundamental, related contribu-tions. The first is the use of an intrinsic shape property—symmetry—as a prior for image segmentation. This is madepossible by a group of theoretical results related tosymmetry and perspectivity that are the essence of thesecond contribution. Specifically, we present the structureof the homography that relates an image (or a level-setfunction) to its symmetrical counterpart. The unknownprojective transformation that distorts the object symmetrycan be recovered from this homography under certainconditions. These conditions are specified, defining theconcept of symmetry preserving transformation. Specifically,we show that transformations that do not distort the imagesymmetry cannot be recovered from this homography. Wealso propose a measure for the “distance” between alabeling function and its aligned symmetrical counterpart.We call it the symmetry imperfection measure. This measure isthe basis of the symmetry constraint that is incorporatedwithin a unified segmentation functional. The suggestedsegmentation method is demonstrated on various images ofapproximately symmetrical objects distorted by planarprojective transformation.

This paper is organized as follows: In Section 2, wereview the level-set formulation, showing its use fordynamic shape representation and segmentation. In Sec-tion 3, the concepts of symmetry and projectivity arereviewed. The main theoretical contribution resides inSection 3.3 which establishes the key elements of thesuggested study. In particular, the structure of the homo-graphy that relates between symmetrical counterpartimages is analyzed. In Section 4, a measure of symmetryimperfection of approximately symmetrical images, based onthat homography, is defined. We use this measure toconstruct a symmetry-shape term. Implementation detailsand further implications are presented in Section 5. Experi-mental results are provided in Section 6. We conclude inSection 7.

2 LEVEL-SET FRAMEWORK

We use the level-set formulation [11], [33] since it allowsnonparametric representation of the evolving object bound-ary and automatic changes of its topology. These character-istics are also most desirable for the dynamic representationof the shape of the image region bounded by the zero level.

RIKLIN-RAVIV ET AL.: ON SYMMETRY, PERSPECTIVITY, AND LEVEL-SET-BASED SEGMENTATION 1459

Fig. 1. Shape transformation. The coordinate transformation applied to

the domain of the object indicator function transforms the represented

shape accordingly.

2. The same expression with a different meaning was introduced in [6],[24] to denote shape prior which captures the temporal evolution of shape.

2.1 Level-Set Formulation

Let I denote an image defined on the domain � � IR2. We

define a closed contour C 2 � that partitions the image

domain � into two disjoint open subsets: ! and � n !.

Without loss of generality, the image pixels fðx; yÞ ¼ xjx 2!g will be attributed to the object domain. Using the level-

set framework [33], we define the evolving curve CðtÞ as the

zero level of a level-set function � : �! IR at time t:

CðtÞ ¼ x 2 � j �ðx; tÞ ¼ 0f g: ð1Þ

The optimal segmentation is therefore inferred by minimiz-

ing a cost functionalEð�; IÞwith respect to�. The optimizer�

is derived iteratively by �ðtþ�tÞ ¼ �ðtÞ þ �t�t. The

gradient descent equation �t is obtained from the first

variation of the segmentation functional using the Euler-

Lagrange equations.The suggested segmentation framework utilizes a uni-

fied functional which consists of an energy term driven by

the symmetry constraint together with the classical terms

derived from the constraints related to the low-level image

features:

Eð�; IÞ ¼ EDATAð�; IÞ þESYMð�Þ: ð2Þ

The data terms of EDATA are briefly reviewed in Sections 2.2,

2.3, and 2.4 based on current state-of-the-art level-set

segmentation methods [3], [2], [20], [22], [51]. Section 2.5

outlines the main idea behind the prior shape constraint

[38], which is the “predecessor” of the shape-symmetry

term. The remainder of this paper relates to ESYM.

2.2 Region-Based Term

In the spirit of the Mumford-Shah observation [32], we

assume that the object contour C ¼ @!, represented by the

zero level of �, delimits homogeneous image regions. We

use the Heaviside function (as in [3]) to assign the positive

levels of � to the object domain ð!Þ and the negative levels

to the background domain ð� n !Þ:

H �ðtÞð Þ ¼ 1 �ðtÞ � 00 otherwise:

�ð3Þ

Let Iin : !! IRþ and Iout : � n !! IRþ be the image

foreground and background intensities, respectively. De-

noting the gray-level averages by fuþ1 ¼ I in; u�1 ¼ Ioutg and

the respective gray-level variances by fuþ2 ; u�2 g, we look for

a level-set function � that minimizes the following region-

based (RB) term [3], [28], [52]:

ERBð�Þ ¼Z�

X2

i¼1

�Gþi IðxÞð Þ � uþi� �2

Hð�Þ

þ G�i IðxÞð Þ � u�i� �2

1�Hð�Þð Þ�dx;

ð4Þ

where Gþ1 ðIÞ ¼ G�1 ðIÞ ¼ I,

Gþ2 ðIÞ ¼ IðxÞ � I in

� �2; G�2 ðIÞ ¼ IðxÞ � Iout

� �2:

The level-set function � is updated by the gradient

descent equation:

�RBt ¼ �ð�ÞX2

i¼1

� Gþi ðIÞ � uþi� �2þ G�i ðIÞ � u�i

� �2h i

: ð5Þ

The sets of scalars fuþi g and fu�i g are updated alternately

with the evaluation of �ðtÞ:


Fig. 3. (a) The image contains an object with a rotational symmetry distorted by an affine transformation. (b) The symmetrical counterpart of

(a)—rotation of the image plane (a) by 90�. (c) Superposition of (a) and (b). (d) Superposition of the image (a) with the image (b) aligned to (a) by an

affine transformation. The object with rotational symmetry can be extracted.

Fig. 2. (a) Image of an approximately symmetrical object distorted by perspectivity. (b) The symmetrical counterpart of (a) is the reflection of (a) along

the vertical symmetry axis of the image domain. (c) The image (b) aligned to (a) by a planar projective transformation. (d) Superposition of the image

(a) with the image (c). The corrupted shape is recovered.

uþi ¼AþZ�

Gþi IðxÞð ÞHð�Þdx;

u�i ¼A�Z�

G�i IðxÞð Þ 1�Hð�Þð Þdx;ð6Þ

where Aþ ¼ 1=R

� Hð�Þdx and A� ¼ 1=R

�ð1�Hð�ÞÞdx.Practically, a smooth approximation of the Heavisidefunction rather than a step function is used [3]:

H�ð�Þ ¼1

21þ 2

�arctan

�

�

: ð7Þ

The evolution of � at each time step is weighted by thederivative of the regularized form of the Heavisidefunction:

��ð�Þ ¼dHð�Þd�

¼ 1

�

�

�2 þ �2:

To simplify the notation, the subscript � will be hereafteromitted.

2.3 Geodesic Active Contour and SmoothnessTerms

The edge-based segmentation criterion is derived from theassumption that the object boundaries coincide with thelocal maxima of the absolute values of the image gradients.Let rIðx; yÞ ¼ fIx; Iyg ¼ f@Iðx;yÞ@x ; @Iðx;yÞ@y g denote the vectorfield of the image gradients. Following [20], [2], the inverseedge indicator function g is defined by

g rIðxÞð Þ ¼ 1= � þ jrIðxÞj2� �

; ð8Þ

where � is a small and positive scalar. The geodesic activecontour (GAC) energy term

EGAC ¼Z�

g rIðxÞð ÞjrH �ðxÞð Þjdx ð9Þ

integrates the inverse edge indicator along the evolvingcontour. The evolution of � is determined by

�GACt ¼ �ð�Þdiv gðrIÞ r�jr�j

: ð10Þ

The GAC term reduces to the commonly used (e.g., [3])smoothness term by setting g ¼ 1:

ELEN ¼Z

�

rH �ðxÞð Þj jdx: ð11Þ

2.4 Edge Alignment Term

The edge alignment (EA) term constrains the level-set

normal direction ~n ¼ r�jr�j to align with the image gradient

direction rI [51], [21], [22]:

EEA ¼ �Z

�

rI; r�jr�j

� � rHð�Þj jdx: ð12Þ

The associated gradient descent equation is

�EAt ¼ ��ð�Þsign hr�;rIið Þ�I; ð13Þ

where h:; :i denotes the inner-product operation.

2.5 From Shape Constraint to Symmetry Constraint

Shape constraint, in level-set-based segmentation techni-ques, is commonly defined by a dissimilarity measurebetween the evolving level-set function and a representa-tion of the prior shape [4], [7], [10], [8], [9], [26], [38], [41],[47]. The main difficulty stems from the fact that the priorshape and the shape to segment are not aligned. Hence,shape transformations must be taken into consideration andthe segmentation should be accompanied by a registrationprocess.

2.5.1 Shape Transformations

Transformation of the shape defined by the propagatingcontour is obtained by a coordinate transformation of theimage domain � [38]. Let H denote a 3 � 3 matrix thatrepresents a planar transformation.3 The matrix H operateson homogeneous vectors X ¼ ðx; 1ÞT ¼ ðx; y; 1ÞT . We definean operator h : �! ~�, where ~� 2 IP2 is the projectivedomain such that hðxÞ ¼ X. Let X0 ¼ ðX0; Y 0; bÞ ¼ HX

denote the coordinate transformation of the projectivedomain ~�. The entries of the two-vector x0 2 � are theratios of the first two coordinates of X0 and the third one,i.e., x0 ¼ ðx0; y0Þ ¼ ðX0=b; Y 0=bÞ. Equivalently, one can use the“inverse” operator h�1 : ~�! � and write h�1ðX0Þ ¼ x0.

Let LðxÞ ¼ Hð�ðxÞÞ denote the indicator function of theobject currently being segmented. Transformation of LðxÞby a planar transformation H will be denoted byL � H � LðHxÞ, where Hx is a shorthand to h�1ðHðhðxÞÞÞ.Fig. 1 illustrates this notion. We can also use the coordinatetransformation to define shape symmetry, for example, ifLðxÞ ¼ Lðx; yÞ embeds a shape with left-right bilateralsymmetry, then Lðx; yÞ ¼ Lð�x; yÞ.

In general, we can think of L as a function defined on IR2,where � is the support of the image. This way, theoperation H maps vectors in IR2 to vectors in IR2 and iswell defined. Note that the shape of the support may bechanged under the action of the operator/matrix H.

2.5.2 Shape Dissimilarity Measure

Let ~L be a binary representation of the prior shape. Thesegmentation process is defined by the evolution of theobject indicator function L. A shape constraint takes theform DðLðxÞ; ~LðHxÞÞ < �, where D is a dissimilaritymeasure between LðxÞ and the aligned prior shaperepresentation ~LðHxÞ. The matrix H represents that align-ment and is recovered concurrently with the segmentationprocess.

When only a single image is given, such a prior is notavailable. Nevertheless, if an object is known to besymmetrical, we can treat the image and its symmetricalcounterpart (e.g., its reflection or rotation) as if they are twodifferent views of the same object. The instantaneoussymmetrical counterpart of the evolving shape provides adynamic shape prior. The symmetry dissimilarity measureis based on a theoretical framework established in Section 3.Section 4 considers the incorporation of the symmetryconstraint within a level-set framework for segmentation.


3. Note that H denotes the transformation, while H denotes theHeaviside function.

3 SYMMETRY AND PROJECTIVITY

3.1 Symmetry

Symmetry is an intrinsic property of an object. An object is

symmetrical with respect to a given operation if it remains

invariant under that operation. In 2D geometry, these

operations relate to the basic planar euclidean isometries:

reflection, rotation, and translation. We denote the symme-

try operator by S. The operator S is an isometry that

operates on homogeneous vectors X ¼ ðx; 1ÞT ¼ ðx; y; 1ÞTand is represented as

S ¼ sRð�Þ t0T 1

; ð14Þ

where t is a 2D translation vector, 0 is the null 2D vector, R

is a 2 � 2 rotation matrix, and s is the diagonal matrix

diagð�1;�1Þ.Specifically, we consider one of the following

transformations:

1. S is translation if t 6¼ 0 and s ¼ Rð�Þ ¼ diagð1; 1Þ.2. S is rotation if t ¼ 0, s ¼ diagð1; 1Þ, and � 6¼ 0.3. S is reflection if t ¼ 0, � ¼ 0, and s is either

diagð�1; 1Þ for left-right reflection or diagð1;�1Þ forup-down reflection.

In the case of reflection, the symmetry operation reverses

orientation; otherwise (translation and rotation), it is orienta-

tion preserving.The particular case of translational symmetry requires an

infinite image domain. Hence, it is not specifically con-

sidered in this manuscript.

Definition 1 (symmetrical image). Let S denote a symmetry

operator as defined in (14). The operator S is either reflection

or rotation. The image I : �! IRþ is symmetrical with

respect to S if

IðxÞ ¼ IðSxÞ � I � S: ð15Þ

The concept of symmetry is intimately related to the notion

of invariance. We say that a vector or a function L is

invariant with respect to the transformation (or operator) S

if SL ¼ L.Since we are interested in symmetry in terms of shape

and not in terms of gray levels, we will therefore consider

the object indicator function L : �! f0; 1g as defined in

Section 2.5. Hereafter, we use the shorthand notation for the

coordinate transformation of the image domain as defined

in Section 2.5.

Definition 2 (symmetrical counterpart). Let S denote a

symmetry operator. The object indicator function LðxÞ ¼L � SðxÞ ¼ LðSxÞ is the symmetrical counterpart of LðxÞ.L is symmetrical iff L ¼ L.

We claim that the object indicator function of a

symmetrical object distorted by a projective transformation

is related to its symmetrical counterpart by projective

transformation different from the defining symmetry.

Before we proceed with proving this claim, we recall the

definition of projective transformation.

3.2 Projectivity

This section follows the definitions in [15].

Definition 3. A planar projective transformation (projectivity) is

a linear transformation represented by a nonsingular 3 � 3

matrix H operating on homogeneous vectors, X0 ¼ HX,

where

H ¼h11 h12 h13

h21 h22 h23

h31 h32 h33

24

35: ð16Þ

Important specializations of the group formed by projective

transformation are the affine group and the similarity group

which is a subgroup of the affine group. These groups form

a hierarchy of transformations. A similarity transformation

is represented by

HSIM ¼�Rð�Þ t

0T 1

� �; ð17Þ

where R is a 2 � 2 rotation (by �) matrix and � is an

isotropic scaling. When � ¼ 1, HSIM is the euclidean

transformation denoted by HE . An affine transformation

is obtained by multiplying the matrix HSIM with

HA ¼K 00T 1

� �: ð18Þ

K is an upper-triangular matrix normalized as DetK ¼ 1.

The matrix HP defines the “essence” [15] of the projective

transformation and takes the form:

HP ¼1 0vT v

� �; ð19Þ

where 1 is the two-identity matrix. A projective transforma-

tion can be decomposed into a chain of transformations of a

descending (or ascending) hierarchy order

H ¼ HSIMHAHP ¼A tvT v

� �; ð20Þ

where v 6¼ 0 and A ¼ �RK þ tvT is a nonsingular matrix.

3.3 Theoretical Results

In this section, we consider the relation between an image

(or an object indicator function) of a symmetrical object

distorted by planar projective transformation H and its

symmetrical counterpart. The object indicator function L

will be treated as a binary image and will be called, for

simplicity, image.Recall from the previous section that a symmetrical

image (or labeling function) L with respect to the symmetry

operator S is invariant to S. We next define an operator

(matrix) SH such that a symmetrical image that underwent a

planar projective transformation is invariant to its operation.

Theorem 1. Let LH ¼ LðHxÞ denote the image obtained from

the symmetrical image L by applying a planar projective

transformation H. If L is invariant to S, i.e.,

LðxÞ ¼ LðSxÞ ¼ LðS�1xÞ, then LH is invariant to SH, where

SH ¼ H�1SH � H�1S�1H.


Proof. We need to prove that LHðxÞ ¼ LHðSHxÞ.We define y ¼ Hx.

LHðSHxÞ ¼LHðH�1SHxÞ ¼ LðSHxÞ¼LðSyÞ ¼ LðyÞ¼LðHxÞ ¼ LHðxÞ:

ð21Þ

tu

We now use the result obtained in Theorem 1 to define

the structure of the homography that aligns symmetrical

counterpart images.

Theorem 2. Let LH denote the image obtained from the

symmetrical image L by applying a planar projective

transformation H. Let LH ¼ LHðSxÞ � LH � S denote the

symmetrical counterpart of LH with respect to a symmetry

operation S. The image LH can be obtained from its

symmetrical counterpart LH by applying a transformation

represented by a 3 � 3 matrix of the form:

M ¼ S�1H�1SH: ð22Þ

Proof. We need to prove that LHðxÞ ¼ LHðMxÞ, where

M ¼ S�1H�1SH.The image LH is invariant to SH; thus,

LHðxÞ ¼ LHðSHxÞ. By definition, LHðxÞ ¼ LHðSxÞ.From the above equations and Theorem 1, defining

y ¼ S�1SHx, we get

LHðxÞ ¼LHðSHxÞ ¼ LHðSS�1SHxÞ¼LHðSyÞ ¼ LHðyÞ¼ LHðS�1SHxÞ ¼ LHðS�1H�1SHxÞ¼ LHðMxÞ:

ð23Þ

The image LH can be generated from its symmetricalcounterpart LH either by applying the inverse of thesymmetry operationS or by a projective transformationMwhich is different from S�1.

Let MINV denote a 3 � 3 nonsingular matrix such thatMINV ¼ H�1S�1HS. MINV is a projective transformationsince HMINV ¼ S�1HS is a projective transformationaccording to

S�1HS ¼ S�1 A tvT 1

� �S ¼ A0 t0

v0T v0

� �¼ H0; ð24Þ

where H is scaled such that v ¼ 1.Thus, M ¼M�1

INV represents a projective transforma-tion. It is easy to prove that M 6¼ S�1 when S is not theidentity matrix. Assume to the contrary that there existsa nonsingular H and a symmetry operation S such thatM ¼ S�1. Then, from (22), S�1 ¼ S�1H�1SH. Thus,H ¼ SH, which implies that either S is the identitymatrix or H is singular, in contradiction to theassumptions. tu

The next theorem gives tighter characterization of M.

Theorem 3. Let LH denote the image obtained from the

symmetrical image L by applying transformation H. Let M

denote the matrix that relates LH to its symmetrical

counterpart LH. The matrices M and H belong to the samesubgroup of transformations (either projective or affine orsimilarity transformation).

The proof of this theorem appears in Appendix A of thesupplemental material, which can be found on the ComputerSociety Digital Library at http://doi.ieeecomputersociety.org/10.1109/TPAMI.2008.160.

Next, we argue that H cannot be explicitly and fullyrecovered from M and S, if the operation of H (or one of itsfactors) on a symmetrical image does not distort itssymmetry.

Definition 4 (symmetry preserving transformation). Let Lbe a symmetrical image with respect to S. The projectivetransformation matrix H is symmetry preserving if

LH ¼ LðHxÞ ¼ LðSHxÞ: ð25Þ

Lemma 1. Let H denote a symmetry preserving transformationwith respect to the symmetry operation S. Then, H and Scommute, i.e., SH ¼ HS.

Proof. L is symmetrical, i.e., LðxÞ ¼ LðSxÞ. Applying thetransformation H on L, we obtain: LðHxÞ ¼ LðHSxÞ.Since H is symmetry preserving, LðHxÞ is symmetrical;thus, LðHxÞ ¼ LðSHxÞ. The latter two equations provethe claim. tu

Theorem 4. Consider the image LH ¼ LðHxÞ, where L is asymmetrical image. Let LH ¼ LðSHxÞ denote its symme-trical counterpart. Let the matrix M satisfy (22), whereLHðMxÞ ¼ LHðxÞ. If H can be factorized such thatH ¼ HS

~H, and HS denotes a symmetry preserving transfor-mation with respect to S, then H cannot be recovered from M.

Proof.

M ¼S�1H�1SH¼S�1ðHS

~HÞ�1SHS~H ¼ S�1 ~H�1H�1

S SHS~H

¼S�1 ~H�1H�1S HSS ~H

¼S�1 ~H�1S ~H:

ð26Þ

tuThe reader is referred to Appendix B of the supplementalmaterial, which can be found on the Computer SocietyDigital Library at http://doi.ieeecomputersociety.org/10.1109/TPAMI.2008.160, which demonstrates Theorems 1-4 by three toy examples.

4 SYMMETRY-BASED SEGMENTATION

4.1 Symmetry Imperfection

In the current section, we formalize the concept ofapproximately symmetrical shape. Specifically, two alternativedefinitions for �—symmetrical images and the symmetryimperfection measure are suggested. A symmetry imperfec-tion measure indicates how “far” the object is from beingperfectly symmetrical. Shape symmetry is constrained byminimizing this measure. When symmetry is distorted byprojectivity, defining a measure of symmetry imperfection


is not trivial. We approach this difficulty relying on theresults presented in Theorems 1 and 2.

Recall that a symmetrical binary image distorted byprojective transformation is invariant to the operator SH asdefined in (21). This is the basis of the first definition.

Definition 5 (symmetry imperfection (1)). The image L� is

�-symmetrical with respect to the symmetry operator S if

DðL�; L� � SÞ �, where D is a dissimilarity measure

(distance function) and � is a small and positive scalar. The

measure DðL�; L� � SÞ quantifies the symmetry imperfec-tion of L�. Let L�H denote the image obtained from L� by

applying a planar projective transformation H. The measure

for the symmetry imperfection of the perspectivelydistorted image L�H is defined by DðL�H; L�H � SHÞ; where

SH ¼ H�1SH.

Although this definition is natural, it is not alwaysapplicable since usually the projective transformation H isunknown. We therefore use an alternative equivalentdefinition of symmetry imperfection. This alternativemeasure involves the concept of the symmetrical counter-part image (see Definition 2) and the homography M thataligns symmetrical counterpart images.

Definition 6 (symmetry imperfection (2)). Let L�H be the

�-symmetrical image distorted by planar projective homogra-

phy H. Let L�H ¼ L�H � S denote the symmetrical counterpart

of L�H. The distance function DðL�H; L�H �MÞ measures the

symmetry imperfection of L�H.

Lemma 2. The two definitions are equivalent.

Proof. Direct consequence of the identityL�H �M ¼ L�H � SH. tu

In the following sections, we show how M can be recoveredduring the segmentation process by minimizing the termDH ¼ DðL�H; L�H �MÞ with respect to M. The recovery of Mis easier when H is either euclidean or affine transforma-tion, following the result of Theorem 3. In that case, onlyfour or six entries of M should be recovered. The matrix Hcan be recovered from M up to a symmetry preservingtransformation (Theorem 4).

4.2 Symmetry Constraint

In the previous section, the symmetrical counterparts wereeither images or object indicator functions. Using the level-set formulation, we will now refer to level-sets and theirHeaviside functions Hð�Þ. Recall that Hð�ðtÞÞ is anindicator function of the estimated object regions in theimage at time t.

Let � : �! IR denote the symmetrical counterpart of� with respect to a symmetry operation S. Specifically,�ðxÞ ¼ �ðSxÞ, where S is either a reflection or arotation matrix. We assume for now that S is known.This assumption will be relaxed in Section 5.6. Notehowever that H is unknown. Let M denote the planarprojective transformation that aligns Hð�Þ to Hð�Þ, i.e.,Hð�Þ �M ¼ Hð�ðMxÞÞ ¼ Hð�MÞ. The matrix M is recov-ered by a registration process held concurrently with thesegmentation, detailed in Section 4.4.

Let D ¼ DðHð�Þ; Hð�MÞÞ denote a dissimilarity measurebetween the evolving shape representation and its symme-trical counterpart. Note that if M is correctly recovered and� captures a perfectly symmetrical object (up to projectiv-ity), then D should be zero. As shown in the previoussection, D is a measure for symmetry imperfection andtherefore defines a shape symmetry constraint. The nextsection considers possible definitions of D.

4.3 Biased Shape Dissimilarity Measure

Consider, for example, the approximately symmetrical (upto projectivity) images shown in Figs. 4a and 4d. Theobject’s symmetry is distorted by either deficiencies orexcess parts. A straightforward definition of a symmetryshape constraint which measures the distance between theevolving segmentation and the aligned symmetrical coun-terpart takes the form:

Dð�; �jMÞ ¼Z

�

H �ðxÞð Þ �Hð�MÞh i2

dx: ð27Þ

However, incorporating this, the unbiased shape constraintin the cost functional for segmentation, results in theundesired segmentation shown in Figs. 4b and 4e. This isdue to the fact that the evolving level-set function � is asimperfect as its symmetrical counterpart �. To support acorrect evolution of � by �, one has to account for the causeof the imperfection.

Refer again to (27). This dissimilarity measure integratesthe nonoverlapping object-background regions in bothimages indicated by Hð�Þ and Hð�Þ. This is equivalent toa pointwise exclusive-or (xor) operation integrated over theimage domain. We may thus rewrite the functional asfollows:

Dð�; �jMÞ ¼Z

�

Hð�Þ 1�Hð�MÞ� �

þ 1�Hð�Þð ÞHð�MÞh i

dx:

ð28Þ

Note that expressions (27) and (28) are identical sinceHð�Þ ¼ ðHð�ÞÞ2. There are two types of “disagreement”between the labeling of Hð�Þ and Hð�MÞ. The first additiveterm in the right-hand side of (28) is not zero for image regionslabeled as object by � and labeled as background by itssymmetrical counterpart �. The second additive term of (28)is not zero for image regions labeled as background by � and


Fig. 4. (a) and (d) Images of symmetrical objects up to a projectivetransformation. The objects are distorted either (a) by deficiencies or(d) by excess parts. (b) and (e) Segmentation (red) of the images in (a)and (d), respectively, using an unbiased dissimilarity measure between� and its transformed reflection as in (27). Object segmentation is furtherspoiled due to the imperfection in its reflection. (c) and (f) Successfulsegmentation (red) using the biased dissimilarity measure as in (30).

labeled as object by �. We can change the relative contributionof each term by a relative weight parameter > 0:

ESYM ¼Z

�

Hð�Þ 1�Hð�MÞ� �

þ 1�Hð�Þð ÞHð�MÞh i

dx:

ð29Þ

The associated gradient equation for � is then

�SYMt ¼ �ð�Þ Hð�MÞ � 1�Hð�MÞ

� �h i: ð30Þ

Now, if excess parts are assumed, the left penalty termshould be dominant, setting > 1. Otherwise, if deficien-cies are assumed, the right penalty term should bedominant, setting < 1. Figs. 4c and 4f show segmentationof symmetrical objects with either deficiencies or excessparts, incorporating the symmetry term (29) within thesegmentation functional. We used ¼ 0:8 and ¼ 2 for thesegmentation of Figs. 4c and 4f, respectively. A similardissimilarity measure was proposed in [40], [39] to quantifythe “distance” between the alternately evolving level-setfunctions of two different object views.

4.4 Recovery of the Transformation

In the previous section, we defined a biased symmetryimperfection measure ESYM (29) that measures the mis-match between Hð�Þ and its aligned symmetrical counter-part Hð�MÞ. We now look for the optimal alignment matrixM that minimizes ESYM.

Using the notation defined in Section 2.5, we define thecoordinate transformation M of �ðxÞ as follows:

MH �ðxÞ� �

¼ H �ðxÞ� �

�M ¼ H �ðMxÞ� �

� Hð�MÞ: ð31Þ

We assume that the matrix M is a planar projectivehomography, as defined in (20). The eight unknown ratiosof its entries mk ¼ mij=h33, fi; jg ¼ f1; 1g; f1; 2g; . . . f3; 2gare recovered through the segmentation process, alternatelywith the evolution of the level-set function �. The equationsfor mk are obtained by minimizing (30) with respect to each

@mk

@t¼Z

�

�ð�MÞ 1�Hð�Þð Þ � Hð�Þ½ @Mð�Þ@mk

dx; ð32Þ

where

@Mð�Þ@mk

¼ @Mð�Þ@x

@x

@x0@x0

@mkþ @x

@y0@y0

@mk

þ @Mð�Þ@y

@y

@x0@x0

@mkþ @y

@y0@y0

@mk

:

ð33Þ

Refer to [38], [36], [35] for detailed derivation of (33).

4.5 Unified Segmentation Functional

Symmetry-based, edge-based, RB, and smoothness con-straints can be integrated to establish a comprehensive costfunctional for segmentation:

Eð�Þ ¼WRBERBð�Þ þWGACEGACð�ÞþWEAEEAð�Þ þWSYMESYMð�Þ;

ð34Þ

with (4), (9), (12), (29).

Note that the smoothness (length) term (11) isincluded in the GAC term EGAC (9). This canbe shown by splitting gðjrIjÞ (8) as follows:gðjrIðxÞjÞ ¼ �=ð� þ jrIj2Þ ¼ 1� jrIj2=ð� þ jrIj2Þ, wherethe � in the numerator of the left-hand side of the equationcan be absorbed in the weight of the GAC term (see also[39]). The associated gradient descent equation �t is

�t ¼WRB ��RBt þWGAC ��GAC

t þWEA ��EAt þWSYM ��SYM

t : ð35Þ

The terms ��TERMt are obtained by slight modification of the

gradient descent terms �TERMt determined by (5), (10), (13),

(30). This issue and the determination of the weights WTERM

for the different terms in (35) are discussed in Section 5.3.Refinement of the segmentation results can be obtained

for images with multiple channels, I : �! IRn, e.g., colorimages. The RB term �RBt and the alignment term �EA

t arethus the sum of the contributions of each channel Ii.Multichannel segmentation is particularity suitable whenthe object boundaries are apparent in some of the channelswhile the piecewise homogeneity is preserved in others.Fig. 8 demonstrates segmentation of a color image.

5 IMPLEMENTATION AND FURTHER IMPLICATIONS

Segmentation is obtained by minimizing a cost functional thatincorporates symmetry as well as RB, edge-based, andsmoothness terms. The minimizing level-set function isevolved concurrently with the registration of its instanta-neous symmetrical counterpart to it. The symmetricalcounterpart is generated by a flip or rotation of the coordinatesystem of the propagating level-set function. The evolution ofthe level-set function is controlled by the constraints imposedby the data of the associated image and by its alignedsymmetrical counterpart. The planar projective transforma-tion between the evolving level-set function and its symme-trical counterpart is updated at each iteration.

5.1 Algorithm

We summarize the proposed algorithm for segmentation ofa symmetric object distorted by projectivity:

1. Choose an initial level-set function �t¼0 that deter-mines the initial segmentation contour.

2. Set initial values for the transformation para-meters mk. For example, setM to the identity matrix.

3. Compute uþ and u� according to (6), based on thecurrent contour interior and exterior, defined by �ðtÞ.

4. Generate the symmetrical counterpart of �ðxÞ,�ðxÞ ¼ �ðSxÞ, when the symmetry operator S isknown.4

5. Update the matrix M by recovering the transforma-tion parameters mk according to (32).

6. Update � using the gradient descent equation (35).7. Repeat steps 3-6 until convergence.

5.2 Initialization

The algorithm is quite robust to the selection of initiallevel-set function �0ðxÞ. The only limitation is that image


4. When S is unknown, use �ðxÞ ¼ �ðTxÞ, where T is of the same type asS. Refer to Section 5.6 for further details.

regions labeled as foreground at the first iteration, i.e.,!0 ¼ fxj�0ðxÞ � 0g, should contain a significant portion ofthe object to be segmented such that the calculated imagefeatures will approximately characterize the object region.Formally, we assume that GþðIð!0ÞÞ � GþðIð!ÞÞ, where ! isthe actual object region in the image. When there exists anestimate of the average gray levels of either the foregroundor the background image regions, this restriction can beeliminated.

We run the algorithm until the following stoppingcondition is met:

Xx2�

H �tþ�tðxÞ� �

�H �tðxÞ� � < s;

where s is a predefined threshold and �tþ�tðxÞ is the level-set function, at time tþ�t.

5.3 Setting the Weights of the Energy Terms

When the solution to an image analysis problem is obtainedby minimizing a cost functional, a “correct” setting of theweights of the energy terms is of fundamental importance.Following our suggestion in [39], the weights of the terms inthe functional presented in (35) are adaptively set. Theautomatic and dynamic weight setting is carried out in twostages. First, we apply a thresholding operation to boundthe dynamic range of �tðxÞ. The threshold is determined bythe standard deviation of �tðxÞ over �:

��TERMt ðxÞ ¼

UB if �TERMt ðxÞ > UB

LB if �TERMt ðxÞ < LB

�TERMt ðxÞ otherwise;

8<: ð36Þ

where

UB ¼ std �TERMt ðxÞ

� �; LB ¼ �UB:

Here, stdð�tðxÞÞ stands for the standard deviation of �tðxÞover �. The functional Bð:Þ operates on �TERM

t to bound its

dynamic range. Next, the range of j ��TERMt j is normalized:

WTERM ¼ 1=maxx

��TERMt ðxÞ

: ð37Þ

Note that the clipping (36) affects only extreme values of

�TERMt , that is, ��TERM

t ðxÞ ¼ �TERMt ðxÞ for most x 2 �. Since

W is recalculated at each iteration, it is time dependent. This

formulation enables an automatic and adaptive determina-

tion of the weights of the energy terms.

5.4 Recovery of the Transformation Parameters

Minimizing the symmetry term (29) with respect to theeight unknown ratios of the homography matrix entries is acomplicated computational task. As in [39], we perform arough search in the 8D parameter space working on acoarse to fine set of grids before applying the gradient-based Quasi-Newton method. The former search, done onlyonce, significantly reduces the search space. The gradient-based algorithm, applied in every iteration, refines thesearch result based on the updated level-set function.

It is worth noting that unlike other registration algo-rithms, obtaining a global minimum is undesired. This“surprising” claim can be understood when observing thata global minimum of the registration process is obtained

when M ¼ S�1, where S is the transformation used togenerate the symmetrical counterpart. In this case, thesegmented object and its symmetrical counterpart wouldhave a perfect match and the symmetry property is actuallynot used to facilitate the segmentation. To obtain a localminimum which is not a global one in the case of rotationalsymmetry, we deliberately exclude the rotation that gen-erates the symmetrical counterpart from the multigrid one-time search performed prior to the steepest descentregistration process. Note that such global minima willnot be obtained in the cases of bilateral symmetry since S isa reflecting transformation while the homography matrix Mdoes not reverse orientation by definition [15].

Fig. 10c demonstrates the main point of the claim madeabove. The figure shows the evolving contour (red) togetherwith the registered symmetrical counterpart (white) thatwas generated by a clockwise rotation of 90�. If thetransformation parameters found by the registration pro-cess were equivalent to counterclockwise rotation by 90�,the “defects” in both the segmented wheel and the registeredsymmetrical counterpart would be located at the sameplace. Thus, the “missing parts” could not be recovered.

5.5 Multiple Symmetrical Counterparts

Consider the segmentation of an object with rotationalsymmetry. Let �ðtÞ denote its corresponding level-setfunction at time t, and LH ¼ Hð�ðtÞÞ denote the respectiveobject indicator function. If LH is invariant to rotation by degrees, where 2�=3, then more than one symme-trical counterpart level-set functions can be used to supportthe segmentation. Specifically, the number of supportivesymmetrical-counterpart level-set functions is N ¼b2�=c � 1 since the object is invariant to rotations byn degree, where n ¼ 1 . . .N . We denote this rotation byRn. In that case, the symmetry constraint takes thefollowing form:

ESYM ¼XNn¼1

Z�

�Hð�Þ 1�Hð�n �MnÞ

� �

þ 1�Hð�Þð ÞHð�n �MnÞ�dx;

ð38Þ

where �n ¼ �ðRnxÞ ¼ � �Rn and Mn is the homography

that aligns �n to �. The computational cost using this

symmetry constraint is higher since N homographies Mn

should be recovered. However, the eventual segmentation

results are better, as shown in Fig. 11.

5.6 Segmentation with Partial SymmetryInformation

In most practical applications, S is not available. In this

section, we generalize the suggested framework, assuming

that only the type of symmetry (i.e., rotation or translation) is

known. For example, we know that the object is rotationally

symmetrical yet the precise rotation degree is not known.

Alternatively, the object is known to have a bilateral

symmetry, however, we do not know if the bilateral

symmetry is horizontal or vertical. LetL ¼ Hð�Þ be invariant

to the symmetry operator S ¼ R, where R is a planar

clockwise rotation by degree. Let LH be a perspective


distortion ofLbyH. Consider the case where the symmetrical

counterpart ofLH is generated by a planar clockwise rotation

� degree, where � 6¼ , thus L�H ¼ LHðR�xÞ � LH �R�. We

define a modified symmetry imperfection measure of the

following form: DðLH; L�H � ~MÞ. The matrix ~M aligns L�H to

LH. The next theorem relates M and ~M.

Theorem 5. Let L ¼ L � S, where S ¼ R. Let M be a planar

projective homography such that LHðxÞ ¼ LHðMxÞ, where

LH ¼ LHðRxÞ. We call LH the standard symmetrical

counterpart. Suppose that L�H ¼ LHðR�xÞ is generated from

LH by an arbitrary rotation �, where � 6¼ . We claim that

LH ¼ L�Hð ~MxÞ, where ~M ¼MR��.

Proof. By definition, LHðxÞ ¼ LHðRxÞ and L�HðxÞ ¼LHðR�xÞ: This implies that

LHðxÞ ¼ L�HðR��xÞ: ð39Þ

Using Theorem 2 for the special case of S ¼ R and the

above definition of LH, we get

LHðxÞ ¼ LHðM�1xÞ: ð40Þ

From (39) and (40), we get LHðM�1xÞ ¼ L�HðR��xÞ. The

latter equation implies that

LHðxÞ ¼ L�HðMR��xÞ ¼ L�Hð ~MxÞ;

defining ~M ¼MR��.The homography ~M is equivalent to M up to a

rotation by � � degrees. tu

Theorem 5 implies that, when the symmetry operator is an

unknown rotation, the symmetrical counterpart level-set

function can be generated by an arbitrary rotation (e.g., �=2).

In this case, the recovered homography is equivalent to the

homography of the standard symmetrical counterpart up to a

planar rotation. The distorting projective transformation Hcan be recovered up to a similarity transformation.

The result presented in Theorem 5 is also applicable to an

object with bilateral symmetry. An image (or any function

defined on IR2) reflected along its vertical axis can be

aligned to its reflection along its horizontal axis by a

rotation of R ¼ �. Thus, SLR ¼ Rð�ÞSUD, where SLR is an

horizontal reflection and SUD is a vertical reflection.

Similarly, SUD ¼ Rð�ÞSLR.

6 EXPERIMENTS

We demonstrate the proposed algorithm for the segmenta-

tion of approximately symmetrical objects in the presence of

projective distortion. The images are displayed with the

initial and final segmenting contours. Segmentation results

are compared to those obtained using the functional (34)

without the symmetry term. The contribution of each term

in the gradient descent equation (35) is bounded to ½�1; 1, as

explained in Section 5.3. We used ¼ 0:8 for the segmenta-

tion of Figs. 6, 8, and 10, assuming occlusions and ¼ 2 for the

segmentation of Figs. 5, 7, 9, and 11, where the symmetrical

object has to be extracted from background image regions

with the same gray levels. In Fig. 5, the approximately

symmetrical object (man’s shadow) is extracted based on the

bilateral symmetry of the object. In this example, the initial

level-set function �0 has been set to an all-zero function—-

which implies no initial contour. Instead, the initial mean

intensities have been set as follows: uþ ¼ 0; u� ¼ 1 assuming

that the average intensity of the background is brighter than

the foreground. We use this form of initialization to

demonstrate the independence of the proposed algorithm

with respect to the size, shape, or orientation of the initial

contour. In Fig. 6, the upper part of the guitar is used to extract

its lower part correctly. In the butterfly image shown in Fig. 7,

a left-right reflection of the evolving level-set function is used

to support accurate segmentation of the left wing of the

butterfly. Since in this example the transformation H is


Fig. 5. (a) Input image of a symmetrical object with the evolvingsegmentation contour (red) after one iteration. Here, �0 was set to zerowhile uþ0 ¼ 0 and u�0 ¼ 1. (b) Successful segmentation (red) with theproposed algorithm. A video clip that demonstrates the contour evolutionof this example is provided as supplemental material, which can befound on the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/TPAMI.2008.160. Original image courtesy ofAmit Jayant Deshpande. URL: http://web.mit.edu/amitd/www/pics/chicago/shadow.jpg.

Fig. 6. (a) Input image of a symmetrical object with the initial segmentation contour (red). (b) Segmentation (red) without the symmetry constraint.

(c) Successful segmentation (red) with the proposed algorithm.

euclidean, the orientation of the butterfly’s symmetry axis can

be easily recovered from M.In the swan example shown in Fig. 8, we used both color

and symmetry cues for correct extraction of the swan and its

reflection. Based on the image intensities alone, the vulture

and the tree in Fig. 9 are inseparable. Adding the symmetry

constraint, the vulture is extracted as shown in Fig. 9b. In this

example, the segmentation is not precise because of the

incorrect registration of the symmetrical counterpart to the

evolving level-set function. The undesired local minimum

obtained in the registration is probably due to the

dominance of none-symmetric tree region.The example shown in Fig. 10 demonstrates segmenta-

tion in the presence of rotational symmetry. The original

image (not shown) is invariant to rotation by 60�n, where

n ¼ 1; 2 . . . 5. Figs. 10a and 10e show the images to segment

which are noisy, corrupted, and transformed versions of the

original image. Fig. 10b shows the symmetrical counterpart

of the image in Fig. 10a obtained by a clockwise rotation of

90�. Practically, only the symmetrical counterpart of the



(c) Successful segmentation (red) with the proposed algorithm. Original image courtesy of George Payne. URL: http://cajunimages.com/images/

butterfly%20wallpaper.jpg.


(c) Successful segmentation (red) with the proposed algorithm. Original image courtesy of Richard Lindley. URL: http://www.richardlindley.co.uk/

links.htm.

Fig. 9. (a) Input image of a symmetrical object with the evolving segmentation contour (red) after one iteration. Here, �0 was set to zero while uþ0 ¼ 0

and u�0 ¼ 1. (b) The vulture is extracted (red) although, based on the image intensities alone, it is inseparable from the tree. Original image courtesy

of Allen Matheson. URL: http://thegreencuttingboard.blogspot.com/Vulture.jpg.

evolving level-set function is used. The symmetrical

counterpart image is shown here for demonstration. The

segmenting contour (red) of the corrupted object in Fig. 10a

is shown in Fig. 10c together with the registered symmetrical

counterpart. Note that the symmetrical counterpart is not

registered to the segmented object by a counterclockwise

rotation of 90� but by a different transformation. Otherwise,

there would have been a perfect match between the

“defects” in the image to segment and its registered

symmetrical counterpart. The successful segmentations

(red) of the image in Figs. 10a and 10e using the proposed

symmetry-based algorithm is presented in Figs. 10d and

10f, respectively. Figs. 10g and 10h demonstrate unsuccess-

ful segmentations (red) of the image in Fig. 10e using the

Chan-Vese algorithm [3]. In Fig. 10h, the contribution of the

smoothness term was four times bigger than in Fig. 10g.

Note that, using the symmetrical counterpart, we success-fully overcome the noise.

In the last example (Fig. 11), segmentation of an object withapproximate rotational symmetry is demonstrated. In thisexample, we could theoretically use four symmetricalcounterparts to support the segmentation. However, asshown in Fig. 11d, two symmetrical counterparts aresufficient. We used the symmetry constraint for multiplesymmetrical counterpart level-set functions according to (38).

7 SUMMARY

This paper contains two fundamental, related contributions.The first is a variational framework for the segmentation ofsymmetrical objects distorted by perspectivity. The pro-posed segmentation method relies on theoretical resultsrelated to symmetry and perspectivity which are theessence of the second contribution.

Unlike most of the previous approaches to symmetry, thesymmetrical object is considered as a single entity and notas a collection of landmarks or feature points. This isaccomplished by using the level-set formulation andassigning, for example, the positive levels of the level-setfunction to the object domain and the negative levels to thebackground. An object, in the proposed framework, isrepresented by the support of its respective labelingfunction. As the level-set function evolves, the correspond-ing labeling function and, thus, the represented objectchange accordingly.

A key concept in the suggested study is the symmetricalcounterpart of the evolving level-set function, obtained byeither rotation or reflection of the level-set function domain.We assume that the object to segment underwent a planar


Fig. 10. (a) The image to segment. It is a noisy, corrupted, and transformed version of an original image (not shown). The original image is invariantto rotation by 60�n, where n ¼ 1; 2 . . . 5. (b) The symmetrical counterpart of the image in (a) obtained by a clockwise rotation of 90�. Practically, onlythe symmetrical counterpart of the evolving level-set function is used. The symmetrical counterpart image is shown here for demonstration. (c) Thesegmenting contour (red) of the object in (a) is shown together with the registered symmetrical counterpart. Note that the symmetrical counterpart isnot registered to (a) by a counterclockwise rotation of 90� but by a different transformation. Otherwise, there would have been a perfect matchbetween the “defects” in the image to segment and its registered symmetrical counterpart. (d) Successful segmentation (red) of the image in (a)using the proposed symmetry-based algorithm. (e) Noisier version of the image in (a) together with the initial contour (red). (f) Successfulsegmentation (red) of the image in (e) using the proposed symmetry-based algorithm. (g) and (h) Unsuccessful segmentation (red) of the image in(e) using the Chan-Vese algorithm [3]. In (h), the contribution of the smoothness term was four times bigger than in (g).

Fig. 11. An image of a rotationally symmetrical object. The object isinvariant to rotation by 72�n, where n ¼ 1; 2; . . . 4. (a) Segmentation(red) without the symmetry constraint. (b) Segmentation (red) withthe proposed algorithm using a single symmetrical counterpart.(c) Segmentation (red) with the proposed algorithm using twosymmetrical counterparts. Original image courtesy of Kenneth R.Robertson. URL: http://inhs.uiuc.edu.

projective transformation, thus the source level-set functionand its symmetrical counterpart are not identical. We showthat these two level-set functions are related by planar

projective homography.Due to noise, occlusion, shadowing, or assimilation with

the background, the propagating object contour is onlyapproximately symmetrical. We define a symmetry im-

perfection measure which is actually the “distance”between the evolving level-set function and its alignedsymmetrical counterpart. A symmetry constraint based on

the symmetry imperfection measure is incorporated withinthe level-set based functional for segmentation. The homo-graphy matrix that aligns the symmetrical counterpart

level-set functions is recovered concurrently with thesegmentation.

We show in Theorem 2 that the recovered homographyis determined by the projective transformation that distorts

the image symmetry. This implies that the homographyobtained by the alignment process of the symmetricalcounterpart functions (or images) can be used for partial

recovery of the 3D structure of the imaged symmetricalobject. As implied from Theorem 4, full recovery is notpossible if the projection of the 3D object on the image plane

preserves its symmetry.The algorithm suggested is demonstrated on various

images of objects with either bilateral or rotationalsymmetry, distorted by projectivity. Promising segmenta-

tion results are shown. In this manuscript, only shapesymmetry is considered. The proposed framework can beextended considering symmetry in terms of gray levels or

color. The notion of symmetrical counterpart can be thusapplied to the image itself and not to binary (or level-set)functions. Possible applications are de-noising, super-resolution from a single symmetrical image, and inpainting.

All are subjects for future research.

ACKNOWLEDGMENTS

This research was supported by the A.M.N. Foundation andby MUSCLE: Multimedia Understanding through Seman-tics, Computation and Learning, a European Network of

Excellence funded by the EC Sixth Framework ISTProgramme. Tammy Riklin-Raviv was also supported bythe Yizhak and Chaya Weinstein Research Institute for

Signal Processing.

REFERENCES

[1] A. Bruckstein and D. Shaked, “Skew-Symmetry Detection viaInvariant Signatures,” Pattern Recognition, vol. 31, no. 2, pp. 181-192, 1998.

[2] V. Caselles, R. Kimmel, and G. Sapiro, “Geodesic ActiveContours,” Int’l J. Computer Vision, vol. 22, no. 1, pp. 61-79, Feb.1997.

[3] T. Chan and L. Vese, “Active Contours without Edges,” IEEETrans. Image Processing, vol. 10, no. 2, pp. 266-277, Feb. 2001.

[4] Y. Chen, H. Tagare, S. Thiruvenkadam, F. Huang, D. Wilson, K.Gopinath, R. Briggs, and E. Geiser, “Using Prior Shapes inGeometric Active Contours in a Variational Framework,” Int’l J.Computer Vision, vol. 50, no. 3, pp. 315-328, 2002.

[5] H. Cornelius and G. Loy, “Detecting Rotational Symmetry underAffine Projection,” Proc. Int’l Conf. Pattern Recognition, pp. 292-295,2006.

[6] D. Cremers, “Dynamical Statistical Shape Priors for Level SetBased Tracking,” IEEE Trans. Pattern Analysis and MachineIntelligence, vol. 28, no. 8, pp. 1262-1273, Aug. 2006.

[7] D. Cremers, T. Kohlberger, and C. Schnorr, “Shape Statistics inKernel Space for Variational Image Segmentation,” PatternRecognition, vol. 36, no. 9, pp. 1929-1943, 2003.

[8] D. Cremers, S.J. Osher, and S. Soatto, “Kernel Density Estimationand Intrinsic Alignment for Shape Priors in Level Set Segmenta-tion,” Int’l J. Computer Vision, vol. 69, no. 3, pp. 335-351, 2006.

[9] D. Cremers and S. Soatto, “A Pseudo-Distance for Shape Priors inLevel Set Segmentation,” Proc. Second IEEE Workshop Variational,Geometric and Level Set Methods in Computer Vision, pp. 169-176,2003.

[10] D. Cremers, N. Sochen, and C. Schnorr, “Multiphase DynamicLabeling for Variational Recognition-Driven Image Segmenta-tion,” Int’l J. Computer Vision, vol. 66, no. 1, pp. 67-81, 2006.

[11] A. Dervieux and F. Thomasset, “A Finite Element Method for theSimulation of Rayleigh-Taylor Instability,” Lecture Notes in Math.,vol. 771, pp. 145-158, 1979.

[12] R. Fawcett, A. Zisserman, and J. Brady, “Extracting Structure froman Affine View of a 3D Point Set with One or 2 BilateralSymmetries,” Image and Vision Computing, vol. 12, no. 9, pp. 615-622, Nov. 1994.

[13] J. Gauch and S. Pizer, “The Intensity Axis of Symmetry and ItsApplication to Image Segmentation,” IEEE Trans. Pattern Analysisand Machine Intelligence, vol. 15, no. 8, pp. 753-770, Aug. 1993.

[14] A. Gupta, V. Prasad, and L. Davis, “Extracting Regions ofSymmetry,” Proc. Int’l Conf. Image Processing, pp. III:133-136, 2005.

[15] R.I. Hartley and A. Zisserman, Multiple View Geometry in ComputerVision, second ed. Cambridge Univ. Press, 2003.

[16] W. Hong, Y. Yang, and Y. Ma, “On Symmetry and Multiple ViewGeometry: Structure, Pose and Calibration from Single Image,”Int’l J. Computer Vision, vol. 60, pp. 241-265, 2004.

[17] H. Ishikawa, D. Geiger, and R. Cole, “Finding Tree Structures byGrouping Symmetries,” Proc. Int’l Conf. Computer Vision, pp. 1132-1139, 2005.

[18] K. Kanatani, “Symmetry as a Continuous Feature: Comment,”IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 3,pp. 246-247, Mar. 1997.

[19] Y. Keller and Y. Shkolnisky, “An Algebraic Approach toSymmetry Detection,” Proc. Int’l Conf. Pattern Recognition,pp. III:186-189, 2004.

[20] S. Kichenassamy, A. Kumar, P. Olver, A. Tannenbaum, and A.Yezzi, “Gradient Flows and Geometric Active Contour Models,”Proc. Int’l Conf. Computer Vision, pp. 810-815, 1995.

[21] R. Kimmel, “Fast Edge Integration,” Geometric Level Set Methods inImaging Vision and Graphics, S. Osher and N. Paragios, eds.,Springer-Verlag, 2003.

[22] R. Kimmel and A. Bruckstein, “Regularized Laplacian ZeroCrossings as Optimal Edge Integrators,” Int’l J. Computer Vision,vol. 53, no. 3, pp. 225-243, 2003.

[23] N. Kiryati and Y. Gofman, “Detecting Symmetry in Grey LevelImages: The Global Optimization Approach,” Int’l J. ComputerVision, vol. 29, no. 1, pp. 29-45, 1998.

[24] T. Kohlberger, D. Cremers, M. Rousson, and R. Ramaraj, “4DShape Priors for Level Set Segmentation of the Left Myocardiumin SPECT Sequences,” Medical Image Computing and Computer-Assisted Intervention, vol. 4190, pp. 92-100, Oct. 2006.

[25] A. Laird and J. Miller, “Hierarchical Symmetry Segmentation,”Proc. SPIE, Intelligent Robots and Computer Vision IX: Algorithms andTechniques, vol. 1381, pp. 536-544, Feb. 1991.

[26] M. Leventon, W. Grimson, and O. Faugeras, “Statistical ShapeInfluence in Geodesic Active Contours,” Proc. IEEE CS Conf.Computer Vision and Pattern Recognition, vol. 1, pp. 316-323, 2000.

[27] T. Liu, D. Geiger, and A. Yuille, “Segmenting by Seeking theSymmetry Axis,” Proc. Int’l Conf. Pattern Recognition, pp. 994-998,1998.

[28] L. Lorigo, O. Faugeras, W.E.L. Grimson, R. Keriven, R. Kikinis, A.Nabavi, and C. Westin, “Codimension Two-Geodesic ActiveContours for the Segmentation of Tabular Structures,” Proc. IEEECS Conf. Computer Vision and Pattern Recognition, pp. 444-451, 2000.

[29] G. Marola, “On the Detection of the Axes of Symmetry ofSymmetric and Almost Symmetric Planar Images,” IEEE Trans.Pattern Analysis and Machine Intelligence, vol. 11, no. 1, pp. 104-108,Jan. 1989.


[30] D. Milner, S. Raz, H. Hel-Or, D. Keren, and E. Nevo, “A NewMeasure of Symmetry and Its Application to Classification ofBifurcating Structures,” Pattern Recognition, vol. 40, pp. 2237-2250,2007.

[31] D. Mukherjee, A. Zisserman, and J. Brady, “Shape fromSymmetry—Detecting and Exploiting Symmetry in AffineImages,” Philosophical Trans. Royal Soc. of London, vol. 351,pp. 77-106, 1995.

[32] D. Mumford and J. Shah, “Optimal Approximations by PiecewiseSmooth Functions and Associated Variational Problems,” Comm.Pure and Applied Math., vol. 42, pp. 577-684, 1989.

[33] S. Osher and J. Sethian, “Fronts Propagating with Curvature-Dependent Speed: Algorithms Based on Hamilton-Jacobi For-mulations,” J. Computational Physics, vol. 79, pp. 12-49, 1988.

[34] V. Prasad and L. Davis, “Detecting Rotational Symmetries,” Proc.Int’l Conf. Computer Vision, pp. 954-961, 2005.

[35] T. Riklin-Raviv, N. Kiryati, and N. Sochen, “Unlevel-Sets:Geometry and Prior-Based Segmentation,” Proc. European Conf.Computer Vision, vol. 4, pp. 50-61, 2004.

[36] T. Riklin-Raviv, N. Kiryati, and N. Sochen, “Prior-Based Segmen-tation by Projective Registration and Level Sets,” Proc. Int’l Conf.Computer Vision, vol. 1, pp. 204-211, 2005.

[37] T. Riklin-Raviv, N. Kiryati, and N. Sochen, “Segmentation byLevel-Sets and Symmetry,” Proc. IEEE CS Conf. Computer Visionand Pattern Recognition, vol. 1, pp. 1015-1022, 2006.

[38] T. Riklin-Raviv, N. Kiryati, and N. Sochen, “Prior-Based Segmen-tation and Shape Registration in the Presence of ProjectiveDistortion,” Int’l J. Computer Vision, vol. 72, no. 3, pp. 309-328,2007.

[39] T. Riklin-Raviv, N. Sochen, and N. Kiryati, “Shape-Based MutualSegmentation,” Int’l J. Computer Vision, 2007, doi: 10.1007/s11263-007-0115-3.

[40] T. Riklin-Raviv, N. Sochen, and N. Kiryati, “Mutual Segmentationwith Level-Sets,” Proc. Fifth IEEE CS Workshop Perceptual Organiza-tion in Computer Vision, 2006.

[41] M. Rousson and N. Paragios, “Shape Priors for Level SetRepresentation,” Proc. European Conf. Computer Vision, pp. 78-92,2002.

[42] I. Shimshoni, Y. Moses, and M. Lindenbaum, “Shape Reconstruc-tion of 3D Bilaterally Symmetric Surfaces,” Int’l J. Computer Vision,vol. 39, no. 2, pp. 97-110, 2000.

[43] J. Stahl and S. Wang, “Globally Optimal Grouping for SymmetricBoundaries,” Proc. IEEE CS Conf. Computer Vision and PatternRecognition, vol. 1, pp. 1030-1037, 2006.

[44] S. Tari and J. Shah, “Local Symmetries of Shapes in ArbitraryDimension,” Proc. Int’l Conf. Computer Vision, pp. 1123-1128, 1998.

[45] D. Terzopoulos, A. Witkin, and M. Kass, “Symmetry-SeekingModels and 3D Object Reconstruction,” Int’l J. Computer Vision,vol. 1, pp. 211-221, 1987.

[46] S. Thrun and B. Wegbreit, “Shape from Symmetry,” Proc. Int’lConf. Computer Vision, pp. 1824-1831, 2005.

[47] A. Tsai, A. Yezzi Jr., W. Wells III, C. Tempany, D. Tucker, A. Fan,W. Grimson, and A. Willsky, “A Shape-Based Approach to theSegmentation of Medical Imagery Using Level Sets,” IEEE Trans.Medical Imaging, vol. 22, no. 2, pp. 137-154, 2003.

[48] C. Tyler, Human Symmetry Perception and Its ComputationalAnalysis. Lawrence Erlbaum Assoc., 2002.

[49] L. Van Gool, T. Moons, D. Ungureanu, and A. Oosterlinck, “TheCharacterization and Detection of Skewed Symmetry,” ComputerVision and Image Understanding, vol. 61, no. 1, pp. 138-150, 1995.

[50] L. Van Gool, T. Moons, D. Ungureanu, and E. Pauwels,“Symmetry from Shape and Shape from Symmetry,” Int’l J.Robotics Research, vol. 14, no. 5, pp. 407-424, 1995.

[51] A. Vasilevskiy and K. Siddiqi, “Flux Maximizing GeometricFlows,” Proc. Int’l Conf. Computer Vision, pp. 149-154, 2001.

[52] L. Vese and T. Chan, “A Multiphase Level Set Framework forImage Segmentation Using Mumford and Shah Model,” Int’l J.Computer Vision, vol. 50, no. 3, pp. 271-293, 2002.

[53] Y. Yang, K. Huang, S. Rao, W. Hong, and Y. Ma, “Symmetry-Based 3-D Reconstruction from Perspective Images,” ComputerVision and Image Understanding, vol. 99, pp. 210-240, 2005.

[54] Y. Yang, S. Rao, K. Huang, W. Hong, and Y. Ma, “GeometricSegmentation of Perspective Images Based on Symmetry Groups,”Proc. Int’l Conf. Computer Vision, vol. 2, pp. 1251-1258, 2003.

[55] H. Zabrodsky, S. Peleg, and D. Avnir, “Completion of OccludedShapes Using Symmetry,” Proc. IEEE CS Conf. Computer Vision andPattern Recognition, pp. 678-679, June 1993.

[56] H. Zabrodsky, S. Peleg, and D. Avnir, “Symmetry as a ContinuousFeature,” IEEE Trans. Pattern Analysis and Machine Intelligence,vol. 17, no. 12, pp. 1154-1166, Dec. 1995.

[57] H. Zabrodsky and D. Weinshall, “Using Bilateral Symmetry toImprove 3D Reconstruction from Image Sequences,” ComputerVision and Image Understanding, vol. 67, pp. 48-57, 1997.

Tammy Riklin-Raviv received the BSc degreein physics and the MSc degree in computerscience from the Hebrew University of Jerusa-lem, Israel, in 1993 and 1999, respectively, andthe PhD degree from Tel Aviv University, Israel,in 2008. She is currently a postdoctoral associ-ate in the Computer Science and ArtificialIntelligence Laboratory at the MassachusettsInstitute of Technology. Her research interestsinclude computer vision and medical image

analysis. She is a member of the IEEE.

Nir Sochen received the BSc degree in physicsand the MSc degree in theoretical physics fromthe University of Tel-Aviv in 1986 and 1988,respectively, and the PhD degree in theoreticalphysics from the Universite de Paris-Sud in1992, while conducting his research at theService de Physique Theorique at the Centred’Etude Nucleaire at Saclay, France. He con-tinued with a one-year research at the EcoleNormale Superieure in Paris on the Haute Etude

Scientifique fellowship and a three-year US National Science Founda-tion fellowship in the Department of Physics at the University ofCalifornia, Berkeley. It is at Berkeley that his interests shifted fromquantum field theories and integrable models, related to high-energyphysics and string theory, to computer vision and image processing. Hespent one year in the Department of Physics at the University of Tel-Avivand two years on the Faculty of Electrical Engineering at the Technion-Israel Institute of Technology. He is currently an associate professor inthe Department of Applied Mathematics at the University of Tel-Aviv. Hismain research interests include the applications of differential geometryand statistical physics in image processing, computer vision, andmedical imaging. He has authored more than 100 refereed papers injournals and conferences. He is a member of the IEEE.

Nahum Kiryati received the BSc degree inelectrical engineering and the post-BA degree inthe humanities from Tel Aviv University, Israel,in 1980 and 1986, respectively, and the MScdegree in electrical engineering and the DScdegree from the Technion-Israel Institute ofTechnology, Haifa, Israel, in 1988 and 1991,respectively. He was with the Image ScienceLaboratory, ETH-Zurich, Switzerland, and withthe Department of Electrical Engineering of the

Technion. He is currently a professor of electrical engineering at Tel AvivUniversity. His research interests include image analysis and computervision. He is a senior member of the IEEE.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


Documents

On Symmetry, Perspectivity, and Level-Set-Based Segmentation