Nonrigid object motion and deformation estimation from three-dimensional data

Nonrigid Object Motion and Deformation Estimation from Three-Dimensional Data

Chang Wen Chen and Thomas S. Huang Beckman Institute and Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, 405 North Mathews Avenue, Urbana, Illinois 61801

ABSTRACT This article presents a model-based approach for nonrigid object motion and deformation analysis from 3 0 data. The modeling primitives used in this research are the superquadrics, which have already been proven useful in describing a variety of natural and man-made objects. This model-based approach is not only in accordance with the human visual perception process but also able to decouple the large and unstructured nonrigid motion estimation system into simple and well structured subsystems. We develop a recursive algorithm for estimating global motion and object shape, which effectively incorporates a priori knowledge of the object into the estimation procedure and obtains a good estimate of the global motion and object shape even if the given 3D points are distributed bias. After compensating for the global motion of the object, a tensor model of local deformation is introduced and a spherical harmonic surface-fitting algorithm is described such that the localized deformations of the object surface can be characterized. The local deformations of the object are then estimated using tensor-description-based analysis and parametrized by the directions and magnitudes of the extreme deformations in a localized surface element. To illustrate the potential of this model- based approach for nonrigid motion analysis, a real data example is presented using the proposed approach. This example involves estimating the left ventricle motion and deformations from a time sequence of 3D coordinates of coronary artery bifurcation points. The estimation results show the success of the model-based approach even when the given bifurcation points are distributed only on half of the left ventricle surface.

1. INTRODUCTION Among research activities in motion analysis, few have dealt with nonrigid objects. However, since we are surrounded by many natural and man-made objects that are nonrigid, motion and deformation analysis of nonrigid objects is a very important topic and a rich area for research in terms of both theory and applications. Several researchers have already started to investigate some aspects of nonrigid motion analysis, in particular, the motion of elastic objects. Chen and Penna [I] studied the motion of elastic objects and proposed several approaches. Goldgof, Lee, and Huang [2] used Gaus- sian curvatures to analyze nonrigid motion by recovering the stretching factor for the surface. Pentland and Horowitz [3] presented an approach for recovery of nonrigid motion based on the finite element method, which produced an overcon-

Received May 15, 1990; revised manuscript received October 30, 1990

strained estimate using an eigenspace approach. Chen and Huang [4] developed an algorithm for a special problem in nonrigid motion analysis, the estimation of epicardial motion and deformation, by decoupling the global motion and local deformations.

In this article, we describe a model-based approach for nonrigid object motion and deformation estimation from 3D data. Our research is motivated by the desire to utilize a priori shape knowledge of the real object in an algorithm of estimating the motion and deformation. We will discuss why using modeling primitives gives better estimation results and how we can incorporate modeling primitives into the estimation algorithm. Following the line of Ref. 4, the global motion and local deformation of the object are decoupled and, in the ideal case when the complete and unbiased 3D data is available, the global motion estimation can be obtained without the information of the object shape. However, when the data collected is incomplete, as in the usual case, the estimation will be biased. We have developed a recursive estimation algorithm in this article such that a good estimate of the global motion can be obtained by utilizing the a priori shape knowledge of the object. As will be indicated later, this model- based approach is not only capable of handling the incomplete data set but also capable of decoupling the complex estimation process into simpler subsystems. Moreover, it should be noted that this approach is in accordance with the human visual perception, which links the objective environment to the cognitive primitives [5].

The modeling primitives used are the superquadrics. It has been shown that an extensive variety of natural and man- made objects can be accurately represented by superquadrics. Among the key researchers, Barr [6,7] showed that the superquadrics, as a modeling primitive, provide a natural extension to traditional computer-aided design (CAD) models. Based on the work of Barr, Pentland [S] explored the representation of natural objects by superquadrics and developed a real-time system called “Supersketch,” which uses deformations to mold superquadrics into more natural-looking forms. The recovery of superquadric models from 3D information has also been investigated by a number of researchers. Bajcsy and Solina [8,9] designed a modeling system to recover the deformed superquadric models from sparse 3D points based on psychological theories of categorization and of human visual perception. Gross and Boult [lo, 111 compared various error-of-fit measures for recovering superquad-

International Journal of Imaging Systems and Technology, Vol. 2, 385-394 (1990) @ 1991 John Wiley & Sons, Inc. CCC 0899-9457/91/040385-I0$04.00

rics from 3D data and presented an initial comparison of these measures and attempted to find optimal error-of-fit measure for shape recovery. The success of these representation and recovery algorithms indicates that the superquadrics can be effectively employed in the description of many deformable and natural-looking objects. However, the superquadrics have not yet been incorporated into nonrigid motion analysis, in which case an object undergoes global motion as well as local motion and deformation.

Section I1 discusses how the incorporation of modeling primitives to nonrigid motion analysis can solve some of the problems encountered in existing algorithms. The modeling primitive superquadrics will then be introduced and their flexibility and compact representation are shown. Section I11 describes the algorithms for the recovery of global motion as well as object shape in the ideal case. Section IV presents a recursive algorithm for estimating global motion in the case when the given data is incomplete. The estimation of local motion and deformation via a tensor-based approach is given in Section V. We also discuss the estimation of displacement field for the nonrigid motion as well as the stretching tensor for local surface patches. A n example involving left ventricle motion is presented in Section VI using the proposed scheme. The model-based approach improves the accuracy of the estimation scheme proposed in Ref. 4. Section VII concludes with some discussions on future research directions.

II. SUPERQUADRICS AS MODELING PRIMITIVES As indicated by Pentland [3], most previous approaches to nonrigid motion analysis conceptualize nonrigid motion as completely unstructured. Three unknowns are required to describe the completely unstructured motion of each point of a nonrigid object, and therefore the problem of estimating nonrigid motion usually becomes badly underconstrained. To estimate the motion and deformation of the nonrigid object using conventional motion estimation approaches, the shape of the object has usually to be recovered and segmented into “approximately” rigid pieces in order to overcome the underconstrained nature of the problem. Recently, Pentland [3] introduced a finite-element-method-based approach to transform nonrigid motion analysis to an overconstrained problem using an eigenspace approach. However, this approach can only be applied to cases where the object geometry at the initial time instant is known. In the case of nonrigid motion analysis from 3D information, the 3D coordinates of some points on the surface of the nonrigid object over successive time instants may be given. However, these points are generally distributed bias due to limitations of the sensor systems. Therefore, the recovery of object shape is generally difficult using only the incomplete set of 3D points on the surface of the nonrigid object.

In this section, we will first discuss the advantages of using the modeling approach to nonrigid motion analysis. We will then introduce the superquadrics, which allow us to characterize the nonrigid object using relatively few parameters so that the problem of estimating the object shape becomes overconstrained.

A. Advantages of Using Modeling Primitives. A s is well known, perception is the mind’s window on the world. The

perceptual link between the objective environment and the cognitive primitives actually makes our surrounding world meaningful. Therefore visual perception can be viewed as the process of recognizing the environment according to one’s model of the world. Most conventional motion estimation algorithms are based on features extracted from the images without high-level knowledge of cognitive primitives. These algorithms use low-level models of image formation based on the well developed fields of optics, material science, and physics. In contrast, the model-based approach for estimating nonrigid motion and deformation proposed in this paper is in accordance with the human visual perception process since it analyzes the motion and deformation according to the object modeling primitives.

A s we indicated earlier, there are several problems in the existing nonrigid motion analysis algorithms that need to be resolved. The assumption of rigidity allows many researchers to define motion estimation algorithms as the solution to overconstrained system of equations. However, when nonrigid objects are being considered, the lack of rigidity assumption makes motion and deformation estimation a fundamen- tally underconstrained problem. The conventional motion estimation methodology has to be modified such that the problem can be reduced to an overconstrained one. Although there have been some attempts to find the solution to some specific problems, the results are suboptimal because these algorithms adopt rigid approximation to the nonrigid problem. For example, Chen and Huang presented an algorithm in their recent paper [4] that aimed at the relaxation of the requirement that three unknowns are needed for each point of the nonrigid object in order to describe the unstructured motion. Their idea of decoupling the global motion and local deformation is very useful. However, the subsequent estimation of object shape suffered from the limited availability of the 3D points and their inability to utilize the a priori shape information. In many cases, a priori knowledge of an object in the scene is available, but is not utilized because it does not fit the low-level models of image formation from which conventional algorithms are derived. The 3D structure of the object recovered in this way may be quite different from our a priori knowledge of the object, as noticed by Chen and Huang [4].

The need for utilizing a priori shape knowledge also stems from another problem encountered in nonrigid motion analysis; the unavailability of complete and unbiased 3D information due to the limitations of the sensor systems. For most laser range finders, the orientation of the sensor is limited such that the 3D information obtained represents only partial object shape. Although the range data may be acquired for various different orientations for a particular object, the price we have to pay will be the dynamic calibration of the data acquisition system, which is generally computationally and mechanically expensive. In cases where the 3D information is extracted from data obtained from some imaging systems (for example, stereo systems, computed tomography, and NMR imaging) the distribution of these features will not be uniform and may cause faulty recovery of the object shape. However, by developing an estimation algorithm based on modeling primitives, a priori shape information can be reflected in the modeling primitives and object shape can be better recovered.

The advantages of using modeling primitives in nonrigid motion analysis now is clear. First of all, this is in accordance

386 Vol. 2, 385-394 (1990)

with the human visual perception that links the objective environment to the cognitive primitives. Second, utilizing a modeling primitive will generally transform the underconstrained nonrigid motion analysis problem into an overconstrained one. Finally, in cases where only partial 3D information is given, the incorporation of a priori information will improve recovery of object shape.

6. Superquadrics as Modeling Primitives. We have already discussed the advantages of using modeling primitives in nonrigid motion analysis. In fact, there are two major model categories investigated extensively by computer vision researchers: models of image formation and specialized computer-aided design-computer-aided manufacturing (CAD- CAM) models [3]. Various models of image formation have been assumed by researchers in the process of feature extrac- tion and subsequently used in high-level analysis. However, the features so obtained, such as edges and corners, have no link to the object primitives and therefore utilization of only these features in nonrigid motion analysis will lead to an underconstrained formulation, as we have indicated earlier. Most machine vision systems are based on the specialized CAD-CAM models which have proven their success in dealing with a variety of object classes. These models are based on very simple geometric primitives (e.g., cube, sphere, cylinder) that incorporate a priori information of the object. However, these object primitives lack the ability to adapt to object deformation and thus have limited application to nonrigid motion analysis.

Here we propose a modeling primitive that can reflect the a priori shape information of the object while still being flexible enough to catch the deformable nature of the nonrigid objects. This is a parametrized family of shapes known as superquadrics, which have been used for shape representation in computer graphics [6,7] as well as computer vision [5,9, 111. Mathematically, superquadric surfaces are the spherical product of two superquadric curves and can be defined in vector form as follows:

where - a 1 2 5 0 5 7~12 and - rr 5 4 5 a. Parameters 0 and 4 corresponds to latitude and longitude angles expressed in the object-centered spherical coordinate system. Angle 4 lies in the xy plane while 0 corresponds to the angle between the vector S(0, 4) and its projection in the xy plane. Scale parameters a,, a , , u2 define the size of the superquadrics in the x , y , and z directions, respectively. E , is the squareness parameter along the z axis and E~ is the squareness parameter in the xy plane. As indicated by many researchers [ 5 ,7 ,9 ] , by varying these parameters superquadrics can model a large set of standard building blocks, such as spheres, cylinders, paral- lelopipeds, as well as shapes in between.

The modeling power of superquadrics is augmented by the application of various deformation operations to the basic models [12]. In particular, certain classes of deformed superquadrics are able to model nonrigid objects that are nonsym- metric and deformable. Manipulating the components of the vector in Eq. (l), an implicit equation of the superquadrics

can be obtained as follows:

Now we can discuss some typical examples of deformation applied to the basic superquadric model according to the implicit equation. These examples include tapering and axial twist. The deformations can be expressed mathematically as the multiplication by homogeneous, generally spatially varying transformation. Since the matrix multiplications are not in general commutative, the application of multiple operations to a simple superquadric model will be sensitive to the order of application.

Tapering: Tapering, similar to scaling, is the operation of differentially changing the length of two global components without changing the length of the third one. If the point ( x , y, z ) is transformed t o (X, Y , Z ) , the tapering deformation can be written as

(3) Z = z ,

where f , ( z ) and f , ( z ) , the tapering function, are usually piecewise linear functions of z . An object increases its size when the derivative of the tapering function is positive and decreases its size when the derivative is negative. The homogeneous transformation matrix for tapering can be written as

(4) L o 0 0 1 1

Axial Twist: A twist can be approximated as a differential rotation, just as tapering is a differential scaling. The deformation is accomplished by rotating the (x, y) vector components as a function of height, without altering the z vector component. A mathematical definition of twist can be written as

X = cos(cp(z))x - sin(cp(z))y ,

Y = sin(cp(z))x + cos(cp(z))y , z = z ,

( 5 )

where cp(z) is considered to be the differential rotation angle around the z axis and changing at a rate of cp'(z) radians per unit length in the z direction. The homogeneous transformation matrix for twisting deformation can be written as [ cos(;(z)) -sin(cp(z))

sin(cp(z)) cos(cp(z)) 0 0

The basic superquadrics of Eq. (1) along with the transformation matrices of Eqs. (4) and (6) define the parametric model that we use for nonrigid motion analysis. The parameters we need to recover in this model will include three

Vol. 2, 385-394 (1990) 387

rotation parameters and three translation parameters for global motion, six motion parameters and six stretching parame-

translation of the centroid followed by a rotation around an axis through the centroid. This alternative model allows us to _ .

ters for local surface patch, three size parameters and two squareness parameters for basic superquadrics, and additional shape deformation parameters for deformed superquadrics. The total number of the parameters is more than 23. How- ever, the decoupling of the recovery process introduced in the subsequent sections will lead to a series of estimation stages in each of which only a few parameters are involved and an overconstrained estimation problem can easily be formulated.

I l l . RECOVER GLOBAL MOTION AND OBJECT SHAPE Section I1 discussed the advantages of incorporating modeling primitives in nonrigid motion analysis and introduced a deformable superquadric model defined by Eq. (1) and the homogeneous transformation matrices in Eqs. (4) and (6). However, the total number of parameters to be recovered is still quite large and simultaneous recovery of these parameters is difficult. Fortunately, the recovery process can be decoupled in this model-based approach such that only a few parameters are involved at each estimation stage. The decoupling of the recovery process follows the idea of decoupling global motion and local deformation presented in Ref. 4.

We will first introduce an object-centered model for nonrigid object motion which is convenient for shape recovery and global motion estimation. Then a global motion estimation algorithm is developed without utilizing the feature point correspondences and object shape information. A recursive algorithm is presented to ensure better performance of the estimation. Finally, the superquadric modeling primitive is recovered by estimating only the shape parameters, since the position parameters have already been recovered in the process of global motion estimation.

A. Nonrigid Object Motion Model. In their recent paper [4], Chen and Huang pointed out that the motion of the left ventricle can be decomposed into a global motion plus local motions that are spatially smooth. Their observation of the nature of the motion of this particular nonrigid object can in fact be applied to most nonrigid objects. The extension of this idea to the general case is also motivated by the judgement, made by Pentland in his recent paper [3] , that most real objects are in fact made of approximately elastic materials.

Our motion model decomposes the nonrigid object motion into two parts: a global motion that every element of the nonrigid object undergoes, represented by one single set of six motion parameters, and local spatially smooth deformations that every local patch of the nonrigid object experiences differently. We will concentrate on global motion estimation in this section and leave the discussion of local deformation estimation to the next section. Our global motion model treats the object as a rigid object and results in a simple, object- centered estimation algorithm. There are several models for the motion of a rigid object; the most common one is a rotation around an axis through the origin followed by a translation. However, both translation and rotation of this model are in terms of the world coordinate system and therefore are not compatible with an object-centered nonrigid motion analysis. From kinematics, the motion of an object between two time instants can also be represented by a

estimate the object-centered translation and rotation, which can then be directly compensated.

B. Global Motion Estimation in Ideal Case. The previous discussion on the nonrigid object model enables us to develop a global motion estimation algorithm independent of the object shape recovery and local deformation estimation. The judgement presented in Ref. 3 that most real objects are in fact made of approximately elastic materials is the basic assumption in developing such an algorithm. In this article, we will analyze the case where the local deformations of the nonrigid object are mainly the results of uniform expansion or contraction. Then the overall position and orientation changes of the nonrigid object are mainly due to the global motion as we defined. In this section, we limit our discussion of global motion estimation to the ideal case, i.e., the 3D points are uniformly distributed on the surface of the nonrigid object.

Under such assumptions, it is easy to show [4] that the centroid of the feature points on the surface of the nonrigid object does not change with the expansion or contraction. Then the translation of the centroid over time will clearly represent the global translation according to the motion model we have discussed previously. The translation vector estimated through the calculation of the centroid is valid only if the 3D point sets available are distributed uniformly and without bias over the surface of the object. A recursive algorithm will be presented later to get a good estimate of both translation vector and rotation matrix of the global motion when the given 3D points are partially distributed over the surface of the given object.

It is well known in motion analysis of rigid objects that the correspondences between two noncolinear vectors associated with the object over two time instants is sufficient to determine the rotation matrix. The global rotation of the object is obtained by construcing the principal axes from the given set of points on the nonrigid object surface. The principle axes are three orthogonal vectors with their origin at the centroid of the point set and also are approximately invariant to local deformation of the surface if we consider the surface as elastic. The detail of how the principal axes are constructed can be found in Ref. 4. In vector and matrix form, the global motion parameters can be written as

(7)

I , where (x,,,, y,, 2,) and ( x m , y , , 2;) are centroids, and e , , e2 and e',, ek are principal axes of the nonrigid object at two consecutive time instants. Again, the translation vector and rotation matrix so obtained will be inaccurate estimates when a set of incomplete data is given. A recursive algorithm will be developed to get a better estimation of both the translation vector and the rotation matrix of the global motion by incorporating the a priori shape information into the estimation procedure.

388 Vol. 2 , 385-394 (1990)

C. Recovering Superquadrics from 3 0 Data. Assume that we have successfully found the translation vectors and rotation matrices of the global motion over successive time instants. Then the position and orientation of the nonrigid object are known for these different times instants. It is a great advantage in the shape recovery process that we know the position and orientation of the nonrigid object, since now only the size parameters, squareness parameters, and shape deformation parameters of the superquadrics are to be recovered. In recent researches into recovering superquadric model from 3D information [8-121, various error-of-fit measures have been investigated, all leading to nonlinear optimization algorithms. By reducing the number of parameters involved in our nonlinear optimization algorithm for object shape recovery, we reduce the dimensions of search space, reduce the computational complexity, and increase the prob- ability of convergence to the right solution.

Among various optimization schemes in recovering superquadrics investigated by many researchers, the most common one is based on the inside-outside function, defined as

+ ( $fl , ( 9 )

where iff(x,, y , , zo) = 1, then (xo, y o , 2 , ) is on the surface; if f(x,, y , , , z , ) < 1 , then (x,, y o , z o ) lies inside the surface; if f (xo , y , , z,)) > 1, then (xo, y , , zo) lies outside the surface. The objective function for the optimization is defined as

Minimize: I f ( x , y , z ) - 11’ , (10) 1 = 1

where the summation is over all known 3D points. Gupta et al. [12] characterized Eq. (10) as a quantitative

measure of the goodness-of-fit. Qualitative measures are needed for a complete evaluation when the 3D data reflect the irregular object surface or suggest further segmentation. In this article, we will assume that the segmentation process has already been completed and the object surface has no signifi- cant global irregularity. Under such conditions, we will evaluate the superquadric model using only the quantitative measure as indicated above.

We introduce here two modifications to the above-defined objective function [12]. First, the inside-outside function is modified as

+ ( The exponent E, is introduced to avoid the rapid growth of the inside-outside function when the value of El is small, thus enhancing the computational stability. Then, the objective function is modified to choose the superquadric model with the smallest volume. This was motivated by the possibility that there may be many sets of parameters fitting the given 3D data. A modified objective function now becomes:

Minimize: 2 ax a,. a,lf(x, y , 2) - 1)’ . (12)

The parameters we want to estimate in this stage include: size parameters ( a x , a y , a,), squareness parameters (E,, E ~ ) , and shape deformation parameters (if any). Obviously, the number of parameters is much smaller than that in the common approaches to recover superquadrics, which usually need 11 or more [8]. As was reported in Ref. 8, when the recovery of superquadrics involves only shape parameters, or in other words, the position and orientation parameters of the superquadrics are known, the superquadric model parameters can be recovered from very sparse data. This is another advantage o f decoupling the estimation procedure into different stages. The recursive algorithm to be introduced in Sec. IV will enhance the performance of the estimation of all these parameters.

r = l

IV. RECURSIVE ALGORITHM FOR BIAS DISTRIBUTED 3D POINT SETS

We have already stated that the global motion and object shape estimation algorithms presented above are valid only if complete and unbiased 3D point sets are available. The estimate of the centroid of the nonrigid object using only the feature points will be inaccurate if these feature points are partially or bias distributed over the surface of the object. The inaccurate estimate of the centroid will then cause faulty estimation of the object orientation and object shape. There- fore the estimation of the object centroid is critical. In the model-based approach and the recursive algorithm presented in this section, we adjust the estimate of the object centroid based on a priori shape information. We use modeling primitives to incorporate a priori shape information into the estimation algorithm.

A. Combat the Bias Via Modeling Primitives. In Sec. ILA, we indicated that complete and unbiased 3D information is usually unavailable due to the limitations of sensor systems. Fortunately, along with the biased information, we generally have a priori shape information of the object. In this model- based approach, a priori information of the object shape can be incorporated into the estimation procedure, and a recursive algorithm can be developed.

The incorporation of a priori knowled-ge into the estimation procedure clearly depends upon the particular problem we are dealing with since the shape of the nonrigid object is different from one problem to another. Here we illustrate, for a specific example, the ability of modeling primitives to reduce bias; but the methodology can be easily adapted to a broad range of situations. Along with the general knowledge of the object shape, we often have some specific constraints on object parameters, such as the relationship between the size parameters, the possible tapering or twist deformation of the object, and so on. This knowledge will help in choosing the particular model and deciding the direction and magnitude of the adjustment if needed. Suppose we are given that the 3D information is obtained by a range finder or a stereo imaging system. Then the 3D feature points will only be distributed approximately on the half of the object surface facing the image-acquisition device. The geometric centroid calculated from the coordinates of these feature points will be

Vol. 2 , 385-394 (1990) 389

different from the real centroid of the object and therefore the subsequent estimates of principal axes and object shape will be biased. Consider now that a priori information of the object shape is available; namely, that it can be modeled as a superellipsoid with approximately fixed ratios for its size parameters. This a priori knowledge of the object shape can then be recursively incorporated by adjusting the estimated centroid until the estimated shape parameters are compatible with knowledge of the object shape.

One remaining question is how to adjust the estimated centroid based on a priori information. From the construction of principal axes of the data set in the aforementioned imaging system, there exists one principal axis that is pointing towards or away from the center of the surface. Since the a priori symmetry of the desired superquadric model is not evidenced in the 3D data set due to limitations of the imaging device, the centroid estimate can be adjusted along this principal axis to enforce the known model symmetry. Since parameters of an object can be roughly estimated from the centroid and principal axes by calculating the maximum distances between the surface and the centroid along the directions of principal axes. These rough estimates of the size parameters are compared with the a priori knowledge and the decision of adjustment is made based on this comparison. The adjustment schemes are perhaps different from one problem to another depending upon the nature of the imaging system and the a priori knowledge. In many cases, this adjustment process may take more than one step before the estimated object shape and position parameters are compatible with the a priori knowledge. This motivates the recursive algorithm we describe in Sec. 1V.B.

8. A Recursive Algorithm. In Sec. IV.A, we presented a scheme to adjust the biased centroid estimate according to the available shape information. Since the optimal adjustment often cannot be accomplished in just one step, we propose a recursive algorithm that adjusts the estimated parameters until they are in accordance with the a priori knowledge of the object. After the centroid of the object is adjusted according to the a priori knowledge, the principal axes are calculated using the algorithm proposed above. Subsequently, the object shape can be recovered through the nonlinear optimization of the inside-outside function and shape parameters can be obtained. If the error-of-fit measure is smaller than that of the previous recovery, then the adjustment is believed to be in the right direction. The adjustment will continue until the error- of-fit measure stops decreasing. Intuitively, this recursive algorithm for estimating global motion and object shape will converge to the right solution; however, a solid theoretical study of the convergence is still needed. The convergence analysis of the recursive algorithm is currently under investi- gation, but the estimation results presented in Sec. VI show the desired convergence in practical applications.

This recursive algorithm consists of the following steps:

Step 1. Estimate the centroid and the principal axes using the algorithms proposed for the ideal case in which only the given 3D points are used;

Step 2. Recover the global shape of the object using the position (centroid) and orientation (principal axes)

parameters via the superquadric model-based approach and calculate the fitting error;

Step 3. Adjust the position parameters according to the a priori knowledge of the data acquisition limitations and estimate the new orientation parameters;

Step 4. Recover the global shape parameters using the adjusted position and orientation parameters and calculate the fitting error again;

Step 5. Stop the recursive algorithm if the current fitting error is equal to or larger than the previous one; otherwise go to Step 3 and continue the recursive algorithm.

V. ESTIMATION OF LOCAL INFORMATION Our previous discussion of nonrigid motion is based on a model that decomposes the motion into a global motion plus local motions and deformations which are spatially smooth. We note that the local motions and deformations are usually crucial information but are too difficult to estimate without first identifying the global motion and compensating for it. In Sec. 111 and IV, algorithms for estimating the global motion have been developed for the ideal case as well as the biased point distribution case. The compensation of global motion can be easily done by first back-translating the given set of points followed by back-rotating the data set using the estimated translation vectors and rotation matrices for each time instant. After global motion compensation, the object undergoes only local motions and deformations.

A. Local Deformation and Its Tensor Model. According to the well known Helmholz decomposition [13], locally, the motion of a sufficiently small volumetric element of a deformable body can be decomposed into the sum of a translation, a rotation, and an expansion (contraction) in three orthogonal directions. Notice that the translation and rotation here are that of the small element and therefore are different from the global rigid motion of the whole deformable body. For a given reference coordinate system, the mathematical expression of the local deformation based on tensor transformation can be written as

where P, and P , , , are point vectors in the time instants i and i + 1, respectively. T, is a translation vector, R , a rotation tensor, and E, an expansion tensor, all varying with space and time. Nonetheless, for the points within the local small volumetric element, T, , R, , and E, can be considered approximately constant. Furthermore, the rotation tensor R , is orthonormal and the expansion tensor symmetric.

The basic and deformed superquadrics used in the modeling global motion and object shape enforce the axial symmetry of the object shape. However, local deformations of the nonrigid object are not constrained to have this symmetry. Therefore the superquadric models are not good for the local deformation analysis. To characterize the local deformations of the object, a more realistic surface recovery algorithm has to be developed. We apply a tensor model to the local deformation analysis, because this model allows more general local deformations in each localized area.

We will see in Sec. V.D that local tensor analysis requires a dense set of data points in each localized region, with corre-

390 Val. 2 , 385-394 (1990)

spondences between the same points at two time instants. Because the original data is usually sparse, some surface interpolation is required to allow us to estimate tensor parameters. In the following two subsections, we outline a method for using spherical harmonic series for surface interpolation, and establishing point correspondences between surfaces in- terpolated at two time instants.

B. Spherical Harmonic Interpolation of Deformable Sur- face. Because superquadric modeling primitives are not suit- able for local deformation analysis, we present here an alternative model for estimating localized deformations in the shape recovery process. The alternative model represents the local surface by finitely many coefficients of a spherical harmonic series expansion, which exists under the assumption that the surface is smooth and isomorphic to the unit sphere.

We consider the surface defined in a polar coordinate system as

where r ( 4 , 0) is a regular function. Then, as is indicated by Ballard and Brown [14], a finite series of spherical harmonics can be used to represent the surface as follows:

r(4r8)- c ( a"P 0 n (cos 8) N

n - l

+ ( b ~ cos m+ + c~ sin m$)P,,,,,(cos 0)) , (15)

,n = I

where P,,(.) is the Legendre polynomial of degree n and Pn,,,(. ) is the general Legendre function given by

(16) d"'

P n , J X ) = (1 - X*)""* __ dxni P , ( x ) .

If we denote the coefficients of the above approximation by a; and the basis functions by B l ( 4 , 0) , then the approximation will be

M

r ( 4 , e ) = C u , ~ , ( $ , 0 ) . (17) , = I

The least-squares method can be applied to get the coefficients of the interpolation. The number of coefficients M may be large depending upon the order of approximation N . For example, if N =3 , then M = 16. However, it is easy to implement since the coefficients we want to estimate can be obtained by solving a system of linear equation.

C. Establishing Point Correspondences. After the representation of the deformable surface by spherical harmonic interpolation, we need to establish point correspondences between points at two time instants. Since we already have the correspondences between the sparse set of observed points on the surface, we can also interpolate these correspondences by estimating a displacement field which is a function of the surface coordinate variables (4, 0) . Without loss of generality, we can write

where the subscripts refer to the time instants. The functions ~ ( 4 ~ , 0,) and u(&, 0,) can also be approximated by the spherical harmonics as we did for the surface fitting. There- fore the displacement field can be estimated through the proposed least-squares approach for every consecutive time instant.

D. Tensor Analysis of Deformable Surface. Equation (13) gives the general relation between the positions of the local elements of a deformable object and the motion and deformation parameters. Since the rotation tensor is orthonormal and the expansion tensor is symmetric, there are twelve unknowns in Eq. (13). For each 3D point correspondence, three equations can be established to specify the relationship of motion and deformation between two successive time instants using Eq. (13). As we have pointed out in Sec. V.A, the local translation, rotation, and expansion tensors can be considered constant over a localized area. Hence, in order to determine all twelve unknowns, at least four point correspondences within each localized area are needed.

Assuming that the deformable surface is specified by the sample points on the surface and the correspondences of these points over consecutive time instants are given, then the tensor analysis parameters can be estimated over each local surface patch containing at least four points. Equation (13) is nonlinear with respect to the three rotation parameters and no exact analytic solution is known. However, if small angle rotation is assumed, the original nonlinear problem can be transformed into a linear one. The rotation matrix can be approximately expressed as

where a , p, and y are the rotation angles around the x, y , and z axes, respectively, and Eq. (13) reduces to a set of linear equations.

As long as we have enough linearly independent equations, the translation vector, rotation tensor, and deformation tensor can be easily obtained. Singular value decomposition method may be introduced to overcome the possible ill-condition of the system. According to Ref. [13], the eigenvectors of the expansion tensor Ei give the directions of extreme deformation and the corresponding eigenvalues specify the amount of deformation.

To get an overall picture of the object deformation, the local deformation analysis of each localized area has to be organized in such a way that the results can easily be visual- ized. We propose dividing the + and 0 axes into small segments such that the estimated surface at the first time instant is covered with dense quadrilateral meshes. Then the corresponding 4 and 0 values of each vertex of the quadrilateral meshes in the next time instant can be estimated using Eq. (18). The new values of (p and 0 will determine the vertices of the quadrilateral meshes in the subsequent time instant using the estimated surface. The correspondences of the quadrilateral meshes between consecutive time instants are therefore established.

Vol. 2 , 385-394 (1990) 391

VI. EXAMPLE An example is presented here to illustrate the potential of this model-based approach for nonrigid motion analysis. The data consist of the 3D coordinates of 30 coronary artery bifurcation points of the left ventricle detected from 10 pairs of biplane angiogram frames. The details of how these bifurcation points are detected from the angiographic images were reported in Ref. 15. The results obtained by the model-based recursive algorithm are compared with the results obtained using the algorithm described in Ref. 4, in which the motion and deformation are estimated using only the bifurcation points that are bias distributed. The comparison shows that the model-based approach produces estimates of the motion and shape which are more consistent with the a priori knowledge of the left ventricle.

We will first discuss the modeling of the left ventricle as a tapered ellipsoid and the reasons behind this choice. Then we examine the biased distribution of the given 3D bifurcation points and describe the recursive approach for global motion and shape recovery by incorporating the a priori knowledge of the left ventricle to the estimation algorithm. The local deformation of the left ventricle is estimated using the algorithm developed in Sec. V. Finally, we present these estimation results along with the results obtained using the algorithm described in Ref. 4 for the purpose of comparison.

A. Modeling of Left Ventricle as a Tapered Ellipsoid. As we have already pointed out in the above discussion, the choice of modeling primitive depends heavily on the a priori knowledge of the object. The modeling primitive plays an important role in recovering global motion, especially when the 3D feature points are distributed bias. In the case of the left ventricle, the a priori shape knowledge leads us to choose an ellipsoid with tapering deformation to model the object shape. Although the modeling of the left ventricle as a tapered ellipsoid will not catch the localized deformation, it is a good approximation of the global shape. The tapering deformation of the ellipsoid allows us to model the varying cross-sectional areas perpendicular to the long axis of the left ventricle. For simplicity, we make the differential scaling functions the same for two global axes, giving tapering equations

x= (kz + 1)x , Y = ( k z + 1)y ,

Z = z ,

where k is a tapering constant and - 1 la , < k < 1 la, . There are several reasons why we did not choose a more

complex model for the left ventricle. First, the modeling primitive is used in the global motion and shape estimation so that detail modeling of the localized surface is not necessary. Second, even if we choose a more complex superquadric model, the localized deformations will be smoothed out because of its axial symmetry property. Finally, recovering an ellipsoid is much easier and more robust than recovering a superellipsoid because the optimization of exponential parameters is difficult in terms of convergence and stability.

B. Biasedness of the Data and the Recursive Algorithm. Now let us examine carefully the given 3D data, which consist

of coronary artery bifurcation points of the left ventricle. It is obvious that the coronary arteries of the left ventricle encircle the surface of the heart and therefore cover only about half of the left ventricle surface. Hence, the 3D bifurcation points are distribured bias on the surface of the left ventricle. The algorithm for estimating global motion and shape for the ideal case provide unacceptable results using the bias distributed feature points. However, the recursive algorithm developed in Sec. IV, which incorporates the a priori knowledge of the left ventricle shape into the estimation process, can be applied to global motion and shape recovery and achieve better results.

The utilization of a priori knowledge of the left ventricle in the selection of tapered ellipsoid as the modeling primitives has already been discussed. Now we shall illustrate the application of a priori knowledge of data acquisition limitations and the size parameter relations to the recursive algorithm. From the geometric position of the coronary arteries, we know that the adjustment of the centroid in the recursive algorithm has to be away from the side of the left ventricle encircled by the coronary arteries. This side of the surface is identified as the side in which the ellipsoid’s size parameter, estimated from fitting the given data, is the smallest. A left ventricle modeled by a tapered ellipsoid may be elongated along only one of the principal axes. This knowledge will be used to guide the recursive algorithm to choose the direction of the centroid adjustment, and to check the recovered model to see if the algorithm is successful. As we have pointed out before, the error-of-fit measure in the process of recovering ellipsoid will be used as a stop criterion in this recursive algorithm. We shall show later that the results obtained via this recursive algorithm are much better in terms of their comparison with the a priori knowledge of the left ventricle.

C. Local Deformation Estimation. After we estimate the position and orientation of the left ventricle, the global motion of the left ventricle is compensated for in the same way as was done in Ref. 4. However, the global motion parameters estimated through the recursive algorithm are much more accurate. Upon compensation, the surface fitting and displacement field estimation using spherical harmonic series is per- formed. We apply the tensor analysis approach to the local deformation estimation of each local area, and we need four correspondences for each area. Using the estimated parametric surface, the left ventricle surface is divided in terms of 4 and 0 such that the surface is covered with dense quadrilateral meshes. The position of the four vertices of the quadrilateral meshes can be calculated using the estimated displacement field through Eq. (18) and thus the required correspondences are established. The local deformation of each small area is then estimated by solving a system of linear equations derived from Eqs. (13) and (19). The results are the direction and magnitude of extreme deformations corresponding to each small localized area on the surface of the left ventricle.

In the case of surface fitting, the order of approximation N has to be decided empirically in order to achieve a good approximation while reducing the number of coefficients to be estimated. According to Coppini et al. [16], we can take the order of approximation N = 3 to get a satisfactory interpolation for the left ventricle surface. Then the summed squares of the distances from the 30 given bifurcation points to the estimated surface is of the following form:

392 Vol. 2 , 385-394 (1990)

Least-squares methods are applied to get the coefficients of the interpolation.

D. Estimation Results and Discussions. In this section, the estimation results obtained via the model-based approach are presented along with the results estimated without the recursive adjustment. The estimation results of object centroid is shown in Table I and the estimation results of object orientation is shown in Table I1 for each individual frame of the image sequence; both of them are with respect to the world coordinate system. The estimation results of both centroid and orientation cannot be used to compare with the a priori knowledge of the object, because they are not directly related to the shape of the object. However, the estimation results of the object shape derived from these estimated centroid and orientation reflects the goodness of these estima- tions when they are compared with the a priori shape knowledge of the object. The comparison of the estimated shape parameters with and without recursive adjustment is presented in Table 111.

Table I11 indicates that the size parameters estimates derived directly from the feature points are incorrect because we know that the aspect ratio of the left ventricle cross section is not so elongated. However, the estimates derived from the recursive algorithm reflect the success of the model-based

Table I. The estimation results of object centroid.

approach because the size parameters estimated in this way are compatible with the a priori knowledge of the left ventricle and the estimated tapering parameter is consistent over the subsequent frames.

The numerical results of estimated local deformation is difficult to present because the left ventricle surface is com- posed of many pieces of local quadrilateral patches and each would be quantified by three scalars and three vectors of extreme deformation values and directions. One way of analyzing and understanding the estimated results is to ani- mate the moving surface and its quadrilatral meshes using scientific visualization techniques. The detail of the animation is very lengthy and will be discussed in a future paper.

VII. CONCLUSIONS We have described a model-based approach for nonrigid motion and deformation estimation. The major contributions o f this research include the introduction of superquadric modeling primitives to nonrigid motion analysis and the de- velopment of a recursive algorithm for the global motion estimation. By incorporating the superquadric model, the seemingly unstructured nonrigid motion analysis problem can be converted into a well structured and overconstrained problem. The recursive algorithm presented in Sec. IV enables us to utilize a priori knowledge of the object shape so that global inotion can be estimated even if the given 3D data is distributed bias. This model-based approach was applied to a real data example, the estimation of left ventricle motion and deforma-

Estimated obiect centroid

Frame

No recursive adjustment

x, Y , 2,

1 2 3 4 5 6 7 8 9

10

32.140 32.804 32.966 31.995 32.574 32.701 32.732 33.018 33.116 33.427

47.569 46.664 45.462 45.324 43.953 43.364 42.507 42.399 42.667 43.267

58.714 58.430 58.163 59.577 56.940 57.192 56.811 57.251 57.231 57.698

x m

29.910 30.874 31.096 30.252 31.120 30.984 30.939 31.004 31.398 31.551

Recursively adjusted

Y ,

50.599 49.430 48.582 48.465 47.363 46.760 45.928 45.821 46.222 46.940

47.319 46.914 46.728 48.127 45.526 45.811 45.449 45.927 45.898 46.429

Table 11. The estimated results of obiect orientation.

Estimated object orientation

Frame el e2 e ,

1 (-0.5575,0.7687,0.3135) (0.8091,0.5877,0.0021) (0.1859, -0.2525,0.9496) 2 (-0.5579,0.7808,0.2811) (0.8142,0.5806,0.0031) (0.1608, -0.2305,0.9597) 3 (-0.5268,0.7942,0.3028) (0.8356,0.5492,0.0132) (0.1558, -0.2599,0.9529) 4 (-0.4898,0.8188,0.2992) (0.8596,0.5108,0.0093) (0.1453, -0.2618,0.9541) 5 (-0.4874,0.8176,0.3064) (0.8647,0.5006,0.0394) (0.1211, -0.2842,0.9511) 6 (-0.4812,0.8174,0.3165) (0.8649,0.5016,0.0192) (0.1431, -0.2830,0.9483) 7 (-0.4725,0.8205,0.3216) (0.8686,0.4954,0.0121) (0.1494, -0.2850,0.9468) 8 (-0.4706,0.8180,0.3308) (0.8662,0.4996,0.0031) (0.1678, -0.2851,0.9437) 9 (-0.4698,0.8195,0.3282) (0.8711,0.4906,0.0219) (0.1431, -0.2962,0.9443)

10 (-0.4608,0.8183,0.3435) (0.8736,0.4864,0.0131) (0.1564, -0.3061,0.9390)

Vol. 2 , 385-394 (1990) 393

Table 111. The comparison of estimated object shape parameters.

Estimated object shape parameters

No recursive adjustment Recursively adjusted

Frame a, a, a: k a, a Y a, k‘

1 58.040 39.699 17.766 - 0.0005 58.037 39.735 28.943 0.0044 2 55.732 41.145 18.097 0.0014 55.726 41.152 29.192 0.0039 3 55.804 39.206 16.839 0.0005 55.742 39.186 28.804 0.0037 4 54.651 37.581 16.322 0.0002 54.653 37.609 28.396 0.0036 5 53.636 36.574 16.332 - 0.0003 53.640 36.602 28.387 0.0026 6 53.338 35.270 16.367 0.0002 53.347 35.306 28.445 0.0028 7 52.618 35.315 16.833 0.0002 52.625 35.352 28.886 0.0027 8 52.853 35.977 18.036 0.0004 52.814 35.969 30.008 0.0028 9 52.566 35.974 17.301 - 0.0003 52.567 36.006 29.352 0.0030

10 53.465 37.249 16.576 0.0001 53.469 37.287 28.658 0.0031

tion from coronary artery bifurcation points, which are distributed bias over its surface. The estimation results are in accordance with our a priori knowledge of the left ventricle, showing the successful application of the model-based approach.

We discussed how local deformation analysis of a nonrigid object cannot be based on the superquadric modeling primitives because of the axial symmetric property of the model. We developed an alternative by interpolating spherical harmonic expansions. To obtain an accurate surface fit, a dense data distribution on the surface of the nonrigid object is needed. Thus the results based on sparsely distributed coronary artery bifurcation points leave much to be desired. Currently, we are also working on the motion and deformation estimation of the left ventricle using computed to- mographic data. The volumetric nature of the data provides us with accurate shape description of the left ventricle. How- ever, a difficult problem is to estiamte the 3D data flow field to establish the correspondences of features over consecutive time instants.

ACKNOWLEDGMENTS This work was supported by National Science Foundation Grant No. IRI-89-08255. The authors wish to thank L.D.R. Smith of IBM UK Scientific Center for providing the angiographic data for the example in Sec. VI and M. Orchard and D. Goldgof for various, discussions.

REFERENCES S. S. Chen and M. Penna, “Shape and motion of nonrigid bodies,’’ Comput. Vision Graphics Image Process. 36, 175-207 (1986). D. Goldgof, H. Lee, and T. S. Huang, “Motion analysis of nonrigid surfaces,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Ann Arbor, MI, June

A. Pentland and B. Horowitz, “Recovery of nonrigid motion and structure,” MIT Media Lab, Vision and Modeling Group, Tech- nical Report No. 136, March, 1990 (unpublished). C.W. Chen and T.S. Huang, “Epicardial motion and deformation estimation from coronary artery bifurcation points,” in Proceed-

1988, pp. 375-380.

ings of the Third International Conference on Computer Vision, Osaka, Japan, December 1990 (IEEE Computer Society Press, Los Alamitos, 1990) pp. 456-459.

5. Alex P. Pentland, “Perceptual organization and the representation of natural form,” Artv. Intell. 28, 293-331 (1986).

6. Alan H. Barr, “Superquadrics and angle-preserving transforma- tions,” IEEE Compur. Graphics Appl. 1, 11-23 (1981).

7. Alan H. Barr, “Global and local deformations of solid primitives,” Comput. Graphics 18, 21-30 (1984).

8. Franc Solina and Ruzena Bajcsy, “Shape and function,” in Intelligent Robots and Computer Vision, Proceedings of SPIE Vol. 726, edited by D.P. Cassent (SPIE, Bellingham, 1986), Vol. V,

9. Ruzena Bajcsy and Franc Solina, “Three-dimensional object representation revisited,” in Proceedings of the First Infernatzonal Conference on Computer Vision, London, England, June 1987 (IEEE Computer Society Press, Washington, D.C., 1987) pp.

10. Terrance E. Boult and Ari D. Gross, “Recovery of superquadrics from 3D information,” in Intelligent Robots and Computer Vi- sion, Proceedings of SPIE Vol. 848, edited by D.P. Cassent and E.L. Hall (SPIE, Bellingham, 1987), Vol. VI, pp. 358-365.

11. Ari D. Gross and Terrance E. Boult, “Error-of-fit measures for recovering parametric solids,” in Proceedings of the Second Inter- national Conference on Computer Vision, Tampa, FL, December 1988 (IEEE Computer Society Press, Washington, D.C., 1988) pp. 690-694.

12. Alok Gupta, Luca Bogoni, and Ruzena Bajcsy, “Quantitative and qualitative measures for the evaluation of the superquadric models,” in Proceedings of the Workshop on Interpretation of 3 0 Scenes, Austin, TX, November 1989 (IEEE Computer Society Press, Los Alamitos, 1989) pp. 162-169.

13. A . Sommerfeld, Mechanics of Deformable Bodies (Academic, New York, 1950).

14. D. H. Ballard and C. M. Brown, Computer Vision (Prentice Hall, Englewood Cliffs, New Jersey, 1982).

15. D.J. Stevenson, L.D.R. Smith, and G. Robinson, “Working towards the automatic detection of blood vessels x-ray angiog- rams,” Pattern Recognition Lett. 6 , 107-112 (1987).

16. G. Coppini, M. Demi, G. D’Urso, A. L‘Abbate, and G . Valli, “Tensor description of 3D time-varying surfaces using scattered landmarks: An application to heart motion,” in Time-Varying Image Processing and Moving Object Recognition, edited by V. Cappellini (Elsevier, Amsterdam, 1987).

pp. 284-291.

23 1-240.

394 Vol. 2 , 385-394 (1990)

Documents

Nonrigid object motion and deformation estimation from three-dimensional data