On-line Precomputation Algorithm for Real-time Haptic Interaction with

Non-linear Deformable Bodies

Jirı Filipovic∗ Igor Peterlık† Ludek Matyska‡

Masaryk University

Czech Republic

ABSTRACT

Soft tissue modelling is important in the realm of haptic interac-tions. The main challenge in this research area is to combine twobasic conditions which are essential—the stability of the haptic in-teraction running on high refresh rate on one hand and realistic be-havior of the tissue assuming computationally expensive mathemat-ical models on the other.

In this paper, a distributed algorithm addressing this challengeis presented. The algorithm is based on the precomputation-interpolation scheme when the force feedback for the actual posi-tion of the haptic interaction point (HIP) is computed by a fast inter-polation from precomputed data running inside the haptic loop. Thestandard precomputation-interpolation scheme is modified, how-ever, so that the data needed for the interpolation are generated di-rectly during the interaction.

The distributed algorithm and the underlying architecture arepresented together with a prototype implementation. Preliminaryevaluation of the algorithm using non-linear models of a humanliver having more than 1700 elements shows the feasibility of theproposed approach.

1 INTRODUCTION AND RELATED WORK

One of the promising areas of research is haptic rendering of de-formable bodies which enables the user to interact with virtual ob-jects that gets deformed according to applied forces. The main ap-plication of this research is represented by construction of surgicalsimulators which become an important technology utilized in theprocess of medical education and training.

Due to the high sensitivity of human touch, loop of the hapticinteraction requires very high refresh rates exceeding 1 kHz. Thisis the main issue of haptic rendering, since such a high frequencymakes any expensive real-time computation of the haptic scene im-possible. On the other hand, the behavior of the deformable objectsis expected to be realistic, i. e. modeled according to the physicallaws.

The realistic behavior of soft tissues is based on the theory ofelasticity which produces the governing equations describing thedeformations under the applied forces or prescribed displacements.Considering the behavior of particular organs, the irregularity ofthe geometry together with the complexity of the governing equa-tions lead to the inevitability of numerical solution. Moreover, itturns out that for realistic haptic modelling of deformable objects,non-linear should be utilized [1]. However, the numerical solutionusually supplied by advanced methods as non-linear finite elementscannot be constructed within the haptic loop due to the computa-tional complexity.

∗e-mail: fila@ics.muni.cz†e-mail:peterlik@ics.muni.cz‡e-mail:ludek@ics.muni.cz

There have been several attempts to address this issue such as sim-plification of the underlying models or employing some precompu-tations before the real interaction occurs. In [2], linearized modelis used with precomputation of elementary displacements. Further,non-linear model is proposed with the drawback of decreased re-fresh rate in [3]. Linear model together with haptic interaction is de-scribed in [4] based on small area paradigm. The model with non-linear geometry is described in [5] when techniques as mass lump-ing are applied but again only low-speed refresh rate is achieved.In [6] both geometrically and physically non-linear model is pro-posed, but no results about the speed of computations are given. Re-cently, in [7] a new approach based on precomputation is presented.During the precomputation coefficients of polynomials derived forSt.Venant-Kirchhoff material law are calculated. Although this ap-proach allows the interaction with complex body having non-lineargeometry properties, it is applicable just for one material type andcannot be used for other material models.

The main contribution of the paper is in introducing an algorithmenabling the haptic interaction with non-linear physically basedmodels without a time-consuming precomputation phase. It canbe regarded as a extension of the approach proposed in [8] based onapproximation of the actual deformation from some regular data.The main difference here is that the data needed for the interpo-lation are generated directly during the interaction by distributedcomputational resources.

The paper is organized as follows. First, the mathematical modelof deformations is briefly presented together with the two-level it-erative solution method in section two. The third section representsthe essential part of the paper providing the description of the on-line precomputations algorithm on both the abstract and detailedlevels. The fourth section briefly presents the prototype imple-mentation and computations together with the description of themodel used in the experiments. The evaluation section demon-strates feasibility of the proposed algorithm showing preliminaryresults achieved with the prototype implementation. Finally, thepaper is concluded and future directions are sketched.

2 MATHEMATICAL BACKGROUND

In this section, the mathematical model used for the deformationmodelling is briefly presented together with the short descriptionof the numerical solution based on a combination of incrementalstaging and Newton method.

2.1 Mathematical model and Solution Methods

In this section, a summary of the underlying mathematical model ispresented. The corresponding formulations can be found in [9]. Asthe realistic behavior of the model is the main goal of the design,the model is completely based on the theory of elasticity where therelation between the applied forces and deformation of the body isderived.

The full non-linear Green strain tensor is used as the measureof the strain allowing the large deformations. The formulation isfurther based on static equilibrium between external forces and in-ternal stresses. The central part of the mathematical formulation is

Third Joint Eurohaptics Conference and Symposium on Haptic Interfacesfor Virtual Environment and Teleoperator SystemsSalt Lake City, UT, USA, March 18-20, 2009

978-1-4244-3858-7/09/$25.00 ©2009 IEEE 24

the definition of constitution law which determines the behavior ofthe material coupling the stress and strain tensors. In this paper,hyperelastic materials are considered for which stored energy func-tion exists. Two distinct definition of the stored energy function areutilized, known as StVenant-Kirchhoff and Mooney-Rivlin materiallaws. Additionally, for Mooney-Rivlin material the incompressibil-ity condition are imposed.

Besides the physical model, the boundary conditions are nec-essary for finding the static solution of the system. The model isfirst fixed in the space by homogeneous Dirichlet conditions. Asthe domain of the deformable body is usually complex and irregu-lar, there is no analytical solution for the governing equation of themodel and numerical methods must be applied.

Perhaps the most known and widely used method is the finite el-ement method (FEM) which discretizes the domain into a 3D meshwhich is given by a set of nodes and elements. The method reformu-lates the governing equations over the regular subdomains of eachelement. Detailed description of the method is beyond the scopeof this paper (see [9] for general FEM and [10] for the applicationin soft tissues). It is sufficient to say that in the case of static non-linear model with N nodes, the utilization of the FEM results in asystem of non-linear algebraic equations

A(u) = b (1)

where A is N×N stiffness matrix, b is a vector representing appliedloads in nodes, and the vector u now contains the displacement ofeach node.

In this paper, following scenario is considered. As thedisplacement-driven interaction is considered, the “input” to the so-lution method is represented by a displacement prescribed for a par-ticular surface node which is denoted as active throughout the text.This can be treated as non-homogeneous Dirichlet condition whichcan be imposed by multiple techniques as penalization, eliminationand Lagrange-multipliers. As the force acting at the displaced nodeis of interest (as the output of the method delivered to the hapticdevice), the Lagrange multipliers are utilized leading to the aug-mented version of Eq. 1 described in [8].

Further, in order to solve the augmented system, incrementalstaging together with Newton iterative method is utilized for com-putations of static equilibrium paths. Suppose that the system isactually in a state C1 corresponding to the position x1 of the activenode. The state C1 contains the reacting force f1 and the overalldeformation vector u1. Now, the active node is moved to a newposition x2. In this case, the trajectory x1 → x2 is divided into nsteps

x1 = x0→ x

1→ . . . → x

n = x2 (2)

and the corresponding configurations Ci are constructed resulting in

a static equilibrium path. On the local level, the state Ci+1 is com-

puted from the state Ci by a Newton method, when the displace-

ment vector ui and force vector fi are taken as the initial estimations

for the solution of the system 1 with the prescribed displacement ofthe active node set to x

i+1.The method described above can be regarded as a two-level tech-

nique when the convergence of the correcting Newton method isimproved by the incremental loading.

3 ON-LINE COMPUTATION ALGORITHM

In this section, the real-time algorithm for on-line precomputationsis presented. First, the motivation for the proposal is given and then,the principle and detailed description are shown, respectively.

3.1 Motivation for the On-line Precomputations

In the following text, a single point displacement-driven interactionis considered when the HIP is fixed to one surface node (the activenode as introduced in section 2.1) of the FE mesh. As the point

gets displaced from its rest position, the body gets deformed andthe corresponding reaction force is delivered to the haptic device.

The main idea of the precomputation-interpolation scheme in-troduced in [8] is that the reaction force and deformation for theactual displacement of the active node are calculated by a fast inter-polation method from precomputed data. The scheme consists oftwo separated phases: during the off-line phase, a large set of staticdeformations is computed according to the space-division schemewhen an active node is displaced to a set of positions organized in3D regular grid and the corresponding forces and deformations arecomputed and stored for each a point of the 3D grid. Then, the in-teraction phase takes place. Here, in each step of the haptic loop,the actual position of HIP is identified with respect to the 3D gridby selecting the adjacent points of the grid and both forces and de-formations are computed by the interpolation of the data stored inthose points.

As the expensive iterative calculations are done in the off-linephase, the calculations needed for the interpolation can be run in thehaptic loop running on a high frequency and the interaction is stableand can be performed on single PC with haptic device attached toit. On the other hand, it is not clear in advance which deformationswill be really needed during the interpolation phase, as this is de-termined by the motion of HIP by the user and so the entire statespace must be precomputed to cover all configurations potentiallyneeded in the future. Further, there are multiple parameters affect-ing directly the behavior of the deformable body. The first set of theparameters is connected to the material model, e. g. the type of thematerial and material coefficients or external forces applied on themodel. The second set is related to geometrical model and user’schoice during the interaction. First, it is the selection of the fixednodes (which can be given or specified by the user) and second, itis the selection of the active node. Both are important factors deter-mining the behavior of the model. It is clear, that if an interactionwith model defined with some combination of the parameters is ofinterest, the entire state-space must be constructed for each combi-nation of parameters which is to be used during the interaction. Eg.the size of the dimensions of the parameter space represented bythe selection of N fixed states on the surface of the body is 2N . Fur-ther, another dimension is given by the selection of the active nodeand finally, the last three dimensions are introduced by the positionwithin the 3D grid.

Addressing the issues described above, a novel approach basedon on-line precomputations is presented in this paper. In summary,the precomputation phase is performed directly during the interac-tion, so that only data needed for the approximation of the defor-mations for the actual position of the HIP are calculated and imme-diately used for the interpolation. This means that we perform theon-line computations of the configurations, which are going to beneeded for the interpolation of the force and deformation vector forthe probe positions in the on-coming moments.

In the following, there are three main conditions which must befulfilled for this approach to be feasible.

• The computational resources must give enough performanceto compute the deformations on-line.

• The speed of motion of the haptic device must be slowenough, so it does not run off the space which has been al-ready precomputed.

• The physical models of the deformations must be stableenough, so the numerical methods converges in small num-ber of iterations as this numerical stability of the model deter-mines the haptic stability of the interaction.

Before presenting the principle of the algorithm and distributedarchitecture designed to address the conditions given above, advan-tages and disadvantages of the approach are briefly sketched.

25

26

27

Figure 3: Model of human liver used in experiments.

5 EVALUATION AND RESULTS

5.1 Methodology of Evaluation

In this section a methodology of the experiments with the prototypeimplementation are presented. The evaluation of the experimentscopes with conditions from the section 3.1. Let’s first identify thefactors which are to be studied when evaluating the algorithm. Thefirst factor which plays a crucial role is how fast the deformationsare computed by the server side. In the two-level-parallelism ar-chitecture being considered in this paper, the transition C1 → C2

needed for the computation of a new state required by the sched-uler is calculated sequentially by one solver thread.

Let’s us assume that there is enough servers to compute all thestates simultaneously, i. e. the scheduler never sends a work to aserver which is not free in that moment (this assumption will beaddressed bellow). In that case, the time needed for the computa-tion of the new state establishes the upper bound on the speed ofthe computations. This factor is given by two quantities fully deter-mined by the mathematical model and FE mesh. The first, which isconstant for the given mesh and material, is the total time needed forassembling and solving the linearized system in each Newton iter-ation. The second quantity which can vary over the space of defor-mations for the given mesh and material is the total number of theNewton iterations needed to attain the desired residual. Therefore,in the result section, the data showing the convergence propertiesfor the particular mesh and material models are presented togetherwith the times needed for the assembling and solving the linearizedsystem.

Another factor affecting the algorithm is represented by the sizeof the cloud being utilized by the scheduler. Since the sphericalcloud is assumed, its size is given by its radius. Since the assump-tion about the distribution of the work among the free servers for-mulated above must hold, the number of servers must be equal orlarger than the size of the cloud for a given radius. For this reason,the result section presents the relation between the size of the cloudand the number of processors.

Finally, the last but extremely important factor which is evalu-ated is the speed of HIP motion. Clearly, the goal of the algorithm isto maximize the speed of this motion in any direction. Initially, theworst case concerning the direction of the motion when the speeddirectly depends on the speed of computations should be identified.First, let’s assume, that a computation of a new state S is started intime t0. Let’s denote t1 the time when S is requested by the inter-polator since HIP has already entered the cell having S as one of itscorners. Further, let l denotes the length of the trajectory which wascovered by the HIP between t0 and t1. In the worst case, the differ-ence t1 − t0 should be minimized for arbitrary speed. Obviously,this is true when the length l is minimal, i. e. the HIP moves alonga straight line. Moreover, a small study shows that in this case themotion of HIP is limited by the speed of the motion of the cloud, i. e.how fast a new states join the set of computed states. However, thisis again limited by the length of calculation of a single deformationas shown above. Putting it together, after leaving the initial cloud,

the HIP moving along a straight line is limited by the speed, howa new deformations are constructed in the direction of the HIP mo-tion. It should be noted, that in case of irregular (random) motionof HIP, that as far as the HIP stays in the cloud computed before,it can move freely with no restriction on the speed. Since naturalmotion of the HIP handled by a human hand can be described bysmooth curves, it is important to use larger clouds allowing betterbehavior for the random motion.

In the result section, the maximal speed of HIP moving alongthe straight line is recorded for the mesh and FE models used in theexperiments. The speed after leaving the initial cloud is considered,so the size of the initial cloud cannot artificially improve the results.

5.2 Results and Discussion

First, the convergence results are briefly presented for both mate-rials. The Tab. 1 shows the length of assembly and solution pro-cedures for both materials. Second, during the large convergenceevaluation when more than 100,000 deformation were computed inoff-line experiments generating large deformations (10 cm in eachdirection) for two distinct sets of fixed nodes and various selec-tions of the active node, the limitation of the mesh in combina-tion with the particular materials was studied. The convergenceproblems were identified by occurrence of failure points where theconvergence was not achieved anymore. The failure points can betherefore regarded as “boundaries” determining the parts of statespace where the deformations can be safely computed for any po-sition of HIP. The more detailed study of the failure points is be-

Material StVenant Mooney

Assembly time [s] 0.21 0.51

Solution time [s] 0.03 0.11

Line Search time [s] 0.05 0.14

Occurrence of failure points [%] 19.7 1.4

Average number of iters. 2.4 4.2

Maximal number of iters. 10 14

Table 1: Quantities characterizing the deformation computations.

yond the scope of this paper and so we only mention that Mooney-Rivlin material shows better behavior that StVenant-Kirchhoff andso it allows bigger amount of configurations to be computed withlarger magnitude of the deformation. For both materials, the av-erage number of Newton iterations needed for achieving the con-vergence was measured as well and the results are also reported inTab. 1. From this point of view, both materials can be used for theexperiments, provided the HIP will be travelling in those parts ofthe state space where the convergence in guaranteed. Second, therelation between the number of server processes needed for varioussizes of the cloud; the results are presented in Tab. 2. The important

Cloud size [cm] 2.0 2.5 3.0 3.5

#processes needed for StVenant 15 31 37 57

#processes needed for Mooney 15 35 44 56

Table 2: The relation between the size of the cloud and numberof solver processes needed for each material to keep the maximalspeed.

point here is that the table shows the number of processes whichare needed to deliver the computations on time according to the as-sumption from the section 3.1. However, as the HIP moves, thenumber of processes actually computing the deformations varies.For the case when the StVenant-Kirchhoff material is used with the

28

radius 2.5 cm, the history of computational load is depicted by thegraph 4 illustrating 40 seconds of the computations. The computa-tions started in time t = 0, whereas the HIP was moving linearly ap-proximately between 15th and 33rd second. During the first sevenseconds of the computations, the initial cloud was computed usingthe maximal number of CPUs available. Nevertheless, the user doesnot need to wait until all the configurations in the cloud are avail-able; only adjacent configurations around the initial position arenecessary for initial interaction. After the motion of HIP started,new states were scheduled for computations. During the motion,the load was varying periodically according to the motion of thecloud.

Finally, the maximal speed of HIP achieved for the linear mo-tion along a straight line are presented. In case of the Mooney-Rivlin material, maximal speed of 3.28 mm/s was experimentallyachieved. In case of StVenant-Kirchhoff material the maximalspeed was 5.25 mm/s. The better result for the StVenant-Kirchhoffmaterial is obviously given by the shorter time needed for the com-putation of a single deformation as shown in the Tab. 1. However, asexplained in the section 5.1, the linear motion is the worst case, sofor an irregular motion (which is cycling or moving around) muchhigher speeds can be achieved, as the states are already computedaccording to the cloud-based scheduling.

In the following, the speeds worst-case maximal speeds are an-alyzed. In [12], the two speeds are used for analysis of the hap-tic perception of forces. Namely, the slow speed was defined as14 mm/s and fast speed was defined as 28 mm/s. When comparingto the speeds achieved in our simulations, the HIP motion for themore complicated material is still 9× slower if fast speed is con-sidered. There are at least two solutions how to overcome this gap.First, during the real haptic interaction, the user can first slowlytravel with HIP, letting the algorithm to do the computation of thepart of state space being visited. Then, after the sufficient numberof states is computed, the speed of the motion is no more limitedby the algorithm. The other possibility that is now being studied,is to introduce another parallelization level. In this case, the com-putation of a single deformation would be parallelized using 4 or 8cores on computational nodes utilized for experiments. Since theassembly process which requires the highest load is embarrassinglyparallel, the speed-up should be sufficient to increase the speed ofcomputations significantly.

6 CONCLUSIONS AND FUTURE WORK

In this paper, a distributed algorithm based on on-line precomputa-tions has been presented. It overcomes the gap between the high re-fresh rate of the haptic loop and computationally demanding calcu-lations needed for realistic non-linear deformation modelling. Themain advantage of the algorithm is that no time consuming off-line precomputation phase is need anymore, as the deformationsare constructed on-line during the interaction. The distributed al-gorithm was described and evaluated showing its performance andlimitations. For this purpose, a prototype implementation was de-veloped and tested on a non-linear model of human liver containingmore than 1700 elements.

During the experiments, the main limitation of the algorithmwas identified as the speed of computation of a single deformationwhich directly determines the maximal allowed speed of the HIP.Moreover, if the HIP moves along a straight line, the speed cannotbe improved by employing more computational resources.

Thus, the future work will be mainly focused on addressing thisissue. First, a new level of parallelization will be added to speed upthe computation of a single deformation and some more advancedscheduling strategies will be studied, e. g.employing motion pre-diction or prioritizing and preempting the computations of states.Further, since our approach enables us to change parameters of themodel directly during the haptic interaction, we plan to focus on

0 5 10 15 20 25 30 35 400

10

20

30

time [s]

# p

roce

sse

s

Figure 4: Number of solver processes used during a linear motion ofHIP for StVenant-Kirchhoff material with cloud of size 2.5 cm.

topological changes such as cutting and tearing, and more complexinteraction with the tissue (e.g. multiple point interaction or contactmodelling).

ACKNOWLEDGEMENTS

The research has been supported by research intent “Integrated Ap-proach to Education of PhD Students in the Area of Parallel andDistributed Systems” (No. 102/05/H050). The access to theMETACentrum computing facilities provided under the research in-tent MSM6383917201 is acknowledged.

REFERENCES

[1] S.Misra, A.Okamura and K.T.Ramesh. Force Feedback is Notice-

ably Different for Linear versus Nonlinear Elastic Tissue Models In

proc.Second Joint EuroHaptics Conference and Symposium on Haptic

Interfaces, 2007,519–524.

[2] S.Cotin, H.Delingette and N.Ayache. Real-time elastic deformations

of soft tissues for surgery simulation. IEEE Trans.On Visual.and

Comp.Graph., 5(1):62–73, 1999.

[3] G.Picinbono, H.Delingette and N.Ayache. Non-linear and anisotropic

elastic soft tissue models for medical simulation. In IEEE Int. Conf.

Robotics and Automation 2001

[4] D.C.Popescu and M.Compton. A model for efficient and accurate

interaction with elastic objects in haptic virtual environments. In

GRAPHITE’03: Proc.of the 1st int.conf. on Computer graphics and

interactive techniques 2003,245–250.

[5] Y.Zhuang and J.Canny. Real-time simulation of physically realistic

global deformation. In IEEE Vis’99, 1999.

[6] X.Wu, M.S.Downes, T.Goktekin and F.Tendick. Adaptive nonlinear

finite elements for deformable body simulation using dynamic progres-

sive meshes. In EG 2001 Proceedings,

[7] J.Barbic and D.L.James. Real-Time Subspace Integration of St.Venant-

Kirchhoff Deformable Models. In ACM Transactions on Graphics

(ACM SIGGRAPH 2005), 2005, 24(3):982–990.

[8] I.Peterlık and L.Matyska. An Algorithm of State-Space Precomputa-

tion Allowing Non-linear Haptic Deformation Modelling Using Finite

Element Method. In Proceedings of the Second Joint EuroHaptics Con-

ference and Symposium on Haptic Interfaces, 2007,231–236

[9] J.T.Oden. Finite Elements of Nonlinear Continua, McGraw Hill, 1972

[10] M.Bro-Nielsen. Finite element modeling in surgery simulation Pro-

ceedings of the IEEE 1998, 86(3):490–503.

[11] E.Samur, M.Sedef, C.Basdogan, L.Avtan and O.Duzgun. Robotic In-

denter for Minimally Invasive Measurement and Characterization of

Soft Tissue Behavior In Medical Image Analysis 2007, 11(4):361–373.

[12] X.-D.Yang, W.F.Bischof and P.Boulanger. The Effects of Hand Mo-

tion on Haptic Perception of Force Direction. In Lecture Notes in Com-

puter Science, 5024/2008, 2008.

[13] J.Pommier and Y. Renard. Getfem++, an open source generic C++

library for FEM, http://www-gmm.insa-toulouse.fr/getfem.

[14] P. R. Amestoy, I. S. Duff and J.-Y. L’Excellent. Multifrontal parallel

distributed symmetric and unsymmetric solvers. In Comput. Methods

in Appl. Mech. Eng., 184, 501-520 (2000).

29