Knowledge of soft-tissue bioelasticityis essential to medical practitionersin making a successful diagnosis andskillfully carrying out surgical pro-

cedures, such as palpation or handling surgicaltools. Intraoperative palpation, for example, letsclinical medical practitioners determine the med-ical status of the body’s organs, so it’s essentialthat practitioners understand soft tissue’s physi-cal characteristics. Currently, general bioelastici-ty training is conducted in a “learn by doing”approach during daily medical work. Thisapproach, the traditional style of clinical educa-tion, offers few chances for practice in clinicalfields. Consequently, it’s difficult for students togain systematic experience in dealing with a widevariety of diseases or rare cases.

In this article, we describe how we construct-

ed haptic media to represent the specific physicalbehavior of a beating aorta for instructing intra-operative palpation in cardiovascular surgery. Weanalyze the results of a communication sequenceduring a user study that involved skilled cardio-vascular surgeons and students who trained onhaptic interaction using our virtual, deformablemodels. The study tested this method’s effective-ness in communicating both key elasticity prop-erties and the means of manipulation required tomaster palpation. It’s difficult to determine an aor-ta’s bioelasticity, especially in cases such as sclero-sis. Therefore, both the approach we created andthe quantified knowledge gained during our studywill be useful indices for the future developmentof haptically valid anatomical models.

We chose this area to investigate because cur-rently, there’s no training environment provid-ed for intraoperative palpation despite its beinga basic technique that all cardiovascular sur-geons need to master. Another reason is that theaorta palpation procedure is simple and is per-formed with a simple push of a finger. Becausecurrent force-feedback devices, such as SensAbleTechnologies’ Phantom (http://www.sensable.com), ably support the push operation, the onlyadditional requirement for creating a realisticinteraction environment is a valid physics-basedaorta model.

Background: Surgical training methodsVirtual reality (VR)-based surgical simulators1-7

have emerged as possible practical environmentsfor residents to attempt repetitive training in sur-gical procedures. Physics-based models have beendeveloped for simulating visual and haptic feed-back of virtual organ manipulation.8-15 Recentstudies report that simulator-based training helpsimprove the results of actual surgery.16,17

Some of these haptic simulators, targeted specif-ically at palpation training, are designed to supportbioelasticity knowledge acquisition.4,5 In fact, wetoo have developed palpation simulators,6,7 whichconcentrate on modeling beating behavior andorgan–organ interaction in human bodies. In gen-eral, these simulators require more accurate hapticdisplays than the training simulators used in toolmanipulation, such as suturing.2,3,13

More work, however, remains to be done toadvance palpation simulators so they can be usedeffectively in training. The main obstacle lies indeveloping valid anatomical models because ofthe difficulty in accurately specifying organs’elasticity. Output from simulations varies accord-

50 1070-986X/06/$20.00 © 2006 IEEE Published by the IEEE Computer Society

TransferringBioelasticityKnowledgethrough HapticInteraction

Megumi Nakao and Kotaro MinatoNara Institute of Science and Technology

Tomohiro KurodaKyoto University Hospital

Masaru KomoriShiga University of Medical Science

Hiroshi OyamaUniversity of Tokyo

Takashi TakahashiKyoto College of Medical Technology

Haptic User Interfaces for Multimedia Systems

This studyestablishes apracticalenvironment fortransferringknowledge onbioelasticitybetween expert andtrainee medicalpractitioners.Through hapticinteraction with adeformable virtualanatomical model,experts set themodel’s elasticityconditions bysimulating a surgicalprocedure. Traineesexperience theelasticity byattempting the samesurgicalmanipulation.

ing to physical parameters and the complexity ofthe virtual organs that are being manipulated.Techniques for modeling patient-specific elastic-ity haven’t yet been established; consequently,it’s difficult to create accurate models for diseasedorgans. In addition, the effectiveness of trainingusing haptic simulators must be further evaluat-ed so that researchers can understand andimprove simulator performance.

On the other hand, instruction by skilledmedical practitioners has traditionally played alarge role in the education of patient-specific elas-ticity. In practical situations, experts sometimescommunicate images that are difficult for resi-dents to conceptualize by referring to similar,known elastic objects. This approach is an accept-ed method of teaching, but verbal communica-tion alone, without hands-on experience, isn’tsufficient for effectively transferring knowledgeon tissue elasticity. Therefore, an optimized edu-cation process requires a combined approachusing both simulation and skilled instruction. Ifan expert’s knowledge on bioelasticity could beefficiently transferred to trainees via a computer-assisted environment, trainees would learn keyelasticity skills, including those needed for dis-ease and rare-case scenarios.

Researchers have made some headway indeveloping haptic communication betweenexperts and trainees. Haptic teaching systemsdeveloped in other fields focus mainly on toolmanipulation18,19 and are used to teach trajecto-ry and applied force, especially in the field ofhandwriting. Satoshi Saga and colleagues19 haveproposed a haptic video system, which replaysthe force history acquired from an expert’smanipulation. These approaches support theinstruction of tool manipulation rather than pro-vide a simulation environment where objects aremanipulated. In manipulating elastic objects,such as during a surgical procedure, a surgeonfeels reaction force other than that resulting fromthe tool’s position. Consequently, a surgeon’s

education in physical behavior of elastic objectsmust include both haptic interaction and physi-cal modeling of the object under manipulation.

In our study, we aimed to establish a commu-nication support environment for transferringknowledge on bioelasticity using virtual anatom-ical models. Figure 1 shows the basic concept. Wepropose an interactive system that lets skilledmedical staff instruct residents or medical stu-dents in organ bioelasticity by haptic interactionwith the models. In our deformable mediaapproach, virtual models are used to transfer elas-tic information from experts to trainees. The elas-tic information is stored quantitatively as part ofthe model’s physical parameters. In a simpleprocess, experts set up the model according totheir experience, and trainees learn how to per-ceive the elastic information by attempting sur-gical procedures on the same model.

Supporting communication onbioelasticity

Medical practitioners generally learn objects’elasticity through attempting to recognize theirphysical behavior by conducting haptic interac-tion maneuvers, such as direct hand manipulationor varying the manner of contact. Practitionersalso learn bioelasticity properties associated withspecific diseases by touching and manipulatingtumors in daily medical work.

Basic principlesTo advance current training methods and

techniques, practitioners require a platform onwhich medical procedures can be practiced. Thisplatform—haptic media—should serve to letexperts share their knowledge and trainees con-duct self-study in the virtual space. Here, we out-line the basic design, and the two primary tasks,of our proposed environment:

1. Expert sets virtual model elasticity. Experts sim-ulate medical procedures on the virtual

51

July–Septem

ber 2006

Interaction

Knowledge

Experts Trainees

Interaction

Knowledge

Elastic model

Figure 1. Basic concept.

Elastic models support

communication

between experts and

trainees as deformable

media. Experts transfer

their elasticity

knowledge onto the

models and trainees

study the elastic

behavior through

haptic interaction.

model. The feedback reaction force and defor-mation help the expert perceive the model’selasticity and compare with previous experi-ence. If the elasticity differs from the expert’sexpectation, the expert changes the model’sphysical parameters. When the model’s phys-ical behavior matches the expert’s expecta-tion, the elastic parameters are stored, whichcompletes this task.

2. Trainee learns virtual model elasticity. In this task,trainees experience the model’s elasticity byrepeatedly performing the same manipulationthat the experts carried out in the first task.Over time—which might involve a trainee’sconducting the same procedure once every fewdays—the trainee acquires a stable perceptionof different conditions, and recognizes the elas-ticity relating to a specific situation.

Virtual aorta palpation system designFor discussion purposes, we’ve considered a

total aortic arch-replacement surgery. In such asurgery, surgeons must perform intraoperativepalpation of the aorta to establish the tissues’elasticity. Surgeons do so because they need toidentify the sclerotic region to determine theoverall surgical strategy required to deal with thetumors (see Figure 2). Because the nature of thesclerotic region cannot be ascertained from visu-al information, such as texture and 3D shapealone, surgeons—using one or two fingers—pushor pinch several points of the aorta.

To provide practitioners with an interactiveenvironment enabling them to rehearse realisticpalpation procedures, we first need a cleardescription of how interaction with the virtualmodel should occur. Furthermore, it’s essentialto engineer realistic visual and haptic feedback.Specifically, the aorta model must aptly simulateboth autonomous beating and volumetric distri-

bution of elasticity equivalent to that of the scle-rotic status of real-life tissue. Additionally, toaccurately model the surgeon’s manner of palpa-tion, the haptic interface should support bothone- and two-finger manipulation. Finally, theinteractive modeling interface should allow sur-geons to modify physical parameters based ontheir empirical bioelasticity knowledge.

On the basis of these requirements and addi-tional discussions with cardiovascular surgeons,we’ve designed a virtual aorta palpation system(see Figure 3). The system visualizes a 3D aorticarch model as a polygonal object in the virtualspace. Users control two sphere-shaped manipu-lators via two Phantom haptic devices—modelPremium 1.5. The manipulators represent the 3Dpositions of a user’s thumb and index finger,respectively. The Phantom’s collision detectionalgorithm responds to any interaction betweenthe manipulators and the model, while thephysics-based simulation algorithm calculatesthe reaction force and deformation using thecontact points and manipulators’ current posi-tions. The Phantom device conveys the calculat-ed reaction force to the user and displays thedeformation result on the screen as a transfor-mation of the 3D model. The Phantom hapticdevice also lets surgeons interactively update themodel’s elasticity and boundary conditions whenthe physical behavior differs from the surgeons’expectations. We provide a graphical user inter-face that supports elastic modeling and managesgiven physical parameters.

To simulate the physical behavior of a beatingaorta while maintaining a haptic-compatiblerefresh rate, we propose new finite-element-basedcomputation methods. Although many studieshave concentrated on developing various kindsof physics-based models,8-15 the finite elementmethod (FEM) is known to be the most accuratecomputational model for simulating the biome-chanical behavior of elastic soft tissues. AlthoughFEM-based simulation provides accurate and sta-ble deformation, however, it carries a high cal-culation cost. To allow real-time interaction witha volumetric-deformable object, Morten Bro-Nielsen and Stephane Cotin have proposed acondensation technique.9 This technique reducesthe size of a stiffness matrix in the preprocessingstage and performs real-time simulation fordetailed objects. More recently, Koichi Hirotaand Toyohisa Kaneko have achieved fast compu-tation of reaction force by using an efficienttranslation of a matrix calculation.14

52

IEEE

Mul

tiM

edia

Push pointby a forefinger

Pincing point

Figure 2. Intraoperative

palpation in total

aortic arch replacement

surgery. Surgeons push

or pinch the aorta’s

surface wth one or two

fingers to identify the

sclerotic region for

diagnosis and overall

strategy planning.

Although most previous studies didn’t try tosimulate dynamic behavior like pulsation or beat-ing status, our previous study detailed the 3Danatomical shapes and approximate time seriespressure required to realistically represent thehaptic feedback of heartbeats.6 In the study,we’ve used these features to create haptic-deformable media to represent a beating aorta’sdynamic behavior during cardiovascular surgery.Our proposed FEM-based computation methodsthen calculate the reaction force and biome-chanical deformation that reflects the internalpressure induced by the user’s manipulation.

Physics-based modeling of a beatingaorta

Physics-based simulation requires a 3D shapewith elastic information of the target organ.

3D shape and elasticity modelingTo construct our virtual aorta model, we

acquired images of patients’ aortas from com-puterized tomography or magnetic resonanceimaging techniques. We extracted a 3D region ofthe aorta from voxels and divided them intofinite tetrahedra. Each tetrahedron represented apart of the 3D region of the aorta and also its par-ticular physical parameters: Young’s modulusand Poisson’s ratio, which are measures of elas-ticity. We used the volumetric grid topology andphysical parameters to calculate a stiffnessmatrix, enabling finite-element-based simula-tion. We categorized all vertices, based on threedifferent boundary conditions, into fixed ver-tices, internal wall vertices, and other free ver-tices. Fixed vertices represented tissue thatconnects or contacts other organs. Internal wallvertices represented the aorta’s beating, holdingdynamic force by a time series of blood pressuredata. We represented the condition of arterialsclerosis by setting a high Young’s modulus tothe desired region.

We created a virtual 3D shape of a normal aor-tic arch from the Visible Human Male data set.20

Using Mercury Computer Systems’ Amira 3.1modeling software (http://www.amiravis.com),we reconstructed the aorta surface and generat-ed tetrahedral grids. Figure 4 illustrates the con-structed aortic arch model. The total number ofthe vertices was 1,651. The edge vertices of the3D model, which simulated connection to theheart and other vessels, were defined as fixedbecause these areas didn’t move in actual palpa-tion. Each vertex was represented as a small

sphere colored according to its boundary condi-tion, shown in Figure 4.

The system we developed also provided an elas-tic modeling interface to support flexible parame-ter setting by skilled surgeons. The view of thevirtual space contained a 3D bounding box (seeFigure 4) whose shape and 3D position were con-trolled interactively using a slider bar and edit boxin the graphical user interface. We also preparedseveral shape templates such as spheres and cubes.This interface let surgeons modify the model elas-ticity and boundary conditions as needed.Following this interactive elasticity modeling, thesystem automatically updated the stiffness matrixby running matrix generation algorithms.

53

July–Septem

ber 2006

Phantom

Internalpressure

Tetrahedralmesh

Deformation andforce calculation

Collision detection

GUI

Object manager

Position ShapeForce

Matrixgenerator

InverseK matrix

Physicalparameter

Matrix manager Pressuremanager

Figure 3. Overall framework for the simulation system developed for

instructing palpation of the aorta. Surgeons and residents used this system to

rehearse realistic palpation procedures on the virtual aorta model. Skilled

surgeons conveyed their bioelasticity knowledge by modifying the model

elasticity.

Figure 4. The aortic

arch model. The 3D

shape was acquired

from the Visible

Human Male data set.

A high Young’s

modulus was given to

tetrahedral elements

in the sclerotic region

using a 3D bounding

box. The colors of

vertices represented

boundary conditions.

Red vertices were

“fixed,” green vertices

were “internal wall,”

and blue vertices were

“free” for deformation.

Aortic arch model

Sclerotic region

Manipulator

54

IEEE

Mul

tiM

edia

Haptic interactionThe collision-detection algorithm, in con-

junction with physics-based simulation, enabledvirtual palpation to be performed on the aortamodel. The algorithm handled the intersectionbetween the model’s fingertip manipulators andthe tetrahedral grids, for which we employedproxy-based haptic rendering methods.21,22 Thealgorithm was based on point-polygon collisiondetection, which, despite being a simplificationof a real aorta, was sufficient to handle the basicinteraction of a surgeon’s fingertips touching theaortic wall in real surgery. The main challenge ofthis study was to simulate and describe interac-tion with a deformable object that’s subject toautonomous beating.

Figure 5 outlines the basis of our haptic inter-action model for aorta palpation. A 2D cylindercross-section approximately illustrates the aorticwall. When blood pressure was applied to thewall (depicted in Figure 5a), the FEM-based sim-ulation responded by expanding the 3D shape.The higher the pressure applied, the larger theexpansion, as Figure 5b shows. The system pro-vided simulation results at discrete time steps,effectively modeling the aorta wall’s autonomousbeating. In addition, when an external force ini-tiated a small displacement in the aorta model,the reaction force was calculated and conveyedto the user’s fingertips via the Phantom devices,indicated in Figure 5c.

Cardiovascular surgeons reported that, duringa palpation procedure, they avoid pinching theaortic wall too hard to avoid damaging soft tis-sues and to minimize the aorta’s deformation.Therefore, our proposed model assumed thatinternal blood pressure wasn’t affected by surgi-cal palpation, evidence that linear finite elementmodels were a good approximation for real-lifeaorta palpation.

FEM-based computationWe based our calculation method—to simu-

late the reaction force and deformation of a beat-ing aorta—on linear elastic theory. Assuming thatthe internal force is in equilibrium at each dis-crete time step, the relationship between externalforce and displacement on an elastic object isdefined by f = Ku where, on all vertices, u is dis-placement and f is external force, including bloodpressure. K denotes the stiffness matrix of theobject constructed by grid topology and physicalparameters (Young’s modulus and Poisson ratio).The K matrix can be efficiently reduced by con-densation9 and elimination of the fixed verticesin the preprocessing stage. This expression is sim-plified to u = Lf using the condensed inverse stiff-ness matrix L, which defines the physicalrelationship between the external force f and thedisplacement u on the surface vertices.

To represent autonomous beating and to sim-ulate the physical behavior of the user’s interac-tion, we divided the surface vertices into threegroups: contacted vertices, internal wall vertices,and other free vertices. Contacted vertices aredirectly displaced by the user’s manipulation.Internal wall vertices are affected by the timeseries of blood pressure. Equation 1 expands u =Lf by using the initial letters of categorized ver-tices to represent the coefficients of the matrices.

(1)

where fi denotes blood pressure that’s applied tothe internal wall vertices and uc is displacementof the contacted vertex manipulated through thePhantom haptic device. Considering that fo isconstant zero, the relationship between uc, fi, andfc is described as uc = Lcifi + Lccfc. Consequently, fc

is given as fc = Lcc-1(uc - Lcifi).

Accordingly, fc is external force on the contact-ed vertex, and −fc is reaction force conveyed to theuser. Note that we can obtain Lcc and Lci by pre-computation because both are defined by Young’smodulus and Poisson’s ratio. We calculate Lcc andLci for all free vertices and perform a refresh rate ofmore than 1,000 Hz to maintain stable force feed-back. The dynamic transition of fc at discrete timesteps lets us present haptic feedback of the beatingaorta under the effects of time series blood pres-sure. Applying fc to Equation 1 provides the dis-placement uo on other free vertices.

uu

u

L L L

L L L

L L Li

o

c

ii io ic

oi oo oc

c

⎛

⎝

⎜⎜⎜

⎞

⎠

⎟⎟⎟

=

ii co cc

i

o

c

f

f

f

⎛

⎝

⎜⎜⎜

⎞

⎠

⎟⎟⎟

⎛

⎝

⎜⎜⎜

⎞

⎠

⎟⎟⎟

Pressure

Time

Blood pressure

External force byuser interactionAortic wall

(a) (b) (c)

Figure 5. FEM-based

computation model for

a beating aorta: (a) 3D

shape; (b) aorta

expansion after higher

pressure exerted; and

(c) the result of

external force applied.

Evaluation and user studyIn this section, we describe the virtual palpa-

tion system’s operation, example results, andoutputs. We then present the design of studiesthat evaluate the effectiveness of our system insupporting communication on bioelasticity.

System verificationWe processed the computation algorithms on

a standard PC with a Pentium 4 2.4-GHz CPUand 1,024 Mbytes of memory. We optimized thematrix calculation on the CPU with Intel’s MathKernel Library.

Our first step was to confirm the visual andhaptic quality and performance of the developedsystem with cardiovascular surgeons. In thisexperiment, we set Young’s modulus of the nor-mal model to 1.0 megapascal (MPa), that of thesclerotic model to 3.0 MPa, and the Poisson’sratio to 0.48, based on the measured data for anormal aortic wall that we’d obtained previous-ly.23 Later, we explain how we tested the model’sphysical characteristics, compared to the empiri-cal knowledge of skilled cardiovascular surgeons.

Figure 6 shows how the user interacted withthe virtual system and gives an example of thedeformation that occurred in the virtual aortafollowing pinching with two fingers. The usertouched two manipulators in real space with thetips of the Phantom devices that were worn onthe fingertips (bottom-left image). These manip-ulators translated the position of the fingertips tothe model aorta in the virtual space. In Figure 6,we highlight the virtual position of the fingertips(top-left image) that corresponded to the finger-tips’ real position (bottom-left image).

Next, we tested the accuracy of the model insimulating both normal and sclerotic aorta con-ditions. The proposed FEM-based model simu-lated reaction force and deformation for an aortasubjected to time series pressure. Figure 7 showsthe relationship between a dynamic transition ofthe reaction force and the applied time seriesblood pressure when the aorta wall was subject-ed to a specific displacement by the fingers. Weprogrammed two regions of the virtual aorta,each with a different state (normal and sclerotic).The two different regions showed differentabsolute values of reaction force and beating sta-tus for the same given displacement. The sclerot-ic part did not reproduce the pulse with as great amagnitude as the normal part. These phenome-na were similar to that observed by Haruo Okinoand colleagues.23 Thus, our model simulated real-

istic haptic feedback reflecting the dynamic phys-ical behavior of both a normal and a scleroticaorta wall.

The developed model had 1,651 vertices, whichprovided sufficient visual quality to represent the3D shape of an aortic arch. In this case, the calcu-lation time was 0.08 ms for force feedback and 0.5ms for deformation. These times confirmed thatthe proposed calculation methods achieved a suf-ficient refresh rate to handle deformable virtualmedia with autonomous beating.

User testsAs we explained, the system’s role was to sup-

port the communication of knowledge on aortabioelasticity. We categorized this communicationin three separate procedures and planned threeuser studies to test how well the system support-ed this communication, basing each experimenton each of the following hypotheses:

55

Figure 6. Real-time

deformation in

palpating the virtual

aorta. Two

manipulators

controlled by two

SensAble Technologies’

Phantom haptic devices

enable users to pinch

the model interactively.

Manipulators(virtual fingertips)

Virtual aorta

Forc

e (N

)

Pres

sure

(m

mH

g)

Normal region

Sclerosis region 300

250

200

150

100

50

0

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

Aorta pressure

849 msec

Figure 7. Reaction force

in pushing

normal/sclerotic region

on a virtual aorta

sclerotic model. This

simulation result

demonstrated that the

sclerotic part did not

reproduce the pulse

with as great a

magnitude as normal

tissue. July–Septem

ber 2006

❚ Elasticity modeling. If experts knew the aorta’scorrect tissue elasticity during palpation andthe system provided a valid elasticity-model-ing environment, we could set up the virtualaorta with identical given elasticity.

❚ Elasticity recognition. If haptic interaction withthe virtual media effectively supported com-munication on bioelasticity, trainees would beable to recognize the given elasticity more eas-ily than with other training methods.

❚ Learning elasticity. If the system supportedinstruction of a specific bioelasticity, traineeswould be able to learn the given elasticitythrough continuous repetition of the palpa-tion procedure.

We focused on two elasticity conditions that wereconsidered key in intraoperative aorta palpation:normal tissue elasticity and the threshold elastic-ity that’s regarded as sclerotic. Cardiovascular sur-geons have to learn both elasticity conditions toidentify the sclerotic region and determine theoverall surgical strategy. In the next three sec-tions, we describe the results of the user study.

Elasticity modelingFirst, we examined the capability of the sys-

tem to carry out elasticity modeling of the virtu-al aorta, working with eight skilled surgeons fromthe Department of Cardiovascular Surgery atKyoto University Hospital. We prepared 20 aortamodels with the same 3D shape (shown in Figure4) but different levels of stiffness. We set the uni-form elasticity of each model by changing the

input parameter of Young’s modulus, which var-ied from 0.2 MPa to 4.0 MPa. The Poisson’s ratiowas set as 0.48. In advance of the experiment, weallowed the surgeons a few minutes of practicetime during which they attempted the palpationprocedure at leisure to accustom themselves tothe virtual environment.

For the study itself, the surgeons conducted anormal palpation procedure on the 3D virtualaorta. If the physical behavior and reaction forcediffered from their expectation of how a normalaorta should have behaved, we changed themodel elasticity accordingly. They repeatedly pal-pated the models and selected the one that mostrealistically simulated the elasticity of a normalaorta. The same procedure was used to determinethe threshold elasticity that should have indicat-ed sclerosis in real surgery. We randomized theorder in which the surgeons tested the differentmodels. The same palpation point on the modelwas used for all surgeons.

Figure 8 shows the statistical results for themodels that were selected for normal and sclerot-ic elasticity conditions. The left-hand graph showsthere were only two different models, 1.0 MPa and1.2 MPa, which the surgeons deemed to corre-spond to their experience of a normal aorta.Specifically, five surgeons selected the 1.0-MPamodel; three chose the 1.2-MPa model. Next, theright-hand graph in Figure 8 shows the results ofselection for the sclerotic aorta. The graph suggeststhat anything over 3.0 MPa corresponded to scle-rosis, according to the experience of the surgeons.These results lead us to the following insights:

❚ A virtual aorta with a Young’s modulusbetween 1.0 MPa and 1.2 MPa effectively dis-plays the physical behavior of a normal aorta.

❚ A virtual aorta having a Young’s modulus ofover 3.0 MPa simulates sclerotic status.

❚ All skilled surgeons recognize the absoluteelasticity of normal and sclerotic conditionsby touch. This means that elasticity is themost important information to communicatewhen trainees try to master the palpation ofan aorta.

The surgeons who evaluated our systemreported that manipulating the virtual modelusing the two Phantom devices was comparableto the experience of real surgery. Also, they stat-ed that they experienced a realistic reaction force

56

IEEE

Mul

tiM

edia

Normal Sclerosis0.0

0.4

0.8

1.2

1.6

2.0

2.4

2.8

3.2

3.6

4.0

Elas

ticity

(M

Pa)

Figure 8. Interactive

modeling results of

normal elasticity and

the threshold elasticity

that should be regarded

as sclerotic. The box

plot indicates 25

percent (minimum)

and 75 percent

(maximum) of the

data. This

demonstrates that

skilled surgeons are

able to recognize key

aorta palpation

bioelasticity and also

show their knowledge

on bioelasticity in aorta

palpation.

following adjustment of the model’s physicalparameters. In addition, a sufficiently small dis-agreement arose among the surgeons in choosingthe models that best represented normal and scle-rotic conditions of the aorta. These results indi-cate that our system enabled virtual palpationthat effectively mirrored the real-life procedure.

Elasticity recognitionAs shown in the first experiment, the ability

to recognize the elasticity of normal and sclerot-ic aortas is important. In our second experiment,we aimed to confirm that haptic interaction witha deformable media was useful for bioelasticitytraining. We initiated a study with 18 medicalstudents who hadn’t experienced palpation of areal aorta and examined their ability to recognizedifferent elasticity conditions.

Ten of the virtual aorta models used in theprevious experiment were again prepared andthe students performed virtual palpation in thesame manner. Again the elasticity ranged from0.2 MPa to 2.0 MPa. Some time was given for thestudents to become accustomed to manipulationand the manner of touching the model in virtu-al space. Then, each student was asked to selectthe model, which they thought best representeda real-life normal aorta. We prepared the studentsfor making their selection under the followingconditions.

Condition 1: No information. The medicalstudents had only experienced the elasticity of acadaver’s aorta during their medical training andhad not received any specific instruction on aortaelasticity prior to starting the experiment.Because the elasticity of a cadaver is totally dif-ferent from that of a living body, they had noexperience of an in vivo aorta and, in selectingfrom among our models, had to imagine what areal aorta would feel like.

Condition 2: Verbal information. The stu-dents underwent instruction from cardiovascu-lar surgeons who described the physicalcharacteristics of a normal aorta using both briefexplanation and rubber hoses as a physical rep-resentation of aorta elasticity. This technique isa conventional teaching method.

Condition 3: Haptic instruction. The med-ical students were given 1 minute to becomefamiliar with the elasticity of a normal aorta bycarrying out virtual palpation on our developed

system. The 1.0 MPa value was specified as theYoung’s modulus of this “normal” virtual aorta.

Figure 9 illustrates the statistical results for theelasticity of the models that the students select-ed. The statistical analysis (F-test) result showssignificant difference between the conventionalmeans of instruction (condition 1 and condition2) and simulator-based instruction (condition 3).The distribution of the graph at condition 2 stillshows a large spread. This result shows that ver-bal communication alone isn’t an effectivemeans for students to learn the elasticity of a spe-cific aorta condition, because students have anexisting expectation of the elasticity that can’t bealtered simply by hearing what it should feel like.The graph of condition 3 is close to a normal dis-tribution, and more than 80 percent of the stu-dents selected models with elasticity between 0.8MPa and 1.2 MPa. This result demonstrates thathaptic interaction using our system is an effec-tive method for enabling students to recognizethe normal stiffness of the virtual aorta model.Furthermore, the Young’s modulus of 1.0 MPahad been configured by expert surgeons in theprevious experiment, meaning that the studentseffectively experienced the elastic characteristicsof an in vivo normal aorta through virtual pal-pation. We only conducted the second experi-ment employing the normal aorta because verbalrepresentation isn’t generally used to describe asclerotic aorta in clinical work.

Learning elasticityIn this third, final experiment, we tested the

effectiveness of learning elasticity through virtu-

57

July–Septem

ber 2006

Figure 9. Statistical

results of elasticity

recognition of a 1.0-

MPa normal aorta

model in three cases:

conventional

instruction (condition 1

or 2) and simulator-

based instruction

(condition 3). (The

asterisks indicate

significant statistical

difference between the

two data values.)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

Condition 1

**

Condition 2 Condition 3

Elas

ticity

(M

Pa)

al palpation. The second experiment had demon-strated that haptic instruction is an efficientmeans of enabling elasticity recognition, but tomaster palpation, students must memorize thecritical elasticity they encounter during the pro-cedure. Therefore, our study needed to furtherexamine whether students were able to retaintheir knowledge of elasticity. In this last experi-ment, we aimed to provide students with a learn-ing curve, in the form of a month-long course oftraining. We carried out the simulator-basedlearning based on the following conditions:

❚ The participants were 10 students who hadn’tpreviously experienced a real aorta throughtouch.

❚ For each participant, seven trials were con-ducted, occurring approximately twice a week.The total experimental period was 23 days.

❚ Each trial consisted of two steps: testing andlearning. Participants completed both steps atevery trial.

❚ Test stage: Participants were asked to select theaorta models that they believed to be normalor sclerotic, from 20 aorta models withYoung’s modulus between 0.2 MPa and 4.0MPa, based on their current bioelasticityknowledge.

❚ Learning stage: The correct normal (1.0 MPa)and sclerotic (3.0 MPa) aorta models wererevealed. Examinees were given a few minutesto try to learn the elasticity of the modelsthrough virtual palpation.

❚ The palpation point on the aorta model wasfixed throughout the experiment.

Figure 10 shows the selection data over thecourse of the third experiment in which the stu-dent participants attempted to choose the nor-mal aorta. The spread of the participants’selections clearly narrowed over time, and themedian eventually approached 1.0 MPa. Our sta-tistical analysis (T-test) reveals a significant dif-ference in the spread between the first and thirdtrials, and the second and third trials. Thesegraphs demonstrate that several separate trainingopportunities effectively contribute to learningthe stiffness characteristics of the normal aorta.

Figure 11, which shows the selection data forthe sclerotic condition, suggests that hapticinstruction using the developed system is effec-tive. However, there is a wider spread in theselections, compared to the learning curve inFigure 10. This tendency is consistent with whatis known about human perception, in thathuman haptic sensitivity to relative physicalbehavior is proportional to the logarithm ofabsolute stiffness.

Final remarksIn evaluating the results of our experiments, we

discuss how the efficacy of realistic VR-based pal-

58

0.0

0.4

0.8

1.2

1.6

2.0

2.4

2.8

3.2

3.6

4.0

2 6 9 13 17 20 23

Elas

ticity

(M

Pa)

Days

Figure 10. Virtual palpation learning curve for the normal model. The target

was 1.0 MPa. The spread narrowed and the median selection approached

1.0 MPa over time.

Elas

ticity

(M

Pa)

0.0

0.4

0.8

1.2

1.6

2.0

2.4

2.8

3.2

3.6

4.0

2 6 9 13 17 20 23Days

Figure 11. Virtual

palpation learning

curve for the sclerotic

model. The target was

3.0 MPa. The spread

narrowed and the

median approached the

target over time.

Compared to Figure 10,

each trial features a

greater spread of

selections.

pation might be further improved. First, toachieve a real-time refresh rate, we simplified thephysical behavior of an in vivo aorta by ignoringsome essential functions. We didn’t consider theeffects of local pressure or blood flow, and simu-lated continuous autonomous beating withdiscrete time-series blood-pressure data. For non-complex 3D aorta models, the blood pressureapplied to the internal walls was assumed to beconstant and independent of local position. Theadvantage of this simplification was that it aidedfast calculation for valid haptic feedback and letus reproduce key soft-tissue behavior (for exam-ple, pulsation and elasticity) in virtual palpation.Our model achieved a refresh rate of more than1,000 Hz in the reaction force calculation owingto our simplification of boundary conditions andinverse matrix calculation.

Another simplification was to reproduce thedynamic force of autonomous beating using a sta-tic FEM model. The variance in elasticity betweenthe model and real life was small, and the sur-geons who evaluated our system agreed that itdemonstrated realistic force feedback and graph-ical deformation for the palpation procedure.

Although VR-based surgical simulators pro-vide an effective training environment in clini-cal training, the representation and validation ofspecific bioelasticity behavior remains a problem.Both our proposed bioelasticity communicationsupport environment and the quantified knowl-edge gained during our study will be usefulindices for the future development of hapticanatomical models.

The approach we followed is potentiallyapplicable to other virtual organ models thatrequire a multiphysics simulation. We suggestthat more detailed modeling, especially focusedon a number of specific diseases and morbidstages, will be carried out as future work. Thiswork will contribute to the compilation of prac-tical instruction courseware. MM

AcknowledgmentThis research was supported by a Grant-in-Aid

for Scientific Research (S) (16100001) and YoungScientists (A) (16680024) from the Ministry ofEducation, Culture, Sports, Science, andTechnology, Japan.

References1. P.J. Gorman, A.H. Meier, and M. Krummel, “Com-

puter-Assisted Training and Learning in Surgery,”

Computer Aided Surgery, vol. 5, 2000, pp. 120-127.

2. U. Kuhnapfel, H.K. Cakmak, and H. Mass,

“Endoscopic Surgery Training Using Virtual Reality

and Deformable Tissue Simulation,” Computer

Graphics, vol. 24, no. 5, 2000, pp. 671-682.

3. C. Wagner, M.A. Schill, and R. Manner, “Collision

Detection and Tissue Modeling in a VR Simulator

for Eye Surgery,” Proc. Workshop on Virtual

Environments, 2002, ACM Int’l Conf. Proc. Series,

ACM Press, pp. 27-36.

4. S. Baillie et al., “Validation of a Bovine Rectal

Palpation Simulator for Training Veterinary

Students,” Proc. Medicine Meets Virtual Reality Conf.

(MMVR), 2005, IOS Press, pp. 33-36.

5. T. Tokuyasu et al., “Mechanical Modeling of a

Beating Heart for Cardiac Palpation Training

System,” Int’l J. Robotics Soc. Japan, vol. 17, no. 6,

2003, pp. 463-479.

6. M. Nakao et al., “Haptic Reproduction and Real

Time Visualization of a Beating Heart for

Cardiovascular Surgery Simulation,” Int’l J. Medical

Informatics, vol. 68, issue 1-3, 2002, pp. 153-161.

7. Y. Kuroda et al., “Interaction Model between Elastic

Objects for Haptic Feedback Considering Collisions

of Soft Tissue,” Computer Methods and Programs in

Biomedicine, vol. 80, no. 3, 2005, pp. 216-224.

8. D. Braff and A. Witkin, “Physically Based Modeling:

Principles and Practice,” ACM Siggraph Course

Notes, ACM Press, 1997.

9. M. Bro-Nielsen and S. Cotin, “Real-Time Volumetric

Deformable Models for Surgery Simulation Using

Finite Elements and Condensation,” Eurographics

Computer Graphics Forum, vol. 15, no. 3, 1996, pp.

57-66.

10. S. Cotin, H. Delingette, and N. Ayache, “A Hybrid

Elastic Model Allowing Real-Time Cutting,

Deformations, and Force Feedback for Surgery

Training and Simulation,” The Visual Computer, vol.

16, no. 7, 2000, pp. 437-452.

11. E. Gladilin, Biomechanical Modeling of Soft Tissue

and Facial Expressions for Craniofacial Surgery

Planning, doctoral dissertation, Mathematics Dept.,

Free University Berlin, 2002.

12. R. Balaniuk and K. Salisbury, “Dynamic Simulation

of Deformable Objects Using the Long Elements

Method,” Proc. 10th Symp. Haptic Interfaces for

Virtual Environments and Teleoperator Systems, IEEE

CS Press, 2002, pp. 58-65.

13. J. Berkley et al., “Banded Matrix Approach to Finite

Element Modeling for Soft Tissue Simulation,”

Virtual Reality Research Development and Application,

vol. 4, 1999, pp. 203-212.

14. K. Hirota and T. Kaneko, “Haptic Representation of

Elastic Object,” Presence, vol. 10, no. 5, 2001, pp.

525-536.

59

July–Septem

ber 2006

15. S. Capell et al., “A Multiresolution Framework for

Dynamic Deformations,” Proc. ACM Siggraph Symp.

Computer Animation, 2002, pp. 41-47.

16. N.E. Seymour, “Virtual Reality Training Improves

Operating Room Performance: Results of a

Randomized Doubleblinded Study,” Annals of

Surgery, vol. 236, no. 4, 2002, pp. 458-463.

17. T.P. Grantcharov et al., “Randomized Clinical Trial

of Virtual Reality Simulation for Laparoscopic Skills

Training,” British J. Surgery, vol. 91, 2004, pp. 146-

150.

18. R. Kikuuwe and T. Yoshikawa, “Haptic Display

Device with Fingertip Presser for Motion/Force

Teaching to Human,” Proc. IEEE Int’l Conf. Robotics

and Automation, 2001, pp. 868-873.

19. S. Saga, N. Kawakami, and S. Tachi, “Haptic

Teaching Using Opposite Force,” poster presenta-

tion and demonstration, Proc. World Haptics Conf.,

IEEE CS Press, 2005.

20. M.J. Ackermann, “The Visible Human Project: A

Resource for Anatomical Visualization,” Proc. Medical

Informatics (Medinfo), 1998, pp. 1030-1032.

21. C.B. Zilles and J.K. Salisbury, “A Constraint-Based

God-Object Method for Haptic Display,” Proc.

IEEE/RSJ Int’l Conf. Intelligent Robots and Systems

(IROS), vol. 3, IEEE CS Press, 1995, pp. 141-151.

22. D. Ruspini, K. Kolarov, and O. Khatib, “The Haptic

Display of Complex Graphical Environments,” Proc.

ACM Siggraph, ACM Press, 1997, pp. 345-352.

23. H. Okino, H. Matsuo, and M. Sugawara, Shinzou

Kekkankei no Rikigaku to Kiso Keisoku [Cardiovascular

System Mechanics and Basic Measurements],

Kodansha Ltd., 1980, pp. 188-189 (in Japanese).

Megumi Nakao is an assistant

professor in biomedical imaging

and informatics at the Graduate

School of Information Science,

Nara Institute of Science and

Technology. His research inter-

ests include medical virtual reali-

ty, computer graphics and haptics, physics-based

modeling and biomedical engineering. Nakao has a

PhD in informatics from Kyoto University.

Tomohiro Kuroda is the vice-

director in the Department of

Medical Informatics at Kyoto

University Hospital. He also holds

a docent position at Osaka Sangyo

University and a visiting-professor

position at the University of

Oulu. His research interests include the human inter-

face, virtual/augmented reality, wearable computing,

and medical and assistive informatics. Kuroda has a PhD

in information science from the Nara Institute of

Science and Technology.

Masaru Komori is a professor of

computational biomedicine at the

Shiga University of Medical

Science. His research interests are

virtual reality applications in the

medical field, medical imaging,

and medical informatics. Komori

has a PhD in electrical engineering from Kyoto University.

Hiroshi Oyama is a professor of

clinical bioinformatics engineer-

ing, and an adjunct professor of

medical informatics and eco-

nomics at the University of

Tokyo. His research interests

include biomedical computing

and modeling and their application to virtual reality,

telemedicine, surgical simulation, palliative medical

systems, and ubiquitous medicine. Oyama has a PhD

in medicine from the University of Tsukuba.

Kotaro Minato is a professor of

biomedical imaging and informat-

ics at the Graduate School of

Information Science, Nara Institute

of Science and Technology. His

research interests include medical

informatics, medical imaging, and

biomedical engineering. Minato has a PhD in electrical

engineering from Kyoto University.

Takashi Takahashi is the presi-

dent of the Kyoto College of

Medical Technology and an

emeritus professor at Kyoto

University. His research interests

include medical informatics,

medical virtual reality, and med-

ical engineering. Takahashi has a PhD in electronic

engineering from Osaka University.

Readers may contact Megumi Nakao at the Nara

Institute of Science and Technology, Graduate School of

Information Science, 8916-5, Takayama, Ikoma, Nara,

630-0192, Japan; meg@is.naist.jp.

For further information on this or any other computing

topic, please visit our Digital Library at http://computer.

org/publications/dlib.

60

IEEE

Mul

tiM

edia