View
1.635
Download
5
Embed Size (px)
DESCRIPTION
A summary of abstracts and validation papers published on ProMIS - all demonstrating validity of ProMIS metrics as indicators of surgical proficiency. Also on www.haptica.com
Citation preview
Validated! PPPrrrooovvveeennn wwwiiittthhh FFFLLLSSS
BBBeeetttttteeerrr ttthhhaaannn VVVRRR
PPPrrreeefffeeerrrrrreeeddd bbbyyy uuussseeerrrsss
Augmented Reality combines VR
with a physical model allowing
virtual bleeding and real haptics.
ProMIS has pure VR Modules, eg
for Instrument Handling. Users
still use real instruments.
Vision technology enables
tracking of errors and automatic
calculation of dissected tissue.
ProMIS assesses performance on
real models where real haptics
are important, eg in Suturing and
Knot-tying
ProMIS and FLS
• ProMIS metrics are “excellent predictors of scores
in the standard FLS simulator” and “predict
readiness for FLS Certification”1 2 3
• ProMIS metrics are valid on peg transfer, pre-tied
loop placement and knot-tying tasks4
• “Initial trials of the metrics on the FLS Precision-
Cutting Task show that ProMIS is as accurate as
the current method” 5
• FLS tasks are transferable to the ProMIS simulator
with traditional FLS scoring and intrinsic ProMIS
metrics being good measurement tools. A ProMIS
total path length <4000 mm or total smoothness
<6000 reliably predicts a passing FLS score. 25
ProMIS vs pure VR
• ProMIS out-performed the virtual reality
simulators24
• Scores for ProMIS were significantly higher than
for SurgicalSIM for overall realism, thread
behavior, reflection of clinical ability, and overall
educational value.6
• In comparison with LapSim, ProMIS was regarded
by all participants as a better simulator for
laparoscopic skills training on all tested features7
• “Only [ProMIS] was able to distinguish between
advanced trainees and beginners (and) was
graded more realistic (70% vs 33%) and more
useful (83% vs 62%)” than Xitact8
• The ability of performance metrics of [ProMIS] to
discern predicted performance differences
between experts and non-experts was better
than for SimSurgery’s robotic surgery simulator and
SurgicalSIM 9
ProMIS: the preferred Simulator
• ProMIS can be used effectively with the DaVinci
robot to obtain performance data with robotic
instrumentation10
• Residents believe that ProMIS is easy to use and
improved their operative skills11
1 SAGES 2006 S064 Ritter et al,
2 SAGES 2006 P237 McCluney MD, et al,
3 SAGES 2007 P279 McCluney et al,
4 SAGES 2006 S065 Vuong et al,
5 SAGES 2007 ETP057 Young et al,
6 SAGES 2006 P224 Fellinger, et al,
7 World J Surg. 2007 Apr;31(4):764-72. Botden et al,
8 SAGES 2007 P270 Hahnloser et al,
9 SAGES 2007 S077 Lin et al,
10 SAGES 2006 Narula et al,
11 SAGES 2006 P219 Chang et al,
24 Heinrichs, et al 2007
25 Hungness, et al 2008
ProMIS surgical simulator
ProMIS Modules range from Basic Laparoscopic Skills to
MIS procedures like LapColectomy. For more information
on ProMIS, please contact us at:
Email: [email protected]
U.S. tel: +1 617 342 7270
RoW tel: +353 (0)1 676 7310
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
1. SAGES 2006 Scientific Session S064
CONCURRENT VALIDITY OF AUGMENTED REALITY METRICS
APPLIED TO THE FUNDAMENTALS OF LAPAROSCOPIC
SURGERY (FLS)
E. Matt Ritter MD, Tamara W Kindelan MD, Curtis Michael,
Elisabeth A Pimentel BA, Mark W Bowyer MD, 1NCA Medical
Simulation Center, Department of Surgery, Uniformed Services
University, 2Division of General Surgery, National Naval Medical
Center, Bethesda Maryland
Objective
Current skills assessment in the Fundamentals of Laparoscopic
Surgery (FLS) program is labor intensive requiring one proctor for
every 1-2 subjects. The ProMIS Augmented Reality (AR) simulator
(Haptica, Dublin IRE) allows for objective assessment of physical
tasks through instrument tracking technology. We hypothesized
that the ProMIS metrics could differentiate between ability
groups as well as standard FLS scoring with fewer personnel
requirements.
Methods
We recruited 60 volunteer subjects. Subjects were stratified
based on their laparoscopic surgical experience. Those who had
performed more than 100 laparoscopic procedures were
considered experienced (n=8). Those with less than 10
laparoscopic procedure were considered novices (n=44). The rest
were intermediates (n=8). All subjects performed up to 5 trials
of the peg transfer task from FLS in the ProMIS simulator. FLS
score, instrument path length, and instrument smoothness
assessment were generated for each trial.
Results
For each of the 5 trials, experienced surgeons outperformed
intermediates who in turn out performed novices. Statistically
significant differences were seen between the groups across all
trials for FLS score (p < 0.001), ProMIS path length (p <0.001) and
ProMIS smoothness (p < 0.001). When the FLS score was
compared to the path length and smoothness metrics, a strong
relationship between the scores was apparent for novices (r =
0.78, r = 0.94 , p < 0.001) respectively), intermediates (r = 0.5, p
= 0.2 , r = 0.98, p < 0.001), and experienced surgeons (r = 0.86, p
= 0.006, r = 0.99, p < 0.001)
Conclusions
The construct that the standard scoring of the FLS peg transfer
task can discriminate between experienced, intermediate, and
novice surgeons is validated. The same construct is valid when
the task is assessed using the metrics of the ProMIS. The high
correlation between these scores establishes the concurrent
validity of the ProMIS metrics. The use of AR for objective
assessment of FLS tasks could reduce the personnel requirements
of assessing these skills while maintaining the objectivity.
2. SAGES 2006 Education/Outcomes–P237
VALIDATION OF THE PROMIS HYBRID SIMULATOR USING
A STANDARD SET OF LAPAROSCOPIC TASKS
A L McCluney MD, L S Feldman MD, G M Fried, Steinberg-
Bernstein Centre for Minimally Invasive Surgery, McGill
University Health Centre, Montreal, QC, Canada
Introduction
SAGES Fundamentals of Laparoscopic Surgery (FLS) tasks are
validated measures of technical skills, however FLS scoring
requires a trained proctor. The ProMIS simulator (Haptica;
Dublin, IR) is a ‘hybrid’ system with physical and virtual
reality tasks. It has the flexibility to incorporate any
physical task and score it with ProMIS metrics. Metrics are
automated and report motion analysis data as instrument
path length (PL) and instrument smoothness (IS). The
purpose of this study was to test for construct and
concurrent validity using FLS tasks in the ProMIS simulator.
Methods
5 laparoscopic novices and 5 experts performed FLS tasks in
both the standard FLS simulator box and the ProMIS
simulator. Assessments were made based on FLS metrics, as
well as PL and IS. Student’s t-test was used to compare the
mean (SD) of total scores for novices and experts. Pearson’s
correlations were calculated for standard FLS scores in
relation to ProMIS FLS scores, total PL, and total IS.
Significance was defined as p < 0.01 (*).
Results
Standard FLS scores correlated strongly with ProMIS FLS
scores (r=0.90), total PL (r=-0.83), and total IS (r=-0.78)
(p< 0.01).
Conclusions
FLS tasks performed in ProMIS, when scored by either
traditional FLS metrics or by intrinsic ProMIS metrics,
discriminate effectively between novices and experts. Based
on the observed correlations, ProMIS FLS scores, total PL,
and total IS are excellent predictors of scores in the
standard FLS simulator.
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
3. SAGES 2007 Education/Outcomes – P279
AUTOMATED PROMIS SIMULATOR METRICS PREDICT
READINESS FOR FLS CERTIFICATION
Anthony L McCluney MD, J Cao, G N Polyhronopoulos MD, D D
Stanbridge, L S Feldman MD, G M Fried MD, Steinberg-Bernstein
Centre for Minimally Invasive Surgery, McGill University,
Montreal, QC, Canada
Introduction
SAGES Fundamentals of Laparoscopic Surgery (FLS) tasks are
validated measures of technical skills. Certification requires
travel to a testing site and a fee, thus a reliable method of
predicting readiness for the exam would be advantageous. The
ProMIS simulator (Haptica) provides automated scoring. FLS tasks
can be placed in the ProMIS simulator and scored using time
(TT), as well as motion analysis metrics: instrument path length
(PL) and instrument smoothness (IS). This study was designed to
evaluate these automated ProMIS metrics and their ability to
predict readiness for FLS certification.
Methods
33 subjects (12 students, 16 residents PGY 1-4, and 5 experts)
performed FLS tasks in the standard simulator and in ProMIS.
Tasks were scored by FLS and ProMIS metrics. For each ProMIS
metric, the total score was calculated by summing the scores for
the 5 FLS tasks. Pearson’s correlations were calculated for
ProMIS metrics versus standard FLS scores. Multivariate
regression analysis identified independent predictors of standard
FLS performance. These variables were then used for sensitivity
and specificity calculations in order to establish a ProMIS pass-
fail score for predicting readiness for FLS certification.
Significance was defined as p<0.05.
Results
TT (r= -0.82), PL (r= -0.56), and IS (r= -0.75) all correlated
significantly with standard FLS score. Multivariate regression
analysis identified TT as the strongest predictor of FLS score. A
TT score of 1000 maximizes sensitivity and specificity and was
identified as the pass-fail for reliably predicting FLS
performance.
Conclusions
Automated ProMIS metrics correlate well with standard FLS
performance. In this study sample, a TT score less than 1000
reliably predicted a passing FLS certification score.
4. SAGES 2006 Scientific Sessions S065
WHAT CAN MOTION DERIVATIVES TELL US ABOUT SKILL
PERFORMANCE?
Laurel N Vuong BS, Steven D Schwaitzberg MD, Caroline G
Cao PhD, Tufts University School of Medicine, Cambridge
Health Alliance, Tufts University School of Engineering
Surgical simulators are a popular topic of discussion on
training in laparoscopic surgery. They reduce the need to
use human cadavers or animal models for skills
development. A subset of the MISTELS methodology has
been employed in the manual skills assessment for the
Fundamentals Laparoscopic Skills (FLS) program because it
was shown to be a valid discriminator of surgical
experience. Pure performance outcome, such as time to
task completion and number of errors, is used for scoring,
which is dependent on the consistency of the scorer. A new
simulator environment has been created which uses motion
tracking for measurement of performance outcome
measures and motion derivatives such as smoothness and
efficiency.
The purpose of the study was to determine if the motion
derivatives can be used to automatically and objectively
discriminate experience levels. Twenty-one subjects (6
medical students, 14 surgical residents, and 1 expert
surgeon) were recruited to perform the following tasks: peg
transfer, pattern cutting, pre-tied loop placement,
extracorporeal and intracorporeal knot-tying in the new
simulator environment. Subjects were evaluated on time to
completion, errors, smoothness and total path length (used
to calculate efficiency).
Results show that experience level is still distinguishable
when using task-dependent parameters to evaluate
performances during peg transfer (p= 0.035), pre-tied loop
placement (p= 0.022), extracorporeal (p= 0.0006) and
intracorporeal (p= 0.025) knot tying in this new simulator
environment.
Evaluation of performance using task-independent
parameters significantly distinguished training level in three
tasks: (1) smoothness of the left instrument was significant
as a function of experience level in extracorporeal knot-
tying (p= 0.016), (2) efficiency (total path length divided by
time to completion) was also significant in the movement of
the right tool as a function of experience level in peg
transfer (p= 0.0011) and (3) pre-tied loop placement (p=
0.013979).
This preliminary analysis shows that automatic and
objectively measured motion derivatives can be associated
with the level of experience. These results indicate a
potential for the application of an automatic and objective
means of skills evaluation.
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
5. SAGES 2007 ETP057
OBJECTIVE MEASUREMENT OF FLS PRECISION CUTTING TASK
Derek Young, Fiona Slevin, Derek Cassidy, Donncha Ryan, Haptica
Inc
The Precision-Cutting Task in the SAGES/ACS FLS Program
requires the user to dissect a circle of specific size and shape
from a marked piece of mesh. Currently, measurement of the
accuracy and area dissected is done by observation and by
measuring the dissected mesh on a measurement grid. Using
advanced vision-tracking, the ProMIS surgical simulator takes an
image of the dissected mesh and automatically generates a
metric, indicating the accuracy of the shape and area dissected.
Method
1. Once the user has completed the Precision-Cutting Task,
ProMIS takes an image of the dissected mesh. The image is
converted to binary image and then scanned using a blob
detection algorithm which produces a list of blobs.
2. The blob with the largest area is taken as the cut out area and
the number of pixels are counted inside this area.
3. The actual measurement for area is given in cm2. This is
calculated by counting the number of pixels in a known area of
the image and then using ratios to determine the area of the cut
out. (Note: this calibration step is achieved by taking the tissue
off the tray and running the blob detection and pixel count on
the uncovered black foam of which the exact area is known)
Results
Initial trials of the metrics on the FLS Precision- Cutting Task
show that ProMIS is as accurate — and frequently more accurate
— than the current human observation method.
6. SAGES 2006 Education/Outcomes–P224
COMPLEX LAPAROSCOPIC TASK PERFORMANCE ON TWO
NEW COMPUTER-BASED SKILLS TRAINING DEVICES
Erika K Fellinger MD, Michael E Ganey MD, Anthony G
Gallagher PhD, Daniel J Scott MD, Ron W Bush BS, Neal E
Seymour MD, Department of Surgery, Baystate Medical
Center, Springfield, MA
Introduction
New computer-based skills training devices can simulate and
measure performance of complex surgical tasks. The aim of
this study is to determine basic face and construct validity
characteristics of two new devices configured for
laparoscopic suturing and knot-tying tasks.
Methods
At the 2005 SAGES meeting, Learning Center attendees
evaluated two computer-based skills training platforms:
SurgicalSIM (SS), a virtual reality (VR) device (METI,
Sarasota, FL; SimSurgery, Oslo, Norway) and ProMIS (PM), a
computer-enhanced video trainer (Haptica, Ltd., Dublin,
Ireland). Demographic and training data were collected
from 73 subjects. All were asked to perform 2 iterations of
laparoscopic suturing and intracorporeal knot-tying (10-
minute time limit) on each device. A 6-question survey was
used to define impressions of task realism, relevance, and
execution using a 5-point Likert scale. Performance data
(SS: time, path length, errors; PM: time, path length,
smoothness) were collected on both devices and
comparisons made between user-defined expert and
nonexpert (intermediate and novice) groups (ANOVA and
Mann Whitney U test).
Results
46 subjects used SS and 56 used PM. Task completion rate
was 80% for SS and 93% for PM. Experts performed better
than nonexperts for all performance measures on SS
(composite score 496±41 vs 699±60, p < 0.005) and PM
(974±111 vs 1466±89, p < 0.005). Post-task survey scores for
PM were significantly higher for perceived realism (overall
realism and thread behavior), reflection of clinical ability,
and overall educational value. Perception of educational
value was not significantly different between the devices
among subjects with prior VR experience.
Conclusions
Using subject-defined expert and nonexpert groups,
construct validity was demonstrated for all performance
measures on both training devices. Surveyed face validity
measures favored the non-VR device, but results also
suggest that subjects with prior VR training experience are
more apt to accept a new VR surgical training platform.
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
7. World J Surg. 2007 Apr;31(4):764-72.
AUGMENTED VERSUS VIRTUAL REALITY LAPAROSCOPIC
SIMULATION: WHAT IS THE DIFFERENCE? : A COMPARISON
OF THE PROMIS AUGMENTED REALITY LAPAROSCOPIC
SIMULATOR VERSUS LAPSIM VIRTUAL REALITY
LAPAROSCOPIC SIMULATOR.
Botden SM, Buzink SN, Schijven MP, Jakimowicz JJ. Catharina
Hospital, Eindhoven, The Netherlands.
Background
Virtual reality (VR) is an emerging new modality for laparoscopic
skills training; however, most simulators lack realistic haptic
feedback. Augmented reality (AR) is a new laparoscopic
simulation system offering a combination of physical objects and
VR simulation. Laparoscopic instruments are used within an
hybrid mannequin on tissue or objects while using video tracking.
This study was designed to assess the difference in realism,
haptic feedback, and didactic value between AR and VR
laparoscopic simulation.
Methods
The ProMIS AR and LapSim VR simulators were used in this study.
The participants performed a basic skills task and a suturing task
on both simulators, after which they filled out a questionnaire
about their demographics and their opinion of both simulators
scored on a 5-point Likert scale. The participants were allotted
to 3 groups depending on their experience: experts,
intermediates and novices. Significant differences were
calculated with the paired t-test.
Results
There was general consensus in all groups that the ProMIS AR
laparoscopic simulator is more realistic than the LapSim VR
laparoscopic simulator in both the basic skills task (mean 4.22
resp. 2.18, P < 0.000) as well as the suturing task (mean 4.15
resp. 1.85, P < 0.000). The ProMIS is regarded as having better
haptic feedback (mean 3.92 resp. 1.92, P < 0.000) and as being
more useful for training surgical residents (mean 4.51 resp. 2.94,
P < 0.000).
Conclusions
In comparison with the VR simulator, the AR laparoscopic
simulator was regarded by all participants as a better simulator
for laparoscopic skills training on all tested features.
8. SAGES 2007 Education/Outcomes – P270
COMPARISON AND VALIDATION OF TWO DIFFERENT
SURGICAL SKILLS SIMULATORS
Dieter Hahnloser MD,Rachel Rosenthal MD,Christian
Hammel,Daniel Oertli,Markus Müller,Pierre-Alain Clavien,
Department of Visceral and Transplantation Surgery,
University Hospital Zurich, Switzerland
Background
Simulators are increasingly incorporated in surgical training
and validation is important. The simulations need to
resemble the task they are based upon (face validity) and
the simulator should be able to differentiate between levels
of experience (construct validity).
Aim
To assess two different types of computer-based simulators:
the fully computerised virtual reality (VR) simulator Xitact
LS500 (VR-simulator) and the hybrid ProMisTM simulator.
Methods: 146 participants (61%) of the 22nd Davos
International Gastrointestinal Surgery Workshop performed
on a voluntary basis three similar exercises (camera
navigation, clip and cut, and dissection) on the two
different simulators. Objective performance parameters
recorded by either simulator and subjective evaluation by
questionnaire were compared between beginner (n=73) and
advanced participants (n=73).
Results
The camera navigation exercise was completed by 52% of
the participants on the VR- and by 47% on the hybrid
simulator with no difference in performance parameters
between beginners and advanced trainees. The hybrid
simulator was graded more realistic (70% vs. 20%, p=.001)
and more useful (65% vs. 36%, p=.043) than the VR-
simulator. Participation was higher at the clip and cut
exercise (75% VR- and. 52% hybrid simulator) and advanced
trainees performed significantly better (shorter tool-tip-
travel distance, smoother, quicker and with higher score) on
both simulators compared to beginners. The clip and cut
exercise was graded more realistic on the hybrid (81% vs.
44%, p=.007) and similar useful on both simulators (77% vs.
72%). The dissection exercise was completed more often on
the hybrid simulator (47% vs. 23%, p=0.002). Only the hybrid
simulator was able to distinguish between advanced
trainees and beginners, with significantly higher scores for
all performance parameters for the latter. The hybrid
simulator was graded more realistic (70% vs. 33%, p=.016)
and more useful (83% vs. 62%, p=.12). Overall, acceptance
of requirement to train on and to be evaluated by such
simulators is still low (53% and 50%, respectively).
Conclusion
Fully computerized VR- or hybrid simulator performance
parameters can distinguish between beginner and advanced
trainees for perceptual motor skills (proving construct
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
validity), but not for visuo-spatial exercises such as the camera
navigation.
__________________________________________
9. SAGES 2007: S077
COMPUTER-BASED LAPAROSCOPIC AND ROBOTIC SURGICAL
SIMULATORS: PERFORMANCE CHARACTERISTICS AND
PERCEPTIONS OF NEW USERS
David W Lin MD, John R Romanelli MD, Renee E Thompson
MD,Michael E Ganey MD, Ron W Bush BS, Neal E Seymour MD,
Baystate Medical Center, Department of Surgery
The expanding inventory of advanced surgical training devices
now includes simulators for laparoscopic and robotic surgery. In
order to define perceptions of the need and value of such
devices, we evaluated the initial experience of surgeons using
both in the course of performance of an advanced laparoscopic
skill.
Methods
At the 2006 SAGES meeting, 62 Learning Center attendees evaluated a
new virtual reality (VR) robotic surgery simulator (RS) [SimSurgery, Oslo,
Norway] as well as either a computer-enhanced laparoscopic [ProMIS
(PM), Haptica, Ltd, Dublin, Ireland] or a VR simulator [SurgicalSIM (SS),
SimSurgery and METI, Inc, Sarasota, FL]. Demographic and training data
were collected and all were assessed during one iteration of
laparoscopic suturing and knot-tying on RS and either PM or SS. An 8-
question survey was used to determine users? impressions of task
realism, interface quality, and educational value (5-point Likert scale).
Performance data [time, path length, smoothness (PM), errors (SS/RS)]
were collected and comparisons made between user-defined groups and
different simulation platforms (Mann-Whitney Test, ANOVA).
Results
Task completion rate was greater for experts than nonexperts on all
platforms (PM 100% vs 75%; SS 100% vs 36%; RS 93% vs 36%). Experts
performed better than nonexperts on all performance measures on PM
(p<0.05: time 154±16 vs 205±12; path length 820±97 vs 1287±97;
smoothness 952±111 vs 1582±127). There were no significant differences
between experts and nonexperts for SS and RS performance measures.
Perception of value of haptic features was less for subjects with prior robot
experience (n=10; p<0.05). Otherwise realism, interface quality, and
educational value scores did not differ on the basis of prior simulator or
robot use. Nonexperts found that robotic simulation better reflected
clinical skill than did experts. Overall, subjective quality was scored higher
for PM than for SS or RS.
Conclusions
The ability of performance metrics of the computer-enhanced simulator
to discern predicted performance differences between experts and
nonexperts was better than for VR devices with a single task iteration.
Initial use of VR devices was associated with a lower overall perception
of realism and educational value as compared to use of physical objects
in the non-VR simulator. This may reflect the need for familiarization
with the computer-generated environment before the educational
potential of VR can be realized.
__________________________________________
10. 11. SAGES 2006 Scientific Sessions S096 A COMPUTERIZED ANALYSIS OF ROBOTIC VERSUS
LAPAROSCOPIC TASK PERFORMANCE
V K Narula MD, W C Watson MD, S S Davis MD, K Hinshaw BS,
B J Needleman MD, D J Mikami MD, J W Hazey MD, J H
Winston MD, P Muscarella MD, M Rubin, V Patel MD, W S
Melvin MD, The Ohio State University. CMIS. Columbus, OH
Introduction
Robotic technology has been postulated to improve
performance in advanced surgical skills. We utilized a novel
computerized assessment system to objectively describe the
technical enhancement in task performance comparing
robotic and laparoscopic instrumentation.
Methods and Procedures
Advanced laparoscopic surgeons (2- 10 yrs experience)
performed 3 unique task modules using laparoscopic and
Telerobotic Surgical Instrumentation (Intuitive Surgical,
Sunnyvale, CA). Performance was evaluated using a
computerized assessment system (ProMIS, Dublin, Ireland)
and results were recorded as time (sec), total path (mm)
and precision. Each surgeon had an initial training session
followed by two testing sessions for each module. A Paired
Student’s T-Test was used to analyze the data.
Results
10 surgeons completed the study. Objective assessment of
the data is presented in the table below. 8/10 surgeons had
significant technical enhancement utilizing robotic
technology Laparoscopic vs Robotic Time (sec) Total
Path(mm) Precision Module 1210 vs 161 # 11649 vs 5571 *
1434 vs 933 * Module 2119 vs 68 * 5573 vs 1949 * 853 vs 406 *
Module 377 vs 55 * 4488 vs 2390 * 552 vs 358 * # = p < 0.009 *
= p<0.001 Conclusions: The ProMIS computerized assessment
system can be modified to objectively obtain task
performance data with robotic instrumentation. All the
tasks were performed faster and with more precision using
the robotic technology than standard laparoscopy.
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
12. SAGES 2006 Education/Outcomes–P219
INTEGRATING SIMULATION LAB TRAINING INTO A SURGICAL
RESIDENCY PROGRAM: IS VOLUNTARY PARTICIPATION
EFFECTIVE
Lily Chang MD, James Petros MD, Donald Hess MD, Caroline
Rotondi BA, Timothy Babineau MD, Boston Medical Center
Objective
Surgical training programs nationwide are struggling with the
integration of simulation training into their curriculum given the
constraints of the 80-hour work week. We examine the
effectiveness of voluntary training in a simulation lab as part of
the surgical curriculum. Methods: The ProMIS simulator was
introduced into the general surgery residency at Boston
University Medical Center. All categorical residents (28) were
required to attend a 2-hour training session and curriculum
review. Non-categorical residents (23) were given the option to
complete training. After the introductory session, time spent in
the lab was encouraged, but voluntary. Use of the simulator was
tracked for all residents. Participation in the simulation
curriculum was defined as 3 or more uses of the simulator. After
3 months, all residents completed a survey regarding the
simulation lab and their simulator usage. Results: 26 (93%)
categorical residents and 3 (6%) non-categorical residents
completed the introductory simulator training session. Over a 3
month period, use of the simulator at least once was 31% among
all eligible residents; 80% of PGY1, 40% of PGY2, 60% of PGY3,
and 0% of PGY4 and PGY5. Four residents (14%) participated in
the simulation curriculum. 71% of simulator usage was during
working hours while 29% was completed post-call or off duty.
Most residents agreed that the simulator was easy to use and
improved their operative skills, but did not think it was a good
substitute for actual operative experience. Reported reasons for
not using the simulator included off-site rotation (44%), no time
(30%), and no interest (11%).
Conclusions
Voluntary use of a surgical simulation lab leads to minimal
participation in a training curriculum. Participation should be
mandatory if it is to be an effective part of a residency
curriculum.
13. SAGES 2007: S080 THE IMPACT OF KNOWLEDGE OF RESULTS IN SURGICAL
SKILLS TRAINING
A. O’Connor MD, C. Cao PhD, S. Schwaitzberg MD,
Department of Mechanical Engineering, Tufts University.
Background
Concerns about the adequacy of advanced laparoscopic
training continue to be raised despite a proliferation of
training systems exist. The manner in which the training
modules are structured to maximize learning has not been
examined. There are many aspects to the accumulation of
laparoscopic skills during training, one of which is
Knowledge of Results (KR), i.e. the information provided to
individuals about the outcomes of their motor responses in
their environment. We studied the effects of KR on the
learning curve of laparoscopic suturing and knot tying.
Aims
We evaluated the learning curves of 9 medical students with
no previous laparoscopic surgical experience under three
different conditions, each with different levels of
knowledge of results.
Methods
Subjects were randomly assigned to one of three groups.
Each subject attended a training session for 1 hour each
day, 6 days a week for 4 weeks. Group 1 (No feedback)
received no knowledge of results (KR) and no performance
feedback. Group 2 (feedback only) received factual KR
following each training session, but no coaching. Group 3
(feedback and coaching) received KR and coaching. Learning
curves were plotted based on task time, smoothness of
instruments and instrument’s path length. The task used
was an intracorporeal suture/knot tying in the ProMIS
laparoscopic simulator. Perceived workload for each session
was recorded using a standardized NASA TLX workload
score.
Results
The variability across each session for each student was
calculated for each of the three parameters. There was
statistical significance between the groups for all
parameters (p-values 0.0002, 0.0002 and 0.009). Significant
differences were found between groups 2 and 3 and group 1
(p values 0.0314-0.0410) Groups 2 and 3 learned
significantly faster than those in Group 1, reaching
performance plateaus at earlier sessions. There were no
significant differences between groups 2 and 3 (p-values
0.1211, 0.1758 and 0.1375). Providing individuals with
knowledge of results lowered their perceived workload,
adding instructional feedback lowered this even further.
These results demonstrate that KR is essential for efficient
surgical skill acquisition. Individual coaching, a labor
intensive proposition, reduces workload but has NO added
beneficial effect on the speed of learning. These results
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
provide a useful basis for developing efficient and cost effective
surgical skills training curriculum.
____________________________________________________
14. SAGES 2007: Education/Outcomes – P267
THE CONSTRUCT VALIDITY OF COMPUTER-DERIVED
PERFORMANCE METRICS FOR SELECTED SIMULATED
LAPAROSCOPIC TASKS
J. A Oostema MD, Matthew Abdel BS, Jon C Gould MD, University
of Wisconsin School of Medicine and Public Health, Department
of Surgery
Introduction
A surgical skills assessment tool is said to demonstrate evidence
of construct validity if users with more experience, and by
inference more skill, perform better or more efficiently.
Computer derived motion metrics such as smoothness (the
number of times an instrument tip changes velocity during a
task) and path length may be more sensitive measures of skill for
a particular task than traditional metrics such as time.
Methods
Twenty-four medical students (third year), 19 surgical residents
(PG1-5), and 3 attending surgeons were asked to perform four
different tasks 3 times in a hybrid computer-based physical
laparoscopic trainer (ProMIS, Haptica Inc., Dublin). The 4 tasks in
order of complexity were laparoscopic orientation (Task 1),
object positioning (Task 2), sharp dissection (Task 3), and intra-
corporeal knot tying (Task 4). Metrics recorded were time, path
length, and smoothness. Laparoscopic operative experience for
each user was quantified using case logs. Correlations were
determined using regression analysis and ANOVA.
Results
A statistically significant correlation was observed between
experience and performance for all three metrics for tasks 2-4
(all p< 0.01). Smoothness was the only metric to correlate in the
laparoscopic orientation task. Within tasks, time and smoothness
correlate much more strongly with experience and to a similar
degree. The strongest correlation was observed for the knot
tying task (r2=0.60 for time and 0.59 smoothness). An r2=1.0
would represent a perfect correlation between experience and
the specified metric.
Conclusions
The computer-derived metrics measured by the hybrid trainer
correlate with laparoscopic experience. Further study is
necessary to determine if specific metrics are better indicators
of actual skill.
15. SAGES 2007 Education/Outcomes – P284
ABSTRACT VIRTUAL REALITY TRAINING DEVELOPS CORE
LAPAROSCOPIC SKILLS COMPARABLE TO EXPERIENCED
LAPAROSCOPIC SURGEONS: RESULTS OF A PROSPECTIVE
RANDOMIZED TRIAL COMPARING TWO VIRTUAL REALITY
TRAINERS
E. Matt Ritter MD, Elisabeth A Pimentel BA, Ryan E Earnest
BS, Randy S Haluck MD, Mark W Bowyer MD, National Capital
Area Medical Simulation Center, Uniformed Services
University, Bethesda, Maryland / Department of Surgery,
Pennsylvania State College of Medicine, Hershey,
Pennsylvania
Introduction
While simulation is becoming more widely accepted in
surgical training, comparative trials on the training
effectiveness of these simulators are lacking. We sought to
compare the effectiveness of two abstract virtual reality
trainers to train laparoscopic skills as assessed by the
Fundamentals of Laparoscopic Surgery (FLS). We then
compared the post training performance of the novice
subjects with a group of experienced surgeons.
Methods and Results
20 novice medical students were recruited. Each subject
performed a pre-test consisting of 3 FLS tasks - Peg Transfer
(PT), Pattern Cut (PC) and Intracorporeal Suture (IS) -
placed in the ProMIS augmented reality simulator (Haptica,
Ireland). They were then randomized to train to
predetermined levels of proficiency on 3 tasks of the
Minimally Invasive Surgical Trainer-Virtual Reality (MIST-VR)
(Mentice, Sweden) or the Rapid Fire/Smart Tutor (RFST)
(Verefi, Elizabethtown, PA). After reaching the proficiency
levels, both groups then took a post test consisting of 3
trials of the same tasks used for the pre-test. Post test
performance by both groups was then compared to a control
group, composed of 10 experienced surgeons who had
completed the same post test.
MIST-VR and RFST groups demonstrated statistically
significant improvement from the pre-test to the post test
on all 3 FLS tasks (p < 0.0001). There was no significant
difference in post test performance between the MIST-VR
and RFST groups. When the simulation trained groups were
compared to experienced controls there was no significant
difference in performance with respect to PT. The
experienced controls did significantly outperformed the
MIST-VR group in PC (p<0.01) and IS (p<0.05), but
differences between the experienced controls and the RFST
group did not reach statistical significance.
Conclusion
Simulation based training on either the MISTVR or the RFST
simulator improves the skill level of novices as assessed by
FLS. The post training skill level of these novices compares
favorably with a group of experienced surgeons. Virtual
Reality trainers, such as RFST and MISTVR, train
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
fundamental laparoscopic skills equally and to a level
comparable to a group of experienced practicing surgeons.
____________________________________________________
16. SAGES07 Ergonomics/Instrumentation P307
A COMPUTERIZED ANALYSIS OF STANDARD VERSUS HIGH
DEXTERITY LAPAROSCOPIC INSTRUMENTATION IN TASK
PERFORMANCE
V K Narula MD,K M Reavis MD,D R Renton MD,D J Mikami MD,B J
Needleman MD,J W Hazey MD,K E Hinshaw BS,W S Melvin MD,
THE OHIO STATE UNIVERSITY HOSPITAL, CENTER FOR MINIMALLY
INVASIVE SURGERY
Introduction
Minimally invasive surgery is becoming the standard of care for
the majority of abdominal procedures. Laparoscopic
instrumentation is constantly undergoing improvements to give
surgeons an advantage. Articulated instrumentation provides a
distinct advantage in the field of robotic surgery. Applying the
same principles to standard laparoscopic instrumentation could
offer increased degrees of freedom to make complex
laparoscopic tasks easier to perform. We utilized a novel
computerized assessment system to objectively evaluate task
performance comparing Standard and High Dexterity (HD)
laparoscopic instrumentation.
Methods
Advanced laparoscopic surgeons (2-12yrs experience) performed
3 unique task modules utilizing Standard and HD laparoscopic
instrumentation (Novare Surgical Systems, Cupertino, CA).
Performance was evaluated using a computerized assessment
system (ProMIS, Dublin, Ireland) and results were recorded as
time (sec), path (mm), and precision. Each surgeon had an initial
training session followed by two testing sessions for each
module. A Paired Student’s T-Test was used to analyze the data.
Results: Nine surgeons completed the study. Objective
assessment of the data is presented in the table below. Module 1
was statistically significant, whereas Module 2 and 3 showed no
difference in task performance with the HD instrumentation.
Conclusion
HD instrumentation is in its infancy. Results showed no
advantage using HD instrumentation. This could be due to the
learning curve associated with new instrumentation and
technology. With future developments in HD technology and
training, the user interface will improve and may offer an
advantage over standard laparoscopic instrumentation.
17. SAGES 2006 Emerging Technologies P036
AUGMENTED REALITY SIMULATOR FOR HAND-ASSISTED
LAPAROSCOPIC COLECTOMY
Derek Young, Derek Cassidy, Fiona Slevin, Donncha Ryan,
Haptica Ltd, Dublin, Ireland.
Training in Hand-Assisted Laparoscopic Colectomy (HALC)
has largely been done using cadavers and porcine models.
These have drawbacks in terms of realism, logistics and lack
of performance measurement. A Simulator would provide
consistent instruction and practice and provide feedback on
performance. However, given the range of instruments used
in the procedure, and especially the use of a hand, pure
virtual reality could not be considered as a solution.
The ProMIS Augmented Reality simulator platform – by
combining physical and virtual reality - enables interaction
and tracking of real instruments with a physical model. And
because of its technological approach (vision-tracking) also
enables the hand to be tracked. In the new ProMIS HALC
simulator, 3D models or graphical objects are overlaid on
the physical model to provide instruction and guidance. For
example, a 3D animation may be used to demonstrate how
to complete a step; a graphical guideline ‘A – B’ may be
used to indicate a target area for dissection. ProMIS HALC
measures surgical skill by gathering data on the movement
of commercial laparoscopic instruments while completing a
standardized task. The main performance metrics are time
taken, total path length and economy of movement.
Additionally metrics specific to a step are calculated to
measure performance associated with a specific instruction
in a specific region of the physical model. Following the
simulated procedure, the user completes a self-assessment
which contributes to the metrics for the full procedure. A
full analysis is of performance is presented to the user on
completion of the procedure and self-assessment.
Results
Initial trials of the ProMIS HALC simulator indicate that
practice on the simulator improves performance as
measured by the metrics gathered by the simulator. While
detailed validation studies remain to be done, initial
indicators are that the HALC simulator represents an
“unparalleled opportunity to practice, step by step, a Hand-
assisted laparoscopic sigmoid resection” and “a huge step
forward in surgical training”.
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
18. May 2006 North of England Surgical Society Annual Registrar’s Meeting Winner of the George Feggeter Gold Medal
LAPAROSCOPIC SKILLS ACQUISITION: IS PSYCHOMETRIC
MOTION ANALYSIS A VALID ASSESSMENT TOOL?
Pellen MGC1;2, Barton JR2, Horgan LF1, Attwood SE1 Northumbria
Upper Gastrointestinal Team of Surgeons1; University of
Newcastle upon Tyne2, Newcastle upon Tyne, United Kingdom
Aims
In an evolving climate of competency-based assessment, reliable
and validated methods of objective skills assessment are
required for trainee surgeons. We aimed to assess whether the
ProMIS Simulator (Haptica, Dublin, Ireland) offers a method of
assessing laparoscopic psychomotor performance.
Methods
Volunteers comprising 17 experienced laparoscopists ( >100
laparoscopic cholecystectomies) and 38 medical students novices
(no laparoscopic experience) performed 3 simulated tasks
comprising virtual reality camera navigation, object transfer and
sharp dissection task (glove over balloon). A further group of 28
basic surgical trainees (experience limited to 1st assistant)
attending BSS Courses were assessed on the same tasks before
and after training in laparoscopic skills. Data metrics of time,
smoothness and path length were measured via optical tracking
of instrument movement. Objective observations of specific
errors were also recorded.
Results
Non-parametric analysis demonstrated experienced
laparoscopists performed all 3 tasks significantly faster,
smoother and with more economy of movement (p<0.05),
excluding camera navigation path length. Experienced
participants performed sharp dissection more accurately (p<0.01)
although no difference in balloon puncture was seen. Repeat
assessment of BSS Course Trainees showed significant
improvements in simulator metrics (Paired T test, P<0.05),
although smaller yet significant improvements in “untrained”
student performance was also seen.
Conclusions
Gross analysis of these metrics can distinguish between
experience levels supporting construct validity of these simulator
tasks. These results suggest potential for objectively measuring
baseline skill level and response to training. Further work will
examine the effect of interface familiarisation and defining
target levels of performance in simulated tasks.
19. EAES 2006, Poster DEVELOPING PSYCHOMETRIC ASSESSMENT OF
LAPAROSCOPIC SKILLS USING THE PROMIS SIMULATOR
Pellen MGC1;2, Barton JR2, Horgan LF1, Attwood SE1
Northumbria Upper Gastrointestinal Team of Surgeons1;
University of Newcastle upon Tyne2, United Kingdom
Aims
Reliable and validated methods of objective skills training
and assessment are required for trainee surgeons. The
ProMIS Simulator (Haptica, Ireland) potentially offers a
method of assessing laparoscopic psychomotor performance.
We present initial data from our Centre and Royal College of
Surgeons Basic Surgical Skills (BSS) Courses.
Methods and results
Volunteers comprising 17 experienced laparoscopists (>100
laparoscopic cholecystectomies) and 38 medical students
novices (no laparoscopic experience) were assessed on a
complex sharp dissection task (glove over balloon). A
further group of 28 basic surgical trainees (experience
limited to 1st assistant) attending BSS Courses were assessed
on the same task before and after training in laparoscopic
skills. Data metrics of time, smoothness and path length
were measured via optical tracking of instrum-ent
movement as well as observations of specific errors.
Data analysis (ANOVA) demonstrated experienced
laparoscopists performed target dissection at least 50%
faster, smoother and with more economy of instrument
movement than students (p<0.05). Experienced participants
performed sharp dissection more accurately (p<0.01)
although no difference in balloon puncture frequency was
seen. Similarly significantly better performance over
trainees was demonstrated. Trainees showed only
significantly smoother instrument handling when compared
to students, possibly reflecting greater baseline dexterity in
this selective group. Repeat assessment following course
training showed significant improvements in all metrics by
32-40% (Paired T test, P<0.05). Whilst significant
improvements were also demonstrated in repeat assessment
of the untrained student group, these were less marked (15-
18%).
Conclusions
The gross analysis of these metrics can distinguish between
experience levels supporting the construct validity of this
simulator task. These results suggest a potential role for
objectively measuring baseline skill level and response to
training in distinct psychomotor challenges. Further work in
progress is examining the effect of interface familiarisation
and repeated task performance on novice learning curves
and defining target levels of performance in a range of
simulated tasks.
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
20. Surg Endosc (May 2006) 20: 900–904 CONSTRUCT VALIDATION OF A NOVEL HYBRID SURGICAL
SIMULATOR
D. Broe, P. F. Ridgway, S. Johnson, S. Tierney, K. C. Conlon
Department of Surgery, Professorial Surgical Unit, Level 4, The
Adelaide and Meath Hospital, incorporating the National
Children’s Hospital, Tallaght, Dublin 24, Ireland
Background
Simulated minimal access surgery has improved recently as both
a learning and assessment tool. The construct validation of a
novel simulator, ProMis, is described for use by residents in
training.
Methods
ProMis is a surgical simulator that can design tasks in both virtual
and actual reality. A pilot group of surgical residents ranging
from novice to expert completed three standardized tasks:
orientation, dissection, and basic suturing. The tasks were tested
for construct validity. Two experienced surgeons examined the
recorded tasks in a blinded fashion using an objective structured
assessment of technical skills format (OSATS: task-specific
checklist and global rating score) as well as metrics delivered by
the simulator.
Results
The findings showed excellent interrater reliability (Cronbach_s
a of 0.88 for the checklist and 0.93 for the global rating). The
median scores in the experience groups were statistically
different in both the global rating and the task-specific
checklists (p < 0.05). The scores for the orientation task alone
did not reach significance (p = 0.1), suggesting that modification
is required before ProMis could be used in isolation as an
assessment tool.
Conclusions
The three simulated tasks in combination are construct valid for
differentiating experience levels among surgeons in training.
This hybrid simulator has potential added benefits of marrying
the virtual with actual, and of combining simple box traits and
advanced virtual reality simulation.
21. Surgical endoscopy ISSN: 0930-2794 (Paper) 1432-2218 (Online)
CONSTRUCT VALIDATION OF THE PROMIS SIMULATOR
USING A NOVEL LAPAROSCOPIC SUTURING TASK
K. R. Van Sickle1, D. A. McClusky III1, A. G. Gallagher and
C. D. Smith1
Background
The use of simulation for minimally invasive surgery (MIS)
skills training has many advantages over current traditional
methods. One advantage of simulation is that it enables an
objective assessment of technical performance. The
purpose of this study was to determine whether the ProMIS
augmented reality simulator could objectively distinguish
between levels of performance skills on a complex
laparoscopic suturing task.
Methodology
Ten subjects — five laparoscopic experts and five
laparoscopic novices — were assessed for baseline
perceptual, visio-spatial, and psychomotor abilities using
validated tests. After three trials of a novel laparoscopic
suturing task were performed on the simulator, measures
for time, smoothness of movement, and path distance were
analyzed for each trial. Accuracy and errors were evaluated
separately by two blinded reviewers to an interrater
reliability of >0.8. Comparisons of mean performance
measures were made between the two groups using a Mann-
Whitney U test. Internal consistency of ProMIS measures was
assessed with coefficient α.
Results
The psychomotor performance of the experts was superior
at baseline assessment (p < 0.001). On the laparoscopic
suturing task, the experts performed significantly better
than the novices across all three trials (p < 0.001). They
performed the tasks between three and four times faster (p
< 0.0001), had three times shorter instrument path length
(p < 0.0001), and had four times greater smoothness of
instrument movement (p < 0.009). Experts also showed
greater consistency in their performance, as demonstrated
by SDs across all measures, which were four times smaller
than the novice group. Observed internal consistency of
ProMIS measures was high (α = 0.95, p < 0.00001).
Conclusions
Preliminary results of construct validation efforts of the
ProMIS simulator show that it can distinguish between
experts and novices and has promising psychometric
properties. The attractive feature of ProMIS is that a wide
variety of MIS tasks can be used to train and assess technical
skills.
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
22. EAES 2004 Abstract nr.: O207 RELATIONSHIP BETWEEN MOTION ANALYSIS, TIME,
ACCURACY, AND ERRORS DURING PERFORMANCE OF A
LAPAROSCOPIC SUTURING TASK ON AN AUGMENTED REALITY
SIMULATOR
Author: D.A.M. McClusky, Emory University School of Medicine,
Atlanta Georgia, United States of America. Co-author(s): K. Van
Sickle, Emory University School of Medicine, Atlanta Georgia,
United States of America
A.G. Gallagher, Emory University School of Medicine, Atlanta
Georgia, United States of America
Background
Time, efficient movement, accuracy, and safety are reliable and
discriminative metrics of proficiency during virtual reality and
box-trainer based minimally invasive surgical (MIS) training. The
role these metrics may serve during more advanced skills training
are not well understood. Using a novel augmented reality
simulator, we sought to gain an understanding of the relationship
between these metrics during an advanced MIS suturing task.
Methods: Eleven subjects completed 3 trials of a suturing task
designed for a box-trainer and adapted for the ProMIS (Haptica,
Dublin, Ireland) simulator. Time, tool path, and smoothness of
movement were assessed using computer algorithms. Measures of
accuracy during suture placement and errors in performance
were assessed by two blinded reviewers trained to assess
performance with inter-rater reliability > 0.8. A Pearson’s
correlation coefficient was used to assess the strength of the
relationship between ProMIS metrics and suturing task
performance. Results: Of the ProMIS metrics, time correlated
with tool path distance and smoothness of movement in three
trials (range 0.914 – 0.957, p < 0.0001). When the suturing task
was analyzed, accuracy and error score demonstrated an equally
strong relationship (range -0.726 - -0.84, all p < 0.0001).
Combining all metrics, path distance correlated strongest with
accuracy (2 trials significant, range -0.67 - -0.93), and error
score (3 trials significant, range 0.54 – 0.61). Smoothness of
movement significantly correlated with accuracy in 2 trials
(range -0.63 - -0.88), and time correlated with error score in 2
trials (range 0.56 – 0.60).
Conclusion
Metrics based on movement efficiency and time, and those based
on task accuracy and error scores strongly correlate when
grouped independently. At this time, a proficiency curriculum
should incorporate both forms of analysis, however further
validation work is needed to replicate these findings and give
further insight into how ProMIS metrics relate to real-world
performance.
23. EAES 2004 abstract nr.: O208 PSYCHOMOTOR SKILLS ASSESSMENT IN PRACTICING
SURGEONS PERFORMING ADVANCED LAPAROSCOPIC
PROCEDURES II: DEMOGRAPHICS AND PERFORMANCE
PROFILES
Author: A.G. Gallagher, Emory University, Atlanta, United
States of America. Co-author(s): C.D. Smith, Emory
University, Atlanta, United States of AmericaR.M. Satava,
University of Washington, Seattle, United States of America
Background
This study reports on the objectively assessed psychomotor
performance of minimally invasive surgeons on a box-trainer
and a virtual reality (VR) task as a function of handedness,
gender, sight corrected status, and age. Methods: Two
hundred and ten surgeons attending the 2001 annual
meeting of the American College of Surgeons (ACS) in New
Orleans who reported having completed more than 50
laparoscopic procedures participated. Subjects completed a
box-trainer laparoscopic cutting task and a similar virtual
reality task twice. Demographic and laparoscopic
experience data was also collected. Results: There were no
significant differences between subjects performance on
either tasks in terms of handedness, gender or whether they
were sight corrected or not. A clear and consistent linear
trend emerged in terms of age. Older subjects (ages 60 – 69)
performed significantly worse than younger subjects (ages
30 – 39, 40 – 49) on the box-trainer task for correct incisions
(13.1 Vs 19.3, p < 0.008) and incorrect incisions (12.3 Vs 2.5,
p > 0.05). They also performed worse on the VR task for
time (132 Vs 71, p < 0.05), error (99 Vs 41, p < 0.05) and
economy of movement (22.8 Vs 11.7, p < 0.05). Conclusions:
Increasing age was found to be associated with a decline in
objectively assessed psychomotor performance on two well
validated laparoscopic tasks.
Acknowledgements
This study was supported with grants from the ACS, SAGES,
SLS, TATRC, and Emory University Endosurgery Unit.
Validation
www.haptica.com | email: [email protected] | tel (NAm) +617 342 7270 | tel (RoW) +353 (0) 1 676 7310
24. JSLS, Journal of the Society of Laparoendoscopic Surgeons, Vol. 11, No. 3. (September 2007), pp. 273-
302.
CRITERION-BASED TRAINING WITH SURGICAL SIMULATORS:
PROFICIENCY OF EXPERIENCED SURGEONS
Heinrichs, Wm, Lukoff, Brian, Youngblood, Patricia, Dev, Parvati,
Shavelson, Richard, Hasson, M Harrith, Satava, M Richard,
Mcdougall, M Elspeth, Wetter, Paul Alan
Objective
In our effort to establish criterion-based skills training for
surgeons, we assessed the performance of 17 experienced
laparoscopic surgeons on basic technical surgical skills recorded
electronically in 26 modules selected in five commercially
available, computer-based simulators.
Methods/Procedures
Performance data were derived from selected surgeons randomly
assigned to simulator stations, and practicing repetitively during
three one-half day sessions on five different simulators. We
measured surgeon proficiency defined as efficient, error-free
performance and developed proficiency score formulas for each
module. Demographic and opinion data were also collected.
Results
Surgeons’ performance demonstrated a sharp learning curve with
the most performance improvement seen in early practice
attempts. Median scores and performance levels at the 10th,
25th, 75th, and 90th percentiles are provided for each module.
Construct validity was examined for two modules by comparing
experienced surgeons’ performance with that of a convenience
sample of lessexperienced surgeons.
Conclusions
A simple mathematical method for scoring performance is
applicable to these simulators. Proficiency levels for training
courses can now be specified objectively by residency directors
and by professional organizations for different levels of training
or post-training assessment of technical performance. But data
users should be cautious due to the small sample size used in this
study and the need for further study into the reliability and
validity of the use of surgical simulators as assessment tools.
Summary comment
The simulators used included pure Virtual Reality Simulators and
two simulators with real haptics (including ProMIS): Lap Mentor
from Simbionix (pure Virtual Reality); LapSim from Surgical-
Science AB (pure Virtual Reality); SurgicalSIM from METI (pure
Virtual Reality); ProMIS from Haptica (real haptics with metrics);
LTS2000 ISM60 from RealSim (real haptics with metrics).
ProMIS out-performed the virtual reality simulators with a mean
effectiveness rating of 3.56 versus 3.22 and 3.11 for LapSim and
SurgicalSim (LapMentor also scored 3.56).
25. SAGES 2008 P174 FLS TEST IS TRANSFERABLE TO PROMIS SIMULATOR
Eric S Hungness MD, Albert Amini BA, Deb E Rooney MS, Eric
T Volckman MD, Nathaniel J Soper MD, Feinberg School of
Medicine, Northwestern University, Chicago, IL
Introduction
The McGill Inanimate System for Training and Evaluation
of Laparoscopic Skills (MISTELS) comprises five tasks with
an objective scoring system, and has been incorporated
by SAGES in their Fundamentals of Laparoscopic Surgery
(FLS) program. MISTELS has high inter-rater and test-
retest reliability and correlates with operative skill.
However, the FLS program is labor intensive, requiring a
trained proctor. The ProMIS simulator allows for
assessment of physical tasks (instrument path length and
instrument smoothness) through instrument tracking
technology. We hypothesized that the FLS scores
obtained in the ProMIS simulator as well as ProMIS
metrics would correlate with standard FLS scoring.
Methods
Twenty general surgery residents (13 junior and 7 senior)
had baseline laparoscopic skills assessed using MISTELS in
the standard FLS and ProMIS simulators (pre-test). Nine
junior and 4 senior residents had a post-test after four
weeks of training. Tasks were scored by FLS and ProMIS
metrics. Total path length (TP) and total smoothness (TS)
were calculated by adding the path lengths and
smoothness of each individual task. ANOVA was used to
compare the mean (SD) of total and individual task scores
for pre- and post-tests in the FLS and ProMIS simulators.
Student's t-test was used to compare ProMIS metrics.
Pearson's correlations were calculated for standard FLS
scores in relation to ProMIS FLS scores, TP and TS.
Significance was defined as p < 0.01.
Results
All residents showed statistically significant improvement in
post-test total and individual task FLS scores on either the
FLS or ProMIS simulator. 100% and 88% of residents achieved
passing post-test scores on the FLS and ProMIS simulator,
compared to 30% and 29.2% on the pre-test. There was no
difference in junior and senior resident posttest scores (87.6
vs 79.1). ProMIS path length and smoothness were
significantly reduced across all tasks (range 14- 68%). Total
ProMIS FLS scores (0.729), TP (-0.753) and TS (- 0.769)
significantly correlated with total standard FLS simulator
scores. All residents with TP < 4000mm or TS < 6000
achieved a passing total FLS score.
Conclusions
All surgical residents achieved a passing FLS score after a
4-week laparoscopic skills curriculum. FLS tasks are
transferable to the ProMIS simulator with traditional FLS
scoring and intrinsic ProMIS metrics being good
measurement tools. A ProMIS total path length <4000 mm
or total smoothness <6000 reliably predicts a passing FLS
score.