Developing an Effective Simulation Lab – “How to Evaluate Residents’ Skills with Simulation”...
If you can't read please download the document
Developing an Effective Simulation Lab – “How to Evaluate Residents’ Skills with Simulation” Ann Van Heest MD Director of Education Professor, University
Developing an Effective Simulation Lab How to Evaluate
Residents Skills with Simulation Ann Van Heest MD Director of
Education Professor, University of MN Dept of Orthopedic
Surgery
How do we measure competence? Idea Face validity: Does the test
is going to measure what it was intended to measure? Construct
validity: Can the test differentiate between experts and novices?
Reliability: Inter-rater and Intra-rater
Slide 4
OUTLINE Matching the Right educational level for residents
level of learning 1. Upper Extremity Surgical Skills Competency
Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy
Skills Box 4. G1 Basic Skills Mandate
n =285742 Program Year Global Rating Carpal Tunnel p =
0.040
Slide 14
n =285742 Program Year Carpal Tunnel Release Detailed Checklist
(100pts) p = 0.002
Slide 15
Pass/Fail Assessment Did the surgeon achieve the goal of
surgery?
Slide 16
n =0 / 21 / 75 / 07 / 04 / 02 / 0 Program Year Pass / Fail
Assessment p < 0.001
Slide 17
Distal Radius Fracture Fixation
Slide 18
Conclusions This study reports that a surgeons ability to
release a trigger finger does not correlate specifically to his or
her ability to perform a carpal tunnel release or to perform plate
fixation of a radius fracture. The results of this study would
indicate that, for 3 different surgical simulations representing
procedures of varying complexity, assessments by a single
assessment tool is not adequate. To completely understand a
residents abilities, assessment by checklist (understanding the
steps of the surgery), global rating scales (assessment of basic
surgical skills in light of lesser or greater complexity
surgeries), and pass/fail assessment (examination of adverse
events) are all necessary components.
Slide 19
Competence High Stakes Exam G3 level Pass vs Needs more
practice Competency advancement, not social advancement
DEMONSTRATING COMPETENCE IS GRADUATION REQUIREMENT
Slide 20
OUTLINE Matching the Right educational level for residents
level of learning 1. Upper Extremity Surgical Skills Competency
Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy
Skills Box 4. G1 Basic Skills Mandate
Slide 21
Surgical Simulation Training Program for Articular Fracture
Surgery Jenniefer Y. Kho, MD University of Iowa Hospitals and
Clinics Karam MD, Ohrt GT, Thomas GW, Yehyawi TM, Lafferty PM,
Anderson DD, Marsh JL
Slide 22
Aims Develop a comprehensive surgical simulation training
program utilizing an articular fracture model Investigate whether
simulator training can improve performance in junior residents
Slide 23
Methods 6 PGY-1 and 6 PGY-2 residents
Slide 24
Methods Interventio n (n=6) PretestTrainingPosttest Control
(n=6) Pretest No Training Posttest 6 PGY-1 and 6 PGY-2
residents
Slide 25
Sawbones Fracture Model with Barium coated articular
surface
Slide 26
Slide 27
Video capture
Slide 28
Methods - Intervention On-line cognitive module Video review
with traumatologist Skills module Interventio n (n=6)
PretestTrainingPosttest
Slide 29
Methods Outcome measures OSATS global rating scale Articular
reduction Hand motion Fluoroscopy time, radiation dose Face
Validity (Questionnaire)
Slide 30
Results OSATS GLOBAL RATING SCALE Control OSATS global rating
score p=0.06p= 0.68 Intervention
Slide 31
Results FLUOROSCOPY TIME Fluoroscopy time (s) Pretest Posttest
Control Intervention p= 0.03p= 0.16
Results Pretest Posttest Control Intervention Cumulative hand
distance (m) CUMULATIVE HAND DISTANCE p= 0.8p= 1.0
Slide 34
Results Face validity Scale ranges from 1 (most) to 5 (least
realistic) How realistic was this exercise overall? 1 2 3 4 5 How
realistic was the bone model visually? 1 2 3 4 5 How realistic was
the movement of the synthetic fracture fragments compared to that
of real bony fragments? 1 2 3 4 5 How realistic was the feel of the
synthetic bone during instrumentation compared to that of real
bone? 1 2 3 4 5 Overall average score 2.75 Acceptable, but room for
improvement
Slide 35
Discussion Improved OSATS score, decrease in fluoroscopy time
and radiation exposure Articular step-off, hand motion no different
Junior residents with little to no articular fracture experience
may benefit from short-term dedicated training
Slide 36
Collaboration with University of Minnesota Incorporation into
surgical skills curriculum at the University of Iowa Potential for
expansion to other programs
Slide 37
OUTLINE Matching the Right educational level for residents
level of learning 1. Upper Extremity Surgical Skills Competency
Testing 2. Articular Fracture Reduction Model 3. Basic Arthroscopy
Skills Box 4. G1 Basic Skills Mandate
Slide 38
Low-Fidelity Arthroscopic Simulation Can Differentiate Between
Experts and Novices Jon Braman MD, Robert Sweet, MD David Hananel,
SimPortal Lab Paula Ludewig, Ph.D Ann VanHeest, MD
Slide 39
Panel of Experts Visualization Triangulation Object
Manipulation
Slide 40
Triangulation 10 LED lights 10 switches Randomly order lights
(training) Pre-determined order lights (testing) Time Errors: Alarm
if touches metal VIDEO
Slide 41
Object Manipulation 3 bars on left with rings 3 bars on right
with rings Transfer rings from left to right (Switch scope) Time
Errors: Dropped rings VIDEO
Slide 42
Triangulation p = 0.0013 p = 0.0073
Slide 43
Object Manipulation p = 0.0190 Errors: Drops p=1.0
Slide 44
Basic Arthroscopic Skills Construct Validity Face Validity G1
level skills: Readiness for OR Multi-Center Trial Planned
Competency Testing: By year in training Translation to OR Scope
Skills
Slide 45
ABOS Surgical Simulation Mandate formal instruction in basic
surgical skills, provided longitudinally or as a dedicated
non-orthopaedic surgery rotation will be required. Skills training
will need to be designed to integrate with subsequent post graduate
years and should prepare the PGY 1 to participate in orthopedic
surgery cases. Skills training will need a dedicated space and a
curriculum which must include: Goals and objectives and assessment
metrics. Skills used in the outpatient management of injured
patients including splinting, casting, application of traction
devices and other types of immobilization. Basic operative skills
including soft tissue management, suturing, bone management,
arthroscopy, flouroscopy and use of basic orthopaedic
equipment.
Slide 46
ABOS/AOA-CORD/AAOS Develop a modular outline of surgical skills
curriculum 3 meetings (Sept, March, May) Posted on ABOS website
with access to all residency programs.
Slide 47
1. Sterile Technique and Operating Room Set-Up 2.Knot Tying
& Suturing 3.Basic Microsurgical Suturing 4.Soft Tissue
Handling Techniques 5.Casting and Splinting 6.Traction
7.Compartment Syndrome 8.Bone Handling Techniques - Osteotomy
9.Fluoroscopic Knowledge and Skills 10.K-Wire Techniques
11.Techniques Basic to Internal Fixation of Fractures 12.Principles
and Techniques of Fracture Reduction 13.External Fixation 14.Basic
Arthroscopy Skills 15.Basic Arthroplasty Skills 16.Joint Sspiration
and Injection 17.Patient Safety, Team Training, Obtaining
Consent
Slide 48
Example of Module
Slide 49
How to measure Competency? 1.Upper Extremity Surgical Skills
2.Articular Fracture Reduction Model 3. Basic Arthroscopy Skills
Box 4. G1 Basic Skills Modules Idea Face validity: Does the test
measure what it was intended to measure? Construct validity: Can
the test teach the desired operative skills? Reliability:
Inter-rater and Intra-rater