Prof. Stephen O'Leary, University of Melbourne - Growing Virtual Reality in the Higher...

Virtual reality, the net and surgical training

Stephen O’Leary

Professor of Otolaryngology The University of Melbourne

Outline

•  VR surgical simulation –  for surgical training – What it is, how it works – Status and prospects

•  Networking and remote surgery

Lessons from Aerospace

• Failure is catastrophic

• Resources are expensive

• Complex tasks

• ZERO tolerance of major error

Lesson from Space Exploration

• No “dressed rehearsal”

• Train for all possibilities “No surprises”

• Reduce “cognitive load”

Courtesy of NASA

Cognitive Load

VR Simulation for Surgery

•  Freedom to fail – Practice until minimal standards are met

•  Controlled training – Curriculum can be standardised

•  Repeatability & Availability

Surgery for Cochlear Implantation

Courtesy of Cochlear

Temporal Bone - Anatomy

•  At risk: – Facial nerve function – Sense of taste – Great Vessels –  Integrity of Inner Ear:

hearing and balance – The dura

VR Simulation for Ear Surgery

•  Scarcity of temporal bones

•  To maximise real drilling experience –  In the temporal bone laboratory –  In the operating theatre

•  To provide real-time feedback in training

Classical cortical mastoidectomy

•  The skills required for this task: – Surgical Anatomy – Surgical Planning (strategy) – Technical drilling skills (psychomotor)

Surgical Planning

•  Surgical “Landmarks”

•  Finding surgical landmarks –  In the correct order – Using the correct techniques

Courtesy Thomas Somers

Building 3D models

Imaging Data

Building 3D models

Manually segment anatomical structures

Mauro Maijorca, Brian Pyman, Yi Zhao, S. O’Leary

Building 3D models

Generate 3D models Assign colours

Building 3D models

Physical properties Sigmoid: Bleeding

The Prototype System

Force-Feedback (Haptics)

Mentoring across a Network

Simulation and Training

Texts Observation

Temporal Bone Laboratory

Operating Theatre

Simulation and Training

Texts Observation

Temporal Bone Laboratory

Operating Theatre

Virtual Surgery

Validation of VR simulation • Transfer of learning • Sensitivity to levels of expertise • Automated feedback to trainee

Transfer of Learning

Participants: • Novice surgeons

Procedure: • Cortical mastoidectomy

Outcome Measures: • Surgical Anatomy • Surgical Planning • Technical drilling skills

Oral assessment temporal bone

Temporal Bone Laboratory

Virtual Surgery

Oral assessment temporal bone

Repeat task until performance is error free

VR training and recognition of anatomy (on human temporal bone)

VR simulation for assessment

Trainees Experts

VirtualSurgery Cortical Mastoid

Correlate observer & automated metrics

Sensitivity to levels of expertise

Participants: • Novices (9) • Registrars (6) • Experts (12)

• Standardised pre-reading • Performed a canal-wall down mastoidectomy • Metrics: Force, speed, stroke

Time to completion Total  Time

0.00

10.00

20.00

30.00

40.00

50.00

60.00

Novice Registrar Expert

Time  (m

in)

Total  Number  of  s trokes

0

10000

20000

30000

40000

50000

Novice Registrar E xpertTotal  N

o.  of  stroke

s  ap

plied

Total  jumps

0.00

500.00

1000.00

1500.00

2000.00

2500.00

3000.00

Novice Registrar ExpertNo.  of  jum

ps  during  the  sim

ulation

Total  voxels  eroded

0.E +00

5.E +05

1.E +06

2.E +06

2.E +06

3.E +06

Novice Registrar E xpert

Total  V

oxel  erode

d

Average  force  <1cm  from  critical  structures

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Sigmoid Dura Facial

Average  Force  (N)

Novice

Registrar

Expert

Self-directed surgical curriculum

•  e.g. exposing the dura

Instructional video

•  Force, distance

Practice with immediate feedback

•  Quality assurance

Comparison with “ideal” end result

•  Improve visual recognition

Operative Photos and videos

Aim/research question

Virtual Reality Simulation

Traditional Methods

Study Method: Randomized, Blinded, Control Trial

Didactic Teaching (20)

VR simulation Group (10)

Cortical mastoidectomy

Traditional Group (10)

Cortical mastoidectomy

Assessment •  1 hour time limit •  “standardized” temporal bones •  Video taped •  3 Blinded assessors •  Multifaceted Assessment tool

focusing on 4 areas of performance

Overall Performance

0

10

20

30

40

50

60

70

VR Traditional

ICC = 0.93 P-value <0.001

End product analysis

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

Dura Sigmoid EAC LSCC Incus

VR

Trad

ICC = 0.78 P-value <0.001

Injury size

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Dura Sigmoid EAC LSCC Incus

VR

Traditional

ICC = 0.88 P-value =0.01

!

!

Formative feedback

•  Timely feedback is key to learning •  Aim is to provide feedback “expert-like”

feedback •  This should facilitate independent

learning on the simulator

Efficacy Real-time feedback

Wijewickrema et al, unpublished data 24 medical students, performing cortical mastoidectomy randomised to receiving feedback or not

Feedback Method

Random Forest algorithm of Bailey , Zhou et al,

Networking and remote surgery

•  The Speed of Light (and networks) •  Ping and Lag

– Real-time interaction impossible when latency exceeds reaction time

– For training this is not critical – But for surgery…….

Working with lag

Courtesy of NASA

Working with lag

COURTESY OF ST. JOSEPH'S HOSPITAL

Conceding to lag: Tele-presence surgery

Courtesy of NASA

Tele-presence surgery

Assisting surgeons in Remote communities

Tele-present Surgery in space Courtesy NASA

Courtesy NASA

Courtesy NASA

Courtesy NASA

On Mars, this won’t work……

Conclusions

•  VR surgery research: – Can discriminate between levels of experience – Transfer of learning to temporal bone dissection – Self-directed learning works – Frontier: automated feedback

•  Networking and telemedicine – Remote-control surgery limited by lag – Tele-presence surgery realistic alternative – Simulation ideal for establishing protocols

The people

•  Gregor Kennedy – Educational psychologist •  Ioanna Ioannou, Sudanthi Widewickrema

- computer engineers •  Yi Chen Zhao, Yun Zhou- PhD’s •  Ioanna Ioannou, Brian Pyman, Richard Hall,

Kumiko Yukawa, Mauro Maijorca, Peter Harris, Liz Sonenburg (Melb. Uni.)

•  M. Hutchins, C. Gunn, A. Krompholz, D. Stevenson (CSIRO)

The organisations

•  Melbourne University •  CSIRO •  Medic Vision

(held license from University/ CSIRO 2006-2010) •  Royal Victorian Eye and Ear Hospital •  Royal Prince Alfred Hospital •  Royal Australasian College of Surgeons •  Medtronic Xomed (grant for early validation)

The funding bodies

US  Air  Force  

Courtesy The Age