View
13
Download
0
Category
Preview:
Citation preview
BSc VR Surgical Simulation – Software Slide 1
Development of Simulation Tools –Software
Vincent Luboz
Department of Biosurgery and Surgical Technology
Imperial College London
Contents
• Virtual Reality
• VR Surgical Simulation
• Modelling
• Collision detection and collision response
• Performance metrics
• Examples
• Summary and Conclusions
BSc VR Surgical Simulation – Software Slide 2
• Virtual Reality
• VR Surgical Simulation
• Modelling
• Collision detection and collision response
• Performance metrics
• Examples
• Summary and Conclusions
Contents
BSc VR Surgical Simulation – Software Slide 3
Classic VR Examples?
BSc VR Surgical Simulation – Software Slide 4
• Flight Simulators
(here from Airbus company)
Classic VR Examples
BSc VR Surgical Simulation – Software Slide 5
• Video Games
(here from Medal of Honor 3 and 4)
Classic VR Examples
BSc VR Surgical Simulation – Software Slide 6
What is Virtual Reality?
“Virtual reality is a way for humans to visualize, manipulate and interact with computers and complex data.”
The user interacts with the virtual world and can directly manipulate objects in it.
The technology attempts to immerse the user into a computer generated world representing the reality.
BSc VR Surgical Simulation – Software Slide 7
What is Immersion?
• Immersion in an environment (real or virtual) is related to the quantity and quality of sensory data from that environment.
• Immersion in a virtual environment can be gauged by the extent to which the computer shuts out sensations from the real world and accommodates different sensory modalities (vision, audio, haptic).
BSc VR Surgical Simulation – Software Slide 8
Examples
Desktop VR SystemWindow on world
Immersa Desk
Immersive systems (HMD) Immersive systems (CAVE)
Telepresence
Mixed reality
BSc VR Surgical Simulation – Software Slide 9
Architecture of a VR system
Graphical Display(easy)
Tactile Feedback(very difficult)
MechanicalInterface
(moderately easy)
Rendering Engine(easy)
Computer Models
Geometric (anatomical)(easy)
Physical (biomechanical)(very difficult)
User
Haptics
Audio interfaces can also be included
BSc VR Surgical Simulation – Software Slide 10
• Virtual Reality
• VR Surgical Simulation
• Modelling
• Haptics
• Collision detection and collision response
• Examples
• Summary and Conclusions
• Hands-on session
Contents
BSc VR Surgical Simulation – Software Slide 11
Elements of a VR Surgical Simulation?
Graphical Display(easy)
Tactile Feedback(very difficult)
MechanicalInterface
(moderately easy)
Rendering Engine(easy)
Computer Models
Geometric (anatomical)(easy)
Physical (biomechanical)(very difficult)
User
Haptics
Audio interfaces can also be included
BSc VR Surgical Simulation – Software Slide 12
Elements of a VR Surgical Simulation
• Computer Model (geometrical or physical)
Liver
VesselsHips
BSc VR Surgical Simulation – Software Slide 13
Elements of a VR Surgical Simulation
• Computer Model (geometrical or physical)
• Rendering Engine (interactions, collision detection and collision response)
Biopsy Catheterization
BSc VR Surgical Simulation Slide 15
Elements of a VR Surgical Simulation
• Computer Model (geometrical or physical)
• Rendering Engine (interactions, collision detection and collision response)
• Interface (haptics)
VSP for catheterization
Phantom for biopsy
Elements of a VR Surgical Simulation
• Computer Model (geometrical or physical)
• Rendering Engine (interactions, collision detection and collision response)
• Interface (haptics)
• Performance metrics (measuring the outcome of a virtual surgery and the performance of the trainee)
BSc VR Surgical Simulation – Software Slide 16
Simulating Minimally Invasive Procedures
• Involves interaction of several systems
Hardest to achieve accuracy.
• A topic of great interest due to the potential commercial value.
• Example: Gall Bladder (producing bile) and Surrounding organs procedures (Laparoscopic Cholecystectomy)
BSc VR Surgical Simulation – Software Slide 17
Simbionix Lap Mentor
Lapsim Laparoscopic Simulator
Sample Commercial Simulators
BSc VR Surgical Simulation – Software Slide 18
Immersion Medical Sigmoidoscopy SimulatorBSc VR Surgical Simulation – Software Slide 19
Patient Specific Simulation
Original CTABSc VR Surgical Simulation – Software Slide 20
BSc VR Surgical Simulation – Software Slide 21
Patient Specific Simulation
Segmentation of the vessels
BSc VR Surgical Simulation – Software Slide 22
Patient Specific Simulation
Brach
R. SC
R. CC L. CC
L. SC
AA
RARA
CA
SMA
IA IA
EIAEIA
Fem Fem
•Arch of the aorta (AA),•Brachiocephalic (brach) artery,•Right and left common carotid (RCC and LCC),•Right and left subclavian artery (RSC and LCC),•Renal arteries (RA),•Celiac artery (CA),•Superior mesenteric artery (SMA),•Internal (IA) and external (EA) iliac arteries,•And femoral artery (fem).
BSc VR Surgical Simulation – Software Slide 23
Patient Specific Simulation
Non-pathological Aorta Aortic Aneurysm Aortic Dissection
BSc VR Surgical Simulation – Software Slide 24
Patient Specific Simulation
Model of liver vasculature from fluoroscopy
BSc VR Surgical Simulation – Software Slide 25
Simulation Complexity
No deformation - easy
- Bronchoscopy
- Colonoscopy
- Upper GI endoscopy
- ERCP
- Laryngoscopy
- Laparascopic Cholesectomy
Large and Complex Deformations
-
+
BSc VR Surgical Simulation – Software Slide 26
• Virtual Reality
• VR Surgical Simulation
• Modelling
• Collision detection and collision response
• Performance metrics
• Examples
• Summary and Conclusions
Contents
BSc VR Surgical Simulation – Software Slide 27
Geometric Modelling
• Geometric modelling can be used to reproduce anatomy
• Geometric models can be derived from scan data (CT, MR, US…) and be patient specific
• Applications: walk through the peritoneum, lungs, colon…
BSc VR Surgical Simulation – Software Slide 28
Extracting Geometric Models
Segmentation
Mesh Generation
ReconstructionCT Data Set
BSc VR Surgical Simulation – Software Slide 29
Creating Walk through
Mesh Model
Render
Texture
BSc VR Surgical Simulation – Software Slide 30
Walk Through
Virtual Colonoscopy – University of Hamburg
BSc VR Surgical Simulation – Software Slide 31
Spherical model of the Colon
• Easy to create effects such as - spasm,
- inflation of the colon
• Anatomy can be constrained to be representative of a real colon.
BSc VR Surgical Simulation – Software Slide 32
Limitation of Geometric Model
• Triangulated models are easy to obtain, but are difficult to deform. (large number of triangles)
• They are suitable for rigid structures (bones, larynx, lung, trachea etc)
• They are less efficient for deformable organs (colon, gall bladder, skin…) and instruments
BSc VR Surgical Simulation – Software Slide 33
Modelling Problem - Colonoscopy
TurnView Direction
Turn
View Direction
Turn Rigid Collar
View Direct ion
BSc VR Surgical Simulation – Software Slide 34
In a real colonoscopy we might find
If we twist does the loop tighten or the image rotate?
BSc VR Surgical Simulation – Software Slide 35
There are many possibilities
BSc VR Surgical Simulation – Software Slide 36
Biomechanical Modelling
• One solution to the previous problem is bio-mechanical modelling - however this is fraught with problems.
- What happens when you push the colon wall?
- What dynamic properties does the endoscope have?
- What frictional forces are acting?
BSc VR Surgical Simulation – Software Slide 37
Behavioural Models
• For many endoscopic procedures we cannot see the deformation that is caused.
• Thus simulators can get away by modelling only the behaviour of the instrument.
BSc VR Surgical Simulation – Software Slide 38
Can we simulate real deformations?
Answer – Almost there but not quite yet!
Certainly not in real time with a high degree of accuracy
BSc VR Surgical Simulation – Software Slide 39
Modelling deformations
• The most accurate modelling tool uses the finite element method.
• In the finite element method a surface or volume is divided into small parts called elements.
Element
Face
Node
BSc VR Surgical Simulation – Software Slide 40
Typical 3D Elements
BSc VR Surgical Simulation – Software Slide 41
Finite Element Model of the Liver
• Geometry extracted from scan data.• Boundary conditions defined initially.• Material properties specified
physically.
BSc VR Surgical Simulation – Software Slide 42
Problems
• Real time finite element solutions are possible for small numbers of isotropic linear elastic elements.
• Muscle tissue is not isotropic or linear elastic, and large numbers of elements are required.
BSc VR Surgical Simulation – Software Slide 43
Possible solutions
• Precomputation and encoding of a large number of accurate solutions using statistical shape modelling.
• Restriction of the number of degrees of freedom (only appropriate for certain techniques).
• Use of hierarchical approach to solve the local deformation and propagate accordingly.
BSc VR Surgical Simulation – Software Slide 44
Basic Interactions
Pressing and dragging
Grabbing and pulling
BSc VR Surgical Simulation – Software Slide 45
• Virtual Reality
• VR Surgical Simulation
• Modelling
• Collision detection and collision response
• Performance metrics
• Examples
• Summary and Conclusions
Contents
Collision Detection/Response?
BSc VR Surgical Simulation – Software Slide 46
Collision Detection/Response?
BSc VR Surgical Simulation – Software Slide 47
BSc VR Surgical Simulation – Software Slide 48
Object Database
Geometry
Material
Position Orientation
Force Torque
Contact Information
Collision Detection
Collision Response
Collision Detection/Response
BSc VR Surgical Simulation – Software Slide 49
• The algorithms combine the information to obtain the positions in Cartesian inside the virtual environment.
• The algorithm uses position information to find collisions.
• The algorithm reports the resulting degree of penetration or indentation.
Collision Detection
BSc VR Surgical Simulation – Software Slide 50
• The algorithm computes interaction forces between objects involved in a collision.
• The algorithm sends interaction forces to the control algorithms.
Collision Response
BSc VR Surgical Simulation – Software Slide 51
• Virtual Reality
• VR Surgical Simulation
• Modelling
• Collision detection and collision response
• Performance metrics
• Examples
• Summary and Conclusions
Contents
• Evaluate the performance of the user.
• Give feedback on the task.
• Point to areas that the user needs to improve.
• Derived from a task analysis.
Performance metrics
BSc VR Surgical Simulation – Software Slide 52
• Common metrics:- Outcome of the procedure.
- Time spent on the procedure.
- Accuracy of the task.
- Respect of the scenario.
- Comparison with an ideal procedure.
- …
Performance metrics
BSc VR Surgical Simulation – Software Slide 53
BSc VR Surgical Simulation – Software Slide 54
• Virtual Reality
• VR Surgical Simulation
• Modelling
• Collision detection and collision response
• Performance metrics
• Examples
• Summary and Conclusions
Contents
BSc VR Surgical Simulation – Software Slide 55
VR Simulator for Liver Biopsy
Raw image dataLabelled Segmentation
Surface meshTetrahedral mesh
DICOM DATA
SIMULATOR
InteractiveSegmentation Framework
Mesh GenerationPipeline
Deformable Modelling
BSc VR Surgical Simulation – Software Slide 56
BSc VR Surgical Simulation – Software Slide 57
VR Simulator for Endovascular operation
BSc VR Surgical Simulation – Software Slide 58
• Virtual Reality
• VR Surgical Simulation
• Modelling
• Collision detection and collision response
• Performance metrics
• Examples
• Summary and Conclusions
Contents
BSc VR Surgical Simulation – Software Slide 59
Summary
We have seen:
• The principles of VR surgical simulator software design
• The concepts of modelling with regard to surgical simulation programs
• How the mechanism of collision is handled by VR simulators
• The basic knowledge and examples of software platforms used for VR simulator
BSc VR Surgical Simulation – Software Slide 60
• VR is a powerful technique for visualisation, interaction and planning.
• Surgical simulation systems for training and assessment are increasingly relevant.
• Still improving:- In accuracy (e.g.: real organs elasticity)- In computing time (e.g.: using computer graphic card)- In realism (e.g.: quality of the texture and the rendering)- …
• The haptic rendering allows adding sense of touch to the visualisation immersion
Conclusion
BSc VR Surgical Simulation – Software Slide 61
Development of Simulation Tools –Software
Vincent Luboz
Department of Biosurgery and Surgical Technology
Imperial College London
Recommended