Upload
dasia-golly
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
Anatomical Sketch Understanding
Peter HaddawyComputer Science & Information Management
Asian Institute of Technology
Anatomical Sketching
• Sketching is ubiquitous in medicine– Patient records– Communication with patients– Consultation– Medical education, particularly PBL
COMET: Collaborative Medical PBL Tutor
Suebnukarn & Haddawy, Clinical-Reasoning Skill Acquisition through Intelligent Group Tutoring, IJCAI-05.
Scenario
Unconsciousness due to car accident
It’s certainly possible to have a contusion in
the frontal lobe, but what about damage to
the brain stem?
Discussion boardRecognize what anatomical structure has been drawn and from what perspective
Parietal view of the brain
Identify anatomical parts of the structure, even if not explicitly drawn
Frontal lobe of the brain
Understand annotations on the sketch
Arrow pointing from the label “contusion” to the frontal lobe
Effectively use the sketch as a medium of communication in a dialog
Recognize what anatomical structure has been drawn and from what perspective
Identify anatomical parts of the structure, even if not explicitly drawn
Why sketch understanding?
• The informalness of sketches is important:– Conveys that the ideas are still rough
and thus invites collaboration and modification.
– Clean, precise-looking diagrams create the impression of more precision than was intended and lead to a feeling of commitment to the sketch as originally drawn.
• Sketching in medical PBL gives students practice in recalling anatomical structure.
The challenge of anatomical sketches
• Previous work (e.g. ASSIST, SILK) starts by recognizing primitive components– circles, lines, rectangles
• This approach works for assemblies of components but not for anatomical sketches
• Anatomical sketches are complex structures that may be drawn with greatly varying degrees of detail.
Sketch Recognition
• By convention, 2-D depictions of anatomy are drawn from one of eight standard views.
• Five external views, corresponding to the sides of a cube:– Anterior– Posterior– Superior– Inferior– Lateral (2 sides)
• Three internal views, corresponding to the cutting planes:– Sagittal– Coronal– Axial
Assumptions
• Sketch drawn from one of the eight standard views
• Sketch contains a single anatomical structure with no extraneous parts
• Sketch is complete: no major parts left out
UNAS Sketch Recognition Algorithm
• Sketch recognition becomes a classification problem: – Identify the anatomical structure and
view• Heart: anterior view
• Naïve Bayes classifier– Features: similarity between sketch
and each of a set of templates– Similarity computed using Shape
Context [Belongie, et al 2002]
Recognition Process
AnatomicalStructure & View
…Template 1 Template m
Naïve Bayes Classifier
Lung External
User Sketch
Template 1 Template m…
Shape ContextMatching
Data Collection
• Collected sample sketches from 3rd – 6th year medical students
• 70 sketches of 6 anatomical structures, 2-3 views per structure: 1050 sketches
Constructing the Classifier
• 70% of the data for training– 49 sketches per view
• Treat each sketch as a potential template
• Calculate 15 conditional Gaussians for each sketch– Distribution of shape context match over
the other sketches conditional on the class (anatomical structure & view)
• Use the Wrapper approach [Kohavi, 1997] to select a small, accurate subset of templates– 32 templates
Evaluation
• 30% of the data for testing– 21 sketches per view
• Asked ten 3rd - 6th year medical students to classify each sketch
• Compared: Student accuracy vs UNAS accuracy
Evaluation: Accuracy
Brain Heart Lung
Parietal Sagittal Basal Anterior Posterior Interior External Internal
Students 92.9 92.9 97.6 87.1 80.5 93.8 93.3 91.0
UNAS 86.0 67.0 57.0 81.0 52.0 57.0 76.0 48.0
Skull Stomach Kidney Overall Accuracy
(%)Anterior Parietal Basal External Internal External Internal
Students 99.0 98.6 97.6 95.7 93.3 97.1 96.7 93.81
UNAS 76.0 81.0 52.0 76.0 67.0 52.0 67.0 66.30
Baseline random classification accuracy = 6.7%
• Between internal & external viewsBetween internal & external views
• Between similarly shaped structuresBetween similarly shaped structures
UNAS Common Errors
Kidney External
Kidney Internal
Skull Basal Brain Basal
Sketch Segmentation
• Task: Identify explicit and implicit anatomical parts of the sketch
• Algorithm:– Find the best correspondence
between the sketch and a pre-labeled canonical template (Shape Context)
– Transfer labeled points to the sketch– Connect the points to form the parts
Transformation
• Transform the template to fit the sketch– Thin-plate splines
• Then project labels from transformed template onto sketch
• Curve reconstruction (TSP [Giesen, 1999])
Evaluation
• Selected five qualitatively different sketches from two views of each of four structures– Brain : Basal, Parietal– Lung : External, Internal– Heart : Anterior, Posterior– Skull : Anterior, Parietal
• Asked a physician to segment the sketches
• Compared: Physician segmentations vs UNAS segmentations
Evaluation
• Segmentations of the system and those of the expert are very similar.
• Segmentations of the system have no missing or overlapping regions
sketch system physician sketch system physician
Evaluation: Disagreements
• The system is less tolerant of inaccurate proportions than the physician
sketch system physician
Conclusions
• First application of sketch-based interfaces to intelligent tutoring
• First in a medical domain other than image retrieval
• Can recognize 15 views of 6 anatomical structures– Accuracy 66% vs baseline random
classification accuracy of 6.7%
• System’s segmentations qualitatively similar to those of the physician
Ongoing & Future Work
• Improve recognition accuracy and speed• Improve segmentation quality and speed• Integrate with annotation understanding
component• Incorporate into COMET tutoring system
(using UMLS)
• Sketches with partial and multiple anatomical structures
Demos gladly given on request
Shape Context
• Represent a shape as a set points sampled from the shape’s contours
• Each point represented as a coarse histogram of the points surrounding it.
Shape Context
• Uniformly randomly sample points from the sketch and template.
• Compute the shape context descriptor for each point.
• Normalize by the mean squared distance between points to make scale invariant.
• The similarity between two shape context descriptors is just the χ2 statistic.
• Find the least cost match between all points in the sketch and template.– Weighted bipartite graph matching– Hungarian method: O(N3)