Simon K. Warfield, Ph.D. Associate Professor of Radiology Harvard Medical School

Computational Radiology LaboratoryHarvard Medical Schoolwww.crl.med.harvard.edu

Children’s Hospital Department of Radiology Boston Massachusetts

A Survey of Validation Techniques for Image Segmentation and Registration, with a focus on the STAPLE algorithm

Simon K. Warfield, Ph.D.

Associate Professor of Radiology

Harvard Medical School

Computational Radiology Laboratory. Slide 2

Outline

• Validation of image segmentation– Overview of approaches– STAPLE

• Validation of image registration

• STAPLE algorithm available as open source software from:– http://www.nitrc.org/projects/staple– http://crl.med.harvard.edu/


Segmentation• Goal: identify or label structures

present in the image.• Many methods:

– Interactive or manual delineation,– Supervised approaches with user

initialization,– Alignment with a template,– Statistical pattern recognition.

• Applications:– Quantitative measurement of

volume, shape or location of structures,

– Provides boundary for visualization by surface rendering.

Newborn MRI Segmentation.


Validation of Image Segmentation• Spectrum of accuracy versus realism in

reference standard.• Digital phantoms.

– Ground truth known accurately.– Not so realistic.

• Acquisitions and careful segmentation.– Some uncertainty in ground truth.– More realistic.

• Autopsy/histopathology.– Addresses pathology directly; resolution.

• Clinical data ?– Hard to know ground truth.– Most realistic model.


Validation of Image Segmentation

• Comparison to digital and physical phantoms:– Excellent for testing the anatomy, noise and

artifact which is modeled.– Typically lacks range of normal or

pathological variability encountered in practice.

MRI of brain phantom from Styner et al. IEEE TMI 2000


Comparison To Higher Resolution

MRI Photograph MRI

Provided by Peter Ratiu and Florin Talos.


Comparison To Higher Resolution

Photograph MRI Photograph Microscopy

Provided by Peter Ratiu and Florin Talos.


Comparison to Autopsy Data

• Neonate gyrification index– Ratio of length of cortical boundary to length

of smooth contour enclosing brain surface


Staging

Stage 3 Stage 5

Stage 4 Stage 6

Stage 3: at 28 w GAshallow indentations of inf. frontal and sup. Temp. gyrus(1 infant at 30.6 w GA, normal range: 28.6 ± 0.5 w GA)

Stage 4: at 30 w GA2 indentations divide front. lobe into 3 areas, sup. temp.gyrus clearly detectable(3 infants, 30.6 w GA ± 0.4 w, normal range: 29.9 ± 0.3 w GA)

Stage 5: at 32 w GAfrontal lobe clearly divided into three parts: sup., middle and inf. Frontal gyrus(4 infants, 32.1 w GA ± 0.7 w,normal range: 31.6 ± 0.6 w GA)

Stage 6: at 34 w GAtemporal lobe clearly divided into

3 parts: sup., middle and inf. temporal gyrus(8 infants, 33.5 w GA ± 0.5 wnormal range: 33.8 ± 0.7 w GA)

“Assessment of cortical gyrus and sulcus formation using MR images in normalfetuses”, Abe S. et al., Prenatal Diagn 2003


Neonate GI: MRI Vs AutopsyGyrification Index versus age in days

0

0.5

1

1.5

2

2.5

3

200 220 240 260 280 300 320 340

Post-conceptional age in days

GI

MRI Scan 2 MRI Scan 1 Armstrong


GI Increase Is Proportional to Change in Age.

'change in GI' versus 'days of growth before final scan'

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

50 55 60 65 70 75 80 85 90

time interval between scans in days

ch

an

ge

of

GI

Change of Total Brain GI Linear (Change of Total Brain GI)


GI Versus Qualitative StagingStaging versus GI

1

1.2

1.4

1.6

1.8

2

2.2

2.4

3 4 5 6 7 8 9

Staging Grade

To

tal

Bra

in G

I

MRI scan 1 MRI scan 2


Neonate Gyrification

GI : interactive versus automatic segmentation.

y = 1.2241x + 0.4443

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

-1 0 1 2 3 4 5

GI - hand segmentation

GI -

au

tom

atic

seg

men

tati

on

Linear (line of equality)


Validation of Image Segmentation• Comparison to expert performance; to other

algorithms.• Why compare to experts ?

– Experts are currently doing the segmentation tasks that we seek algorithms for.

– Surgical planning.– Neuroscience research.

• What is the appropriate measure for such comparisons ?


Measures of Expert Performance• Repeated measures of volume

– Intra-class correlation coefficient

• Spatial overlap– Jaccard: Area of intersection over union.– Dice: increased weight of intersection.– Vote counting: majority rule, etc.

• Boundary measures– Hausdorff, 95% Hausdorff.

• Bland-Altman methodology:– Requires a reference standard.

• Measures of correct classification rate:– Sensitivity, specificity ( Pr(D=1|T=1), Pr(D=0|T=0) )– Positive predictive value and negative predictive value

(posterior probabilities Pr(T=1|D=1), Pr(T=0|D=0) )


Validation of Image Segmentation

• STAPLE (Simultaneous Truth and Performance Level Estimation):– An algorithm for estimating performance

and ground truth from a collection of independent segmentations.


STAPLE papers– Image segmentation with labels:

• Warfield, Zou, Wells ISBI 2002• Warfield, Zou, Wells MICCAI 2002.• Warfield, Zou, Wells, IEEE TMI 2004.• Commowick and Warfield IPMI 2009

– Image segmentation with boundaries:• Warfield, Zou, Wells MICCAI 2006.• Warfield, Zou, Wells PTRSA 2008.

– Diffusion data and vector fields:• Commowick and Warfield IEEE TMI 2009


STAPLE: Estimation Problem

• Complete data density:• Binary ground truth Ti for each voxel i.

• Expert j makes segmentation decisions Dij.

• Expert performance characterized by sensitivity p and specificity q.

– We observe expert decisions D. If we knew ground truth T, we could construct maximum likelihood estimates for each expert’s sensitivity (true positive fraction) and specificity (true negative fraction):

)|( qp,TD,f

)|,(lnmaxargˆ,ˆ qp,TDqpqp,

f


Expectation-Maximization• Since we don’t know ground truth T, treat T as a

random variable, and solve for the expert performance parameters that maximize:

• Parameter values θj=[pj qj]T that maximize the conditional expectation of the log-likelihood function are found by iterating two steps:– E-step: Estimate probability of hidden ground truth T given a

previous estimate of the expert quality parameters, and take expectation.

– M-step: Estimate expert performance parameters by comparing D to the current estimate of T.

€

Q(θ |θ (t−1)) = E ln f (D,T |θ) | D,θ (t−1)[ ]


Probability Estimate of True Labels

Wsik f (T

is | D

i, k )

f (T

is) f (D

ij| T

is, k )

j

f (Ti s ) f (D

ij| T

i s , k )

j

s

Estimate probability of tissue class in reference standard:


Binary Input: True Segmentation

, ,

,

( 1| , )

( | 1 ) ( 1)

( | , ) ( )

k k ki i

k kij i j j i

j

k kT ij i j j ii

j

k

k k

W f T

f D T p q f T

f D T p q f T

αα β

≡ =

= ==

∑

=+

∏∏

iD ,p q

: 1 : 0

: 0 : 1

( 1) (1 )

( 0) (1 )

( 1) : prior probability true label at voxel i is 1.

: conditional probability that true label is 1.

ij ij

ij ij

k k ki j jj D j D

k k ki j jj D j D

i

ki

f T p p

f T q q

f T

W

α

β= =

= =

= = −

= = −

=

∏ ∏∏ ∏


Expert Performance Estimate

: 11

: 1 : 0

: 01

: 1 : 0

(1 )

(1 ) (1 )

ij

ij ij

ij

ij ij

kii Dk

j k ki ii D i D

kii Dk

j k ki ii D i D

Wp

W W

Wq

W W

p (sensitivity, true positive fraction) : ratio of expert identified class 1 to total class 1 in the image.

q (specificity, true negative fraction) : ratio of expert

identified class 0 to total class 0 in the image.

:1 ij

ksi

i D skjs s k

sii

W

W


Newborn MRI Segmentation


Newborn MRI Segmentation

Summary of segmentation quality (posterior probability Pr(T=t|D=t) ) for each tissue type for repeated manual segmentations.

Indicates limits of accuracy of interactive segmentation.


Expert and Student Segmentations

Test image Expert consensus Student 1

Student 2 Student 3


Phantom Segmentation

Image Expert Students Voting STAPLE

Image Expertsegmentation

Studentsegmentations


STAPLE Summary• Key advantages of STAPLE:

– Estimates ``true’’ segmentation.– Assesses expert performance.

• Principled mechanism which enables:– Comparison of different experts.– Comparison of algorithm and experts.

• Extensions for the future:– Prior distribution or extended models for

expert performance characteristics.– Estimate bounds on parameters.


Image registration

• A metric: measures similarity of images given an estimate of the transformation.

• Best metric depends on nature of the images.

• Alignment quality ultimately possible depends on model of transformation.

• The transformation is identified by solving an optimization problem.– Seek the transform parameters that

maximize the metric of image similarity


Validation of Registration

• Compare transformations– Take some images, apply a transformation

to them.– Estimate the transform using registration– How well does the estimated transformation

match the applied transform?

• Check alignment of key image features– Fiducial alignment– Spatial overlap

• Segment structures, assess overlap after alignment.


Intraoperative Nonrigid Registration

• Fast: it should not take more than 1 min to make the registration.

• Robust: the registration should work with poor quality image, artifacts, tumor...

• Physics based: we are not only concerned in the intensity matching, but also interested in recovering the physical (mechanical) deformation of the brain.

• Accurate: neuro-surgery needs a precise knowledge of the position of the structures.

• Archip et al. NeuroImage 2007


Block Matching Algorithm

Divide a global optimization problem in many simple local ones

Highly parallelizable, as blocks can be matched independently.

Similarity measure: coefficient of correlation ]1:0[∈


Block Matching Algorithm

Displacementestimates are noisy.


Patient-specific Biomechanical Model

Pre-operativeimage

Automatic brain segmentation

Brain finite element model (linear elastic)


Registration Validation

• Landmark matching assessment in six cases

• Parallel version runs in 35 seconds on a 10 dual 2GHz PC cluster – 7x7x7 block size– 11x11x25 window– 1x1x1 step– 50 000 blocks– 10 000 tetrahedra

Registration Error Evaluation Using Landmarks Correspondences

0

0,5

1

1,5

2

2,5

3

0 5 10 15

Displacement

Measured Error

Patient 1

Patient 2Patient 3

Patient 4Patient 5

Patient 6

• 60 landmarks:– Average error = 0.75mm– Maximum error = 2.5mm– Data voxel size 0.8x0.8x2.5 mm3


Registration Validation

• 11 prospective consecutive cases,

• Alignment computed during the surgery.

• Estimate of the registration accuracy – 95% Hausdorff distance of the edges of the registered preoperative MRI and the intraoperative MRI.


Automatic selection of fiducials

(1)Non-rigid alignment of preoperative MPRAGE.

Contours extracted from (1) with the Canny edge

detector

(2) Intraoperative whole brain SPGR at 0.5T

Contours extracted from (2) with the Canny edge

detector

95% Hausdorff metric

computed


Alignment improvement

Tumor position Tumor pathology

Non-rigid registration – preop to intraop scans(95% Hausdorff distance)

Max Displacement

measured(mm)

Rigid registration

accuracy – preop to intraop

(mm)

Non-Rigid registration accuracy – preop to intraop(mm)

Ratio Rigid/Non-

Rigid

Case 1 right posterior frontal

oligoastrocytoma Grade II

10.68 5.95 1.90 3.13

Case 2 left posterior temporal

glioblastoma Grade IV 21.03 10.71 2.90 3.69

Case 3 left medial temporal

glioblastoma Grade IV 15.27 7.65 1.70 4.50

Case 4 left temporal anaplastic oligoastrocytoma Grade III

10.00 6.80 0.85 8.00

Case 5 right frontal oligoastrocytoma Grade II

9.87 5.10 1.27 4.01

Case 6 left frontal anaplastic astrocytoma Grade III

17.48 10.20 3.57 2.85

Case 7 right medial temporal

anaplastic astrocytoma Grade III

19.96 9.35 2.55 3.66

Case 8 right frontal oligoastrocytoma Grade II

17.44 8.33 1.19 7.00

Case 9 right frontotemporal

oligoastrocytoma Grade II

15.08 7.14 1.87 3.81

Case 10 right occipital anaplastic oligodendroglioma Grade III

9.48 5.95 1.44 4.13

Case 11 left frontotemporal

oligodendroglioma Grade II 10.74 4.76 0.85 5.60

AVG 14.27 7.44 1.82 4.58


Visualization of aligned data

• Matched preoperative fMRI and DT-MRI aligned with intraoperative MRI.

Tensor alignment: Ruiz et al. 2000


Conclusion

• Validation strategies for registration:– Comparison of transformations.– Fiducials

• Manual, automatic.

– Overlap statistics – as for segmentation.

• Validation strategies for segmentation:– Digital and physical phantoms.– Comparison to domain experts.– STAPLE.


Acknowledgements

• Neil Weisenfeld.• Andrea Mewes.• Richard Robertson.• Joseph Madsen.• Karol Miller.• Michael Scott.

This study was supported by:R01 RR021885, R01 EB008015, R01 GM074068

Collaborators• William Wells.• Kelly H. Zou.• Frank Duffy.• Arne Hans.• Olivier Commowick.• Alexandra Golby.• Vicente Grau.

Documents

Simon K. Warfield, Ph.D. Associate Professor of Radiology Harvard Medical School