Model-based Auto-Segmentation of Knee Bonesand Cartilage in MRI Data

Heiko Seim, Dagmar Kainmueller, Hans Lamecker, Matthias Bindernagel,Jana Malinowski, and Stefan Zachow

Zuse Institute Berlin (ZIB), Medical Planning Group,Takustr. 7, 14195 Berlin, Germany

{seim,kainmueller,lamecker,bindernagel,malinowski,zachow}@zib.de

http://www.zib.de/visual/medical

Abstract. We present a method for fully automatic segmentation ofthe bones and cartilages of the human knee from MRI data. Based onstatistical shape models and graph-based optimization, first the femoraland tibial bone surfaces are reconstructed. Starting from the bone sur-faces the cartilages are segmented simultaneously with a multi objecttechnique using prior knowledge on the variation of cartilage thickness.We validate our method on 40 clinical MRI datasets acquired before kneereplacement.

Keywords: Segmentation, Statistical shape model, Magnetic resonanceimaging, Knee, Cartilage, Femur, Tibia, Joint

1 Introduction

Osteoarthritis (OA) is a disabling disease affecting more than one third of thepopulation over the age of 60. Monitoring the progression of OA or the responseto structure modifying drugs requires exact quantification of the knee cartilageby measuring e.g. the bone interface, the cartilage thickness or the cartilage vol-ume [4]. Manual delineation for detailed assessment of knee bones and cartilagemorphology, as it is often performed in clinical routine, is generally irreproducibleand labor intensive with reconstruction times up to several hours [10].

Due to the increasing availability of MRI scanners in clinical routine the de-mand for automatic segmentation of knee bone and cartilage tissue is growing.Though semi-automatic approaches remain necessary and useful in their ownright, we focus here on fully automatic methods. To this end, an approach forsegmentation of articular cartilage based on supervised learning was presentedin [1], where an evaluation on 46 MRI datasets with no or mild OA symptomsresulted in an average Dice similarity coefficient (DSC) of 0.80, a sensitivity of90.0% and specificity of 99.8%. A similar approach in combination with an elas-tic registration scheme was presented in [11] but only evaluated on a single MRIdataset. Fitting of a probabilistic atlas to MRI data exploiting linear program-ming was presented in [3] to segment the cartilage of the patella, achieving a DSC

215

Fig. 1. Segmentation system (see text for explanations)

of 0.84, a sensitivity of 94.1%, a specificitiy of 99.9% and an average surface dis-tance of 0.49 mm evaluated on 28 MRI datasets from unspecified sources. In [2],statistical shape models (SSMs) are used to extract the bone-cartilage interface(BCI). 20 MRI datasets from healthy subjects were segmented with DSCs of0.96, 0.96 and 0.89 for the femur, the tibia and the patella, respectively, andan average point-to-surface error of 0.16mm on the BCI. A multi-object graph-based approach to segment the bones and the cartilage of the knee with minormanual interaction (≈ 30 sec) was proposed in [12], where an evaluation on 16MRI datasets showed average surface positioning errors of 0.2 to 0.3 mm for thebones and 0.5 to 0.8 mm for the cartilage.

Comparing the performance of these different approaches is difficult, becauseevaluation results strongly depend on varying properties and origins of the em-ployed image data (e.g. MR sequences, varying types of pathologies) as well asdifferent evaluation metrics. Very recently, a standardized benchmark for au-tomatic knee segmentation systems has been introduced [5]. In this paper, wepropose a new model-based segmentation approach and validate it using thedata and evaluation metrics put forth in [5].

2 Auto-Segmentation System for Knee MRI

The general outline of our automatic segmentation system is shown in Fig. 1. Itconsists of two major parts: (1) An SSM-based reconstruction method for bonesurfaces [9] applied to knee bones, and (2) a method for simultaneous segmen-tation of adjacent structures via shared displacement directions [6] applied tothe tibial/femoral cartilage. In the following, due to space limitations we onlydescribe the modifications with which we adapted these two methods to theapplication-specific situation of bone and cartilage in knee MRI. For more detailwe refer the reader to [9] and [6].

216

2.1 Statistical Shape Models

The statistical shape models of proximal tibia and distal femur used in this workwere generated from 60 MRI datasets provided by the MICCAI 2010 WorkshopMedical Image Analysis for the Clinic - A Grand Challenge [5]. For each of thesedatasets an expert segmentation was available for the bones and the associatedcartilage. We fitted existing SSMs of the femur and the tibia to the gold-standardsegmentation using the method in [9], thus extrapolating the femoral and tibialshafts not included in the field-of-view of the MRI datasets. We used the resultingreconstructed surfaces to generate a new SSM for each bone covering the rangeof bone lengths occurring in the 60 MRI datasets.

2.2 Bone Segmentation

Side Selection and Positioning. We compute the initial transformation pa-rameters (rigid transformation + uniform scaling) using a low resolution surfacetemplate of the left distal femur (500 vertices, mean shape of the training data)via the Generalized Hough Transform (GHT) [7]. To detect the correct side ofthe body, i.e. left or right knee, this method is repeated with a shape templatemirrored on the mid-sagittal plane. Accepting the transformation with the max-imum vote count after both detection cycles yields the side of the body and theinitial transformation T0 to position the SSMs of the bones in the data.

Parameter Initialization and Model Adaptation. The model adaptationprocedure described in the following is performed independently for the femurand for the tibia. Given the transformation T0 of the SSM, the parameters ofthe image feature based adaptation strategy are initialized. To this end, we rateimage features along surface normals with a new cost function that is particularlysuitable for MRI: Locations on the surface normals that exhibit intensities withinan intensity interval [t1, t2] and directional derivatives along the normal thatexceed a gradient threshold gmin are equipped with low costs gmin/g, where gdenotes the respective directional derivative. All other locations are equippedwith high costs (some constant larger than all low costs).

The parameters t1, t2 and gmin are determined automatically via histogram-analyses: The bone intensity threshold t1 is computed by analyzing the histogramof intensities inside the initialized SSM (see Fig. 2). It is set such that 5% ofthe voxels that contribute to the histogram have higher intensities. The upperthreshold t2 is set to the maximum intensity that occurs within the initializedSSM. To determine the gradient threshold gmin, a histogram of the gradientmagnitude of the entire image is computed, and gmin is set to the value abovewhich 15% of the gradient magnitudes lie.

In addition we check for a special case to handle large, bright artifacts (seeFig. 3 left): A weighted sum of 5 gaussians

∑i

wi · gi with gi(x) =1

σi

√2π

exp

[−0.5

(x − μi

σi

)2]

217

Fig. 2. Left: Schematic MRI histogram of the whole image and within the spatiallyinitialized femur and tibia SSMs. Right: Parameter initialization: The brightest 5% ofvoxels within the initialized SSM define the intensity window [t1, t2]. Special case: Letga be the highest of the 5 gaussians fitted via EM. We look for the highest brightergaussian gb with non-overlapping standard deviation. If such a gb exists, t1 is set tothe intersection x0 of the two gaussians.

is fitted to the histogram of intensities via the Expectation Maximization (EM)algorithm. We dermine the highest gaussian ga (i.e. a = arg maxi

wi

σi

√2π

), andthen we look for the highest gaussian gb for which holds μa + σa < μb − σb.If there is such a gb and a value x0 where ga and gb have equal height, i.e.wa · ga(x0) = wb · gb(x0), and furthermore μa < x0 < μb and x0 < t1 holds, thent1 is set to x0 and t2 is set to x0 + 0.25σb.

SSM adaptation followed by a graph-cuts based fine-grain adaptation is per-formed as in [9] employing the cost function described above. The results aresurfaces of the femoral and tibial knee bone.

2.3 Cartilage Segmentation

The surfaces of femur and tibia are coupled with shared intensity profiles [6] anddeformed with multi-object graph cuts [8] employing a new cost function that wespecifically designed for cartilage in MRI: From the training data, we learn theminimum, mean and maximum cartilage thickness in normal direction per vertexof the bone surface mesh. Costs are generally high (“infinite”) at locations thatlie below the learnt minimum and above the learnt maximum cartilage thicknessper vertex in surface normal direction. Locations closer to the mean thicknessare slightly preferred over those further apart (but still within the minimum-maximum range). Locations within the minimum-maximum range are equippedwith low costs −gminCart/g if the directional derivative in normal direction, g,satisfies g < −gminCart, and the location exhibits an absolute intensity within anintensity interval determined by histogram analysis. The respective histogram iscomputed from voxels in-between the bone surface and a second surface “bone +maximum cartilage thickness along normals”. A weighted sum of five gaussians

218

is fitted, and the mean ± standard deviation of the highest gaussian serves asthe respective intensity interval. The gradient threshold gminCart is set to 10(determined heuristically). Note, this can be seen as an “inverted” version of thecost function described above for bone segmentation.

To obtain the final segmentation, the resulting bone and cartilage surfacesare converted into binary voxel representations with the same extension andvoxel size as the original MRI image.

3 Results and Discussion

For evaluation, 40 additional clinical MRI datasets acquired before knee replace-ment were made available by the workshop organizers. A detailed evaluation ispresented in Table 1. For tibial and femoral bones the average (AvgD) and theroots mean square (RMS) surface distances were computed. Cartilage segmenta-tion is quantified by volumetric overlap (VOE) and volumetric difference (VD)measures. For all four structures a score was computed indicating the agreementwith human inter-observer variability. Reaching the inter-observer variability re-sults in 75 points, while obtaining an exact match to one destinguished manualsegmentation results in 100 points. An error twice as high as the human rater’sgets 50 points, 3x as high gets 25 points and if 4x as high or more receives 0points (no negative points). All points are averaged for each image, which resultsin a total score per image. Details on the evaluation procedure will be publishedin an overview article of the Grand Challenge workshop [5].

The average performance of our auto-segmentation system for knee bonesand associated cartilage was 54.4± 8.8 points. Note that this value correspondsto the status of the software at the time the first results were uploaded to theworkshop organizers. New results including bugfixes and other improvements willbe available by the time of the workshop and will be published on the workshopwebsite [5]. The automatic segmentation of one dataset took approximately 6minutes on a standard desktop PC (Core 2 Duo CPU, 3.00GHz, 16GB RAM).It is implemented as an extension to the ZIBAmira software (amira.zib.de).

With no failures the GHT proved to be a reliable method for the initializa-tion of the the bone segmentation (side selection and positioning). The bonesegmentation achieves scores that indicate an error larger than that obtainedby human experts. This may be due to relatively large mismatches of the SSMat the proximal and distal end of the MRI data due to missing or weak imagefeatures related to intensity inhomogeneities stemming from the MRI sequence(see Fig. 3). A strong artifact of unknown source (see Fig. 3) lead to the worstsegmentation result for the tibia. The scores for cartilage segmentation are basedon different error measures (volumetric) and are generally better, presumablydue to a higher inter-observer variability.

For the reasons already discussed at the end of the introduction, it is diffi-cult to compare our results to previous approaches. However, from now on thestandardized evaluation system of [5] will allow for an unbiased comparison ofour method with other systems.

219

(a) (b) (c) (d)

Fig. 3. Problem cases: Artifacts introducing ”false”image features (a), intensity inho-mogeneities typical for MRI (b), similar intensities for cartilage and neighboring softtissues (d bottom) and invisible bone-soft-tissue interfaces (d top).

In the future we will investigate how the intensity inhomogeneities in theMRI data can be dealt with. We are already examining the use of a combinedactive shape model of the femur and tibia to increase the robustness and accu-racy of the bone segmentation. Currently, our results indicate that some manualpost-processing is still required to draw level with human-rater performance.However, we are confident that with our proposed framework and the suggestedimprovements we will be able to increase the number of cases where manualpostprocessing will become obsolete.

Acknowledgments. H. Seim and H. Lamecker are supported by the EU-FP7 Project

MXL (FP7-ICT-2009.5.2-248693). H. Lamecker is also funded by the DFG Research

Center Matheon. D. Kainmueller is funded by the DFG Collaborative Research Center

SFB760.

References

1. Folkesson, J., Dam, E., Olsen, O.F., Pettersen, P., Christiansen, C.: Automaticsegmentation of the articular cartilage in knee MRI using a hierarchical multi-classclassification scheme. In: International Conference on Medical Image Computingand Computer-Assisted Intervention MICCAI. vol. 8, pp. 327–334 (2005)

2. Fripp, J., Crozier, S., Warfield, S.K., Ourselin, S.: Automatic segmentation of thebone and extraction of the bone-cartilage interface from magnetic resonance imagesof the knee. Phys Med Biol 52(6), 1617–1631 (2007)

3. Glocker, B., Komodakis, N., Paragios, N., Glaser, C., Tziritas, G., Navab, N.:Primal/Dual Linear Programming and Statistical Atlases for Cartilage Segmenta-tion. In: Ayache, N., Ourselin, S., Maeder, A. (eds.) International Conference onMedical Image Computing and Computer-Assisted Intervention MICCAI. LectureNotes in Computer Science, vol. 4792, pp. 536–543. Springer Berlin Heidelberg,Berlin, Heidelberg (Jan 2007)

4. Graichen, H., von Eisenhart-Rothe, R., Vogl, T., Englmeier, K.H., Eckstein, F.:Quantitative assessment of cartilage status in osteoarthritis by quantitative mag-netic resonance imaging: technical validation for use in analysis of cartilage volume

220

and further morphologic parameters. Arthritis and rheumatism 50(3), 811–6 (Mar2004)

5. Heimann, T., Morrison, B., Warfield, S., Styner, M., Niethammer, M.: MedicalImage Analysis for the Clinic - A Grand Challenge: Segmentation of Knee Images2010 (2010), http://www.ski10.org/

6. Kainmueller, D., Lamecker, H., Zachow, S.: Multi-object segmentation with cou-pled deformable models. Annals of the British Machine Vision Association (BMVA)5, 1–10 (2009)

7. Khoshelham, K.: Extending generalized hough transform to detect 3d objects inlaser range data. In: ISPRS Workshop on Laser Scanning 2007 and SilviLaser 2007.pp. 206–210 (2007)

8. Li, K., Millington, S., Wu, X., Chen, D.Z., Sonka, M.: Simultaneous segmentationof multiple closed surfaces using optimal graph searching. Inf Process Med Imaging19, 406–417 (2005)

9. Seim, H., Kainmueller, D., Heller, M., Lamecker, H., Zachow, S., Hege, H.C.: Au-tomatic Segmentation of the Pelvic Bones from CT Data Based on a StatisticalShape Model. In: Eurographics Workshop on Visual Computing for Biomedicine(VCBM). pp. 93–100. Delft, Netherlands (2008)

10. Shim, H., Chang, S., Tao, C., Wang, J.H., Kwoh, C.K., Bae, K.T.: Knee cartilage:efficient and reproducible segmentation on high-spatial-resolution MR images withthe semiautomated graph-cut algorithm method. Radiology 251(2), 548–56 (May2009)

11. Warfield, S., Kaus, M., Jolesz, F.A., Kikinis, R.: Adaptive, template moderated,spatially varying statistical classification. Medical Image Analysis 4(1), 43–55 (Mar2000)

12. Yin, Y., Zhang, X., Anderson, D.D., Brown, T.D., Hofwegen, C.V., Sonka, M.:Simultaneous segmentation of the bone and cartilage surfaces of a knee joint in3D. SPIE (2009)

221

Img Femur bone Tibia bone Fem. cartilage Tibial cartilage TotalAvgD RMSD Scr AvgD RMSD Scr VOE VD Scr VOE VD Scr Score[mm] [mm] [mm] [mm] [%] [%] [%] [%]

1 0.99 1.70 45 0.82 1.19 48 19.1 8.0 79 21.2 -6.5 81 63.22 0.87 1.25 55 0.91 1.31 43 33.7 -13.6 64 22.0 -17.2 62 55.93 0.83 1.25 57 0.66 0.95 58 28.3 23.2 49 31.2 -7.8 75 59.74 0.83 1.17 58 0.97 1.42 39 46.4 24.9 39 33.6 -32.0 38 43.45 1.20 1.76 38 0.78 1.08 52 62.4 -25.1 33 33.6 10.2 70 48.26 0.97 1.45 50 0.92 1.41 41 36.6 27.6 38 19.5 -13.3 70 49.47 0.99 1.66 45 0.73 1.13 52 30.5 3.8 82 26.3 20.6 54 58.58 1.91 2.58 8 0.83 1.15 49 80.4 -59.4 21 40.2 -16.8 56 33.39 1.06 1.54 46 0.83 1.17 49 33.3 -20.4 52 35.0 -27.3 39 46.3

10 1.01 1.42 49 0.72 1.04 55 32.0 -22.1 49 28.0 -26.3 44 49.111 1.07 1.83 40 0.71 1.08 54 23.6 14.1 67 24.0 -5.8 81 60.612 0.68 1.07 64 0.63 0.94 60 27.9 10.9 71 28.4 12.1 68 65.613 1.23 1.85 36 0.76 1.17 51 45.1 32.7 34 28.0 24.6 46 41.614 0.83 1.28 56 0.66 1.00 58 37.2 25.5 42 21.5 2.9 87 60.615 0.73 1.09 62 0.66 1.09 56 34.4 21.7 49 28.1 33.4 40 51.716 1.15 1.64 42 0.67 1.00 57 21.5 8.9 76 34.2 -14.7 62 59.217 1.14 1.63 42 1.13 1.47 32 34.3 28.5 37 41.6 -35.8 35 36.618 0.95 1.59 48 0.67 1.01 57 24.1 9.3 75 21.1 15.7 65 61.219 0.82 1.23 57 0.86 1.29 45 26.9 22.8 50 18.6 -2.4 89 60.320 0.93 1.37 52 1.15 1.72 26 31.6 7.5 75 49.0 4.3 75 57.021 0.95 1.39 51 0.70 1.03 56 28.7 -9.2 73 13.0 2.5 91 67.722 1.12 1.51 44 0.68 1.05 56 58.8 -19.6 44 30.3 -26.7 42 46.523 1.04 1.61 45 1.04 1.48 35 38.2 9.6 69 33.4 11.4 68 54.224 0.88 1.38 53 1.28 1.63 24 30.6 13.6 65 30.4 -9.6 72 53.625 1.12 1.65 42 0.85 1.24 46 20.7 6.4 81 27.2 24.3 47 54.226 0.96 1.38 51 0.67 0.97 58 22.9 -0.3 91 26.5 -9.5 74 68.427 1.05 1.57 45 0.87 1.28 45 38.2 -9.2 70 25.5 -19.2 57 54.228 1.28 2.13 30 0.90 1.24 45 38.0 8.6 71 22.5 -11.8 71 54.129 1.21 1.66 39 0.79 1.13 51 22.8 -10.6 73 21.3 -1.7 89 63.030 0.95 1.57 48 0.99 1.34 40 24.8 9.8 74 51.9 -0.4 80 60.431 1.12 1.62 43 0.78 1.14 51 51.8 12.6 59 26.5 -8.0 76 57.132 0.85 1.48 52 0.83 1.18 48 36.8 26.6 40 20.0 -1.9 89 57.433 0.60 0.97 68 0.85 1.26 46 25.4 14.2 66 30.0 -9.4 73 62.934 1.13 1.72 41 0.61 0.92 61 40.1 45.1 35 32.0 6.8 76 53.335 1.00 1.52 48 0.94 1.46 39 28.7 9.4 73 45.2 -33.2 33 48.236 0.91 1.39 52 1.11 1.51 32 18.4 7.7 80 25.8 -13.9 66 57.537 1.19 1.83 37 0.88 1.37 43 32.1 18.0 56 38.1 39.9 36 43.138 0.81 1.25 57 0.56 0.86 64 26.6 9.1 74 17.7 5.1 85 69.939 1.18 1.78 38 1.47 2.43 1 45.3 21.6 45 38.6 15.5 59 35.940 1.20 1.77 38 0.84 1.30 45 20.5 14.3 67 25.8 15.8 63 53.4

Avg 1.02 1.54 47 0.84 1.24 47 34.0 7.7 60 29.2 -2.7 65 54.4±0.22 ±0.30 ±10 ±0.19 ±0.28 ±12 ±12.7 ±19.2 ±17 ±8.6 ±18.2 ±17 ±8.8

Table 1. Results of the comparison metrics and scores for all 40 test cases. AvgD andRMSD are the average and RMS surface distance, respectively, VOE is the volumetricoverlap error and VD indicates the volumetric difference.

222

Fig. 4. Different views on selected test cases 8 (top), 16 (center), and 31 (bottom). Theoutline of the reference segmentation is displayed in green, the outline of the automaticmethod described in this paper in red.

223