Sketch-Based Interactive Segmentation and

Segmentation Editing for Oncological Therapy Monitoring

Frank HeckelMarch 17, 2015

BVM-Award 2015– PhD Thesis –

2 / 22

Medical Background

Change in tumor size is an important criterion for assessing the success of a chemotherapy RECIST1 1.1: Sum of maximum diameters of target lesions

Relative change

Volume is a more accurate measure Many tumors grow/shrink irregularly in 3D Requires appropriate segmentation

Oncological Therapy Response Monitoring

1 RECIST: Response Evaluation Criteria In Solid Tumors

Complete Response

Partial Response

Stable Disease

Progressive Disease

Disappearance

< -30% -30% … 20%

> 20%

3 / 22

The Segmentation Problem

Ultimate Goal: Automatic segmentation Reproducible results with no effort for the user Solutions for specific purposes Might fail (low contrast, noise, biological variability) Unsolved or insufficient for many real-world problems

Solutions: Manual segmentation Interactive tools Automatic segmentation + manual correction

Drawbacks: Higher effort Lower reproducibility

4 / 22

Interactive Segmentation

Based on common 2D user interaction: drawing contours Segmentation as an object reconstruction problem

Energy-minimizing surface reconstruction from a point cloud based on RBFs

3D surface based on contours from a few slices in arbitrary orientations

Variational Interpolation

𝑓 (�⃗� 𝑖 )=𝑃 (�⃗� 𝑖 )+∑𝑗=1

𝑘

𝑤 𝑗𝜙 (�⃗�𝑖− �⃗� 𝑗 )=h𝑖

5 / 22

Interactive Segmentation

Computation time optimization Shape preserving constraint reduction Parallelization

Robustness improvement Approximation instead of interpolation for resolving

contradictions Detection and consideration of self-intersection points

Main Challenges

6 / 22

Interactive Segmentation

Computation time: Speedup ≈80

Evaluation: Data: 15 liver metastases, 1 liver Participants: 2 experienced radiology technicians

Results

Before1

After2

Metastasis

57,53 s

0.7 s

Liver 629,1 s

8.3 s

1 CLAPACK, 1 thread, no reduction 2 MKL, 4/8 threads, reduction by ≈80%

Manual RBF-based Interpolation

Metastasis

111 s

21 contours

64 s 7 contours Overlap: 75%

Liver 1272 s

106 contours

665 s

22 contours

Overlap: 94%

7 / 22

Segmentation Editing

Segmentation Algorithm

Start

Semi-automatic

AutomaticSegmentation

ResultSatisfying?

Initial Algorithm allows

modification?

SegmentationEditing Algorithmno no

Stop

yes yes

Segmentation Algorithm

InteractiveSegmentation

ResultSatisfying? Stop

yes

no

Most existing methods are low-level and unintuitive in 3D High-level correction has not received much attention in

research

8 / 22

Segmentation EditingSketch-Based Editing in 2D

add

remove

add + remove

replace

9 / 22

Segmentation Editing

Estimate 3D size of the error by the „diameter“ of the edited region in

The Correction Depth

𝑪𝒔𝒖

𝑪𝒔𝒆𝒔

10 / 22

Segmentation Editing

Sample user contour into reference points Move reference points to next slice using a block matching Connect seed points using a shortest-path algorithm

Image-Based 3D Extrapolation

11 / 22

Segmentation Editing

Utilizes the RBF-based interpolation approach Reconstruct the new segmentation with contours in the

edited slice and a start / end slice given by the correction depth

Restrict the new segmentation to the edited region

Image-Independent 3D Extrapolation

12 / 22

Evaluation of Editing Tools

131 representative tumor segmentations in CT (lung nodules, liver metastases, lymph nodes)

5 radiologists with different level of experience

Editing rating score:

Qualitative Evaluation

𝑟 edit=1𝑁

¿

13 / 22

Evaluation of Editing ToolsQuantitative Evaluation

Analyze quality over time Editing quality score:

14 / 22

Evaluation of Editing Tools

Problem: High effort and bad reproducibility of user studies Idea: Replace user by a simulation Benefits:

Objective and reproducible validation Objective comparison Improved regression testing Better parameter tuning

Simulation-Based Evaluation

IntermediateSegmentation

Target Segmentation

Segmentation Editing

Satisfying?

User

Validationno

yes

Stop

Start

Control flow

Data flow

User Input

Previous Inputs

IntermediateSegmentation

Reference Segmentation

Segmentation Editing

Satisfying?

Simulation

Validationno

yes

Stop

Start

Control flow

Data flow

User Input

Previous Inputs

15 / 22

Evaluation of Editing Tools

Step 1: Find most probably corrected 3D error Step 2: Select slice and view where the error is most

probably corrected Step 3: Generate user-input for sketching Step 4: Apply editing algorithm

Simulation-Based Evaluation

16 / 22

Evaluation of Editing ToolsSimulation-Based Evaluation

17 / 22

Partial Volume Correction

Smoothing effect caused by limited spatial resolution (of CT)

Ill-defined border between tumor and healthy tissue, making segmentation an ill-defined problem

Could cause significant differences in size measurements

The Partial Volume Effect

28.4 ml(-27.5%)

39.2 ml 56.8 ml(+44.9%)

18 / 22

Partial Volume Correction

Spatial subdivision into spherical sectors to cover different tissues

Define reference tissue values inside and outside of the object ( and to) per sector

For each sector : compute the weight w of each partial volume voxel

Method

1.0

0.0

0.5

0.75

0.25

𝑤 (𝑉 )=𝑡𝑜 𝑠−𝑣

𝑡𝑜 𝑠− 𝑡𝑖 𝑠

,𝑉∈𝑃 𝑖𝑠∪𝑃𝑜𝑠

𝑉𝑜𝑙𝐿=∑𝑉 ∈𝐿

𝑤 (𝑉 )𝑉𝑜𝑙𝑉71.1 ml70.8 ml

19 / 22

Partial Volume CorrectionSoftware Phantom Results

20 / 22

Partial Volume CorrectionHardware Phantom Results

21 / 22

Partial Volume CorrectionMulti-Reader Data Results

22 / 22

Summary

Contributions: General image-independent interactive segmentation method Efficient and intuitive segmentation editing tools +

methodologies for their evaluation Fast algorithm for compensation of partial volume effects

Future Work: Improve algorithms for irregular and large objects Combine image-based and image-independent editing Make editing simulation more realistic HCI aspects in editing 4D and multi-label segmentations Establish volumetric measurements in clinical routine

Acknowledgement

Thanks to all colleagues at (Fraunhofer) MEVIS, particularly Dr. Jan Moltz, Lars Bornemann, Dr. Hans Meine, Dr. Stefan Braunewell, Dr. Markus Lang, Michael Schwier, Dr. Volker Dicken, Dr. Benjamin Geisler, Olaf Konrad, Wolf Spindler and Prof. Horst Hahn. Special thanks to Dr. Christian Tietjen, Dr. Grzegorz Soza, Andreas Wimmer, Dr. Ola Friman, Prof. Bernhard Preim, Prof. Andreas Nüchter, all clinical partners and the Visual Computing in Biology and Medicine community.An finally, my wife and my children!

Thank you!frank.heckel@mevis.fraunhofer.de

Bei Herausforderungen geht es nicht ums Gewinnen, sondern darum, herauszufinden, was für ein Mensch man ist.