3D Multiscale Residual U-Net Architecture for BrainLesion Segmentation

Youngwon Choi, Yongchan Kwon*, Hanbyul Lee, Myunghee Cho Paik,Joong-Ho Won

Department of Statistics, Seoul National University, Seoul, Korea

ykwon0407@snu.ac.kr

Seoul National University ISLES 2016 1 / 27

Contents

1 Introduction

2 Literature reviews

3 Task 1

4 References

Seoul National University ISLES 2016 2 / 27

Introduction

Introduction

Seoul National University ISLES 2016 3 / 27

Introduction ISLES 2016

Ischemic stroke lesion segmentation challenge

Seoul National University ISLES 2016 4 / 27

Introduction ISLES 2016

Ischemic stroke lesion segmentation challenge

Seoul National University ISLES 2016 5 / 27

Introduction ISLES 2016

Ischemic stroke lesion segmentation challenge

Seoul National University ISLES 2016 6 / 27

Introduction ISLES 2016

Ischemic stroke lesion segmentation challenge

Seoul National University ISLES 2016 7 / 27

Introduction ISLES 2016

Ischemic stroke lesion segmentation challenge

Compute the DC, ASSD, and HD values for each case

Establish each teams rank for DC, ASSD, and HD separately for eachdataset

Compute the mean rank over all three evaluation measures/case toobtain the teams rank for the case

Compute the mean over all case-specific ranks to obtain the teamsfinal rank

Seoul National University ISLES 2016 8 / 27

Introduction ISLES 2016

MRI data

ISLES 2015 ISLES 2016Images type 3D-gray MR scans 3D-gray MR scansDimension

200 × 200 × 200 200 × 200 × 20

Number of cases 28 30

ModalitiesT1w TFE/TSE, FLAIR PWI (4D images), Tmax, ADC

T2w TES, DWI TTP, rBF, rBV, MTTDisease type sub-acute hyper-acuteProportion of

0.001% ∼ 1.91% 0.006% ∼ 3.10%ischemic lesion

Table: Training dataset information. The dimensions are arranged in order ofwidth, height, and depth.

Seoul National University ISLES 2016 9 / 27

Introduction ISLES 2016

ISLES 2016

Seven modalities are given. (+)

Binary classification. (+)

Extremely imbalanced data. (−)

Only 30 subjects are given. (−)

Predict brain lesion after 90 days. (−)

3-dimensional inputs. (−)

Seoul National University ISLES 2016 10 / 27

Literature reviews

Literature reviews

Seoul National University ISLES 2016 11 / 27

Literature reviews U-Net

U-Net, Convolutional Networks for Biomedical ImageSegmentation

U-Net has the unique U-shaped architecture consisting of contractingand expanding path.Contracting path for capturing context and expanding path forlocalizing interest feature.

Seoul National University ISLES 2016 12 / 27

Literature reviews Multi-Scale 3D Convolutional Neural Networks

Multi-Scale 3D Convolutional Neural Networks for LesionSegmentation in Brain MRI

This method exploits 3-dimensional multiscale inputs for voxel-wiseclassification.Different scale patches look both small and big context of images byusing multiscale inputs.

Seoul National University ISLES 2016 13 / 27

Literature reviews Brain Tumor Segmentation with Deep Neural Networks

Brain Tumor Segmentation with Deep Neural Networks

Two-phase training is used to solve imbalanced problem.Oversampling the data such that labels are equiprobable at firsttraining phase.Training the network taking account for the imbalanced nature atsecond phase.

Seoul National University ISLES 2016 14 / 27

Literature reviews Brain Tumor Segmentation with Deep Neural Networks

Brain Tumor Segmentation with Deep NeuralNetworks(cont’d)

Seoul National University ISLES 2016 15 / 27

Task 1

Task 1

Seoul National University ISLES 2016 16 / 27

Task 1

Pipeline overview

Seoul National University ISLES 2016 17 / 27

Task 1 Pre-processing

Pre-processing

We resized all modalities to the same dimension (width, height,depth) = (256, 256, 32) and standardized.

From the fixed size images, we extracted (32, 32, 4) 3D patches withmultiscale.

Sagittal axis reflection was used for data augmentation.

Seoul National University ISLES 2016 18 / 27

Task 1 Pre-processing

Pre-processing

Seoul National University ISLES 2016 19 / 27

Task 1 Main architecture

3D Multiscale U-Net augmented by Resnet

Seoul National University ISLES 2016 20 / 27

Task 1 Main architecture

Convolutional and Residual block

Seoul National University ISLES 2016 21 / 27

Task 1 Main architecture

Two-phased training

In the first phase training, we oversampled patches near the brainlesion while sampled patches randomly in the second phase.

Seoul National University ISLES 2016 22 / 27

Task 1 Main architecture

Prediction: Ensemble

We used 5-fold cross-validation and obtained 0.42 mean Dicecoefficient from a single model.

We considered two ensemble methods of 9 variants of the proposedmodel, the majority voting and the unanimous voting.

Methods Cases ASSD Dice HD Precision Recall

Majority 29/306.00 0.44 45.64 0.47 0.58

(4.43) (0.22) (23.19) (0.28) (0.25)

Unanimous 29/305.29 0.42 39.93 0.54 0.47

(4.40) (0.22) (21.29) (0.28) (0.26)

Table: Mean and standard deviation of ASSD, Dice, HD, precision, and recallmeasures for two ensemble methods. Standard deviations are in the parenthesis.

Seoul National University ISLES 2016 23 / 27

Task 1 Main architecture

Prediction: Visualization

Figure: (Left) ground truth (Middle) majority vote model prediction (Right)unanimity model prediction for ‘training 16’ case

Seoul National University ISLES 2016 24 / 27

References

References

Seoul National University ISLES 2016 25 / 27

References

Key references

Model architecture and two phase training

Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net:Convolutional Networks for Biomedical Image Segmentation.arXiv:1505.04597 [cs.CV].

Kamnitsas, K., Ledig, C., Newcombe, V.F.J., Simpson, J.P., Kane,A.D., Menon, D.K., Rueckert, D., and Glocker. B. (2016). EfficientMulti-Scale 3D CNN with Fully Connected CRF for Accurate BrainLesion Segmentation. arXiv:1603.05959 [cs.CV].

Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A.,Bengio, Y., Pal, C., Jodoin, P.-M., and Larochelle, H. (2016). BrainTumor Segmentation with Deep Neural Networks. arXiv:1505.03540[cs.CV].

Seoul National University ISLES 2016 26 / 27

References

Key references

Initialization and residual block

Shah, A., Kadam, E., Shah, H., Shinde,S., and Shingade, S. (2016).Deep Residual Networks with Exponential Linear Unit.arXiv:1604.04112 [cs.CV].

He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving Deep intoRectifiers: Surpassing Human-Level Performance on ImageNetClassification. arXiv:1502.01852 [cs.CV].

Seoul National University ISLES 2016 27 / 27