Deep Active Lesion SegmentationAli Hatamizadeh1*, Assaf Hoogi2*, Debleena Sengupta1, Wuyue Lu1, Brian Wilcox2, Daniel Rubin2†, and Demetri Terzopoulos1†

1 Computer Science Department, University of California, Los Angeles, CA, USA 2 Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

∗Co-primary authorship, †Co-senior authorship

MotivationLesion segmentation is an important problem in computer assisted diagnosis that remainschallenging due to the prevalence of low contrast, irregular boundaries that are unamenable toshape priors. We introduce Deep Active Lesion Segmentation (DALS), which effectively leveragesthe strengths of CNNs and ACMs to create a fully automated segmentation framework.

Figure 1: Segmentation comparison

Overview Methodology

DALS: Best of Both Worlds

Results Discussion

• An input image is fed into theencoder-decoder to localizethe lesion and produce asegmentation probability mapYprob that specifies theprobability that any point (x,y)lies in the interior of thelesion.

• The Transformer converts Yprob

to a Signed Distance Map,YSDM, that initializes the level-set ACM and estimates the

parameter functions λ1(x,y)and λ2(x,y) in the energyfunctional (1).

• During training, Yprob and theground truth map Ygt are fedinto a Dice loss function andthe error is back-propagatedaccordingly.

• During inference, a forwardpass through the frameworkproduces a final SDM, which isconverted back into aprobability map, producingthe final segmentation mapYout.

How Does it Perform?

Figure 3: Comparison of the output segmentation of our DALS (red) against the U-Net [9] (yellow) and ground truth (green) segmentations

MLS DatasetDataset captures variation in:• Organ types (brain, liver, lung)• Size• Modality (MR, CT)• Lesion shapesDataset will be publicly released.

Figure 2: The DALS Framework: Multiscale Encoder-decoder + Level Set ACM

Table 1: MLS dataset statistics. GC: Global Contrast; GH: Global Heterogeneity.

• Boundary Delineation: The DALS segmentation contours conform well to the irregular shapes of the lesionboundaries (Fig. 3), since the learned parameter maps, λ1(x,y) and λ2(x,y), provide flexibility toaccommodate boundary irregularities. DALS performs well for different image characteristics, including lowcontrast and heterogeneous lesions.

• Parameter functions

ConclusionThe DALS framework includes a custom encoder-decoder that feeds a level-set ACM with per-pixelparameter functions. We evaluated our framework in the challenging task of lesion segmentation with a newdataset, MLS, which includes a variety of images of lesions of various sizes and textures in different organsacquired through multiple imaging modalities. Our results affirm the effectiveness our DALS framework.

Multiscale Encoder-Decoder

Level-Set ACM With Parameter Functions

• Encoder consists of dense blocks that concatenate feature maps from layers 0 to l-1before being fed to successive dense blocks.

• Last dense block in the encoder is fed into a custom multiscale dilation block. This, alongwith dense connectivity, assists in capturing local and global context for highly accuratelesion localization

𝑶𝒓𝒈𝒂𝒏 𝑴𝒐𝒅𝒂𝒍𝒊𝒕𝒚 # 𝑺𝒂𝒎𝒑𝒍𝒆𝒔 𝑴𝒆𝒂𝒏𝑮𝑪 𝑽𝒂𝒓𝑮𝑪 𝑴𝒆𝒂𝒏𝑮𝑯 𝑽𝒂𝒓𝑮𝑯 L𝒆𝒔𝒊𝒐𝒏 𝑹𝒂𝒅𝒊𝒖𝒔(𝒑𝒊𝒙𝒆𝒍𝒔)

Brain MRI 369 0.56 0.029 0.901 0.003 17.42 ± 9.516

Lung CT 87 0.315 0.002 0.901 0.004 15.15 ± 5.777

Liver CT 112 0.825 0.072 0.838 0.002 20.48 ± 10.37

Liver MRI 164 0.448 0.041 0.891 0.003 5.459 ± 2.027

The DALS frameworkbenefits from:• A novel multiscale

encoder-decoderCNN that learns aninitializationprobability map &parameter maps forthe ACM.

• An improved level-set ACM formulationwith a per-pixel-parameterized energyfunctional

Dataset Brain MR

𝑀𝑜𝑑𝑒𝑙 𝐷𝑖𝑐𝑒 𝐶𝐼 𝐻𝑎𝑢𝑠𝑑𝑜𝑟𝑓𝑓 𝐶𝐼 𝐵𝑜𝑢𝑛𝑑𝐹

U-Net 0.776 ± 0.214 0.090 2.988 ± 1.238 0.521 0.826

CNN Backbone 0.824 ± 0.193 0.078 2.755 ± 1.216 0.490 0.892

Manual Level-set 0.796 ± 0.095 0.038 2.927 ± 0.992 0.400 0.841

DALS 0.888 ± 0.076 0.030 2.322 ± 0.824 0.332 0.944

Lung CT

𝐷𝑖𝑐𝑒 𝐶𝐼 𝐻𝑎𝑢𝑠𝑑𝑜𝑟𝑓𝑓 𝐶𝐼 𝐵𝑜𝑢𝑛𝑑𝐹

0.817 ± 0.098 0.080 2.289 ± 0.650 0.530 0.898

0.822 ± 0.115 0.094 2.254 ± 0.762 0.622 0.900

0.789 ± 0.078 0.064 3.270 ± 0.553 0.451 0.879

0.869 ± 0.113 0.092 2.095 ± 0.623 0.508 0.937

Dataset Liver MR

𝑀𝑜𝑑𝑒𝑙 𝐷𝑖𝑐𝑒 𝐶𝐼 𝐻𝑎𝑢𝑠𝑑𝑜𝑟𝑓𝑓 𝐶𝐼 𝐵𝑜𝑢𝑛𝑑𝐹

U-Net 0.769 ± 0.162 0.093 1.645 ± 0.598 0.343 0.920

CNN Backbone 0.805 ± 0.193 0.110 1.347 ± 0.671 0.385 0.939

Manual Level-set 0.739± 0.102 0.056 2.227 ± 0.576 0.317 0.954

DALS 0.894 ± 0.036 0.036 1.298 ± 0.434 0.239 0.987

Liver CT

𝐷𝑖𝑐𝑒 𝐶𝐼 𝐻𝑎𝑢𝑠𝑑𝑜𝑟𝑓𝑓 𝐶𝐼 𝐵𝑜𝑢𝑛𝑑𝐹

0.698 ± 0.149 0.133 4.422 ± 0.969 0.866 0.662

0.801 ± 0.178 0.159 3.813 ± 1.701 1.600 0.697

0.765 ± 0.039 0.034 3.153 ± 0.825 0.737 0.761

0.846 ± 0.090 0.081 3.113 ± 0.747 0.667 0.773

Table 1: Segmentation metrics for model evaluations. CI denotes the confidence interval.

Figure 4: Box and whisker plots of Hausdorff (left) and Dice (right) scores

We evaluate our lesion segmentation model on a new Multiorgan Lesion Segmentation (MLS) dataset. Our results demonstrate favorable performance compared to competing methods, especially for small training datasets.

(a) Labeled Img (b) Level-set (d) λ1(x,y) (e) λ2(x,y) (c) DLAS

Link to code repository

Figure 5: The contribution of the parameter functions was validated by comparing the DALS (a) againsta manually initialized level-set ACM with scalar parameters constants (b). The learned maps (λ1(x,y)and λ2(x,y) ) serve as an attention mechanism that provides additional degrees of freedom for thecontour to adjust itself precisely to regions of interest.

𝐸 ∅ = Ω𝛿𝜖 ∅ 𝑥, 𝑦, 𝑡 μ|∇∅ 𝑥, 𝑦, 𝑡 | + Ω𝑊𝑠𝐹 ∅ 𝑢, 𝑣, 𝑡 𝑑𝑢 𝑑𝑣 𝑑𝑥 𝑑𝑦,

Where 𝜇 penalizes the length of the contour (we set 𝜇 = 0.1) and the energy density is

F ∅ = 𝜆1(𝑢, 𝑣)(𝐼 𝑢, 𝑣 − 𝑚1 𝑥, 𝑦 )2𝐻𝜖𝐼 ∅ + 𝜆2(𝑢, 𝑣)(𝐼 𝑢, 𝑣 − 𝑚2 𝑥, 𝑦 )2𝐻𝜖

𝐸 ∅

𝜆1 𝑥, 𝑦 = 𝑒𝑥𝑝2 − 𝑌𝑝𝑟𝑜𝑏(𝑥, 𝑦)

1 + 𝑌𝑝𝑟𝑜𝑏(𝑥, 𝑦)𝜆2 𝑥, 𝑦 = 𝑒𝑥𝑝

1 + 𝑌𝑝𝑟𝑜𝑏(𝑥, 𝑦)

2 − 𝑌𝑝𝑟𝑜𝑏(𝑥, 𝑦)

• We compared the DALSframework to manually-initialized Level-set withscalar λ parameters, U-Net,and DALS’s standalonebackbone CNN.

• DALS achieves superioraccuracies under all metricsand in all datasets (Table 1).

Input

Level-set ACM

2×2 Conv + BN + ReLU

2×2 Maxpool

2×2 Avg Pool Transition

2×2 Transpose Conv + BN

Dense Block

Series of 3x3 Conv + BN + ReLU

Multiscale Dilation Block

Sigmoid

Transformer

+

2 4 8 16

c

c

c

c

c Concatenation

[256, 256, 1]

[128, 128, 64]

[64, 64, 64] [64, 64, 136]

[32, 32, 6]

[16, 16, 6]

[8, 8, 6] [8, 8, 1176]

[16, 16, 512]

[32, 32, 256]

[64, 64, 128]

[128, 128, 64]

[256, 256, 128]

[32, 32, 150]

[16, 16, 438]

[256, 256, 1]

𝜆2(𝑥, 𝑦)

𝒀𝑺𝑫𝑴

𝒀𝒈𝒕

𝜆1(𝑥, 𝑦)

𝒀𝒐𝒖𝒕

𝒀𝒑𝒓𝒐𝒃

Brain MRLiver CTLiver MRLung CT

(a) Brain MR (b) Liver MR (c) Liver CT (d) Lung CT

Gro

un

d T

ruth

DA

LS O

utp

ut

U-N

et

Ou

tpu

t

How Does it Work?

Brain

LiverCT

LiverMR

Lung

ModelCNN Backbone

DALS

Level-set

U-Net

1.00.80.60.40.2

Brain

LiverCT

LiverMR

Lung

CNN Backbone

DALS

Level-set

U-Net

Model

76543210