Image segmentation

Image Segmentation (deformable segmentation)

Xianghua Xie Computer Science Department, Swansea University, UK

http://csvision.swan.ac.uk

n  high level understanding of images n  May involve segmentation

n  mid level image understanding n  From low level representations, such as pixels and edges, to provide

representation that is compact and expressive n  Segmentation often involves grouping, perceptual organisation and

fitting. n  Image partition is taken place in the image spatial domain, but grouping,

fitting and so on can be in other domain, e.g. frequency or spatio-frequency

Why Segmentation

Image Segmentation

Original image Bottom up segmentation Object level segmentation

Berkeley Segmentation Dataset is a good starting point (evaluation metrics)

Image Segmentation n  Segmentation with little constraint

n  Thresholding n  Region growing, split and merge n  Watershed

n  With weak constraint n  Graph cut n  Deformable models, such as active contour n  Interactive segmentation, such as intelligent scissors, grab

cut n  With strong constraint

n  Active shape model, active appearance model based segmentation

n  Atlas model based segmentation n  Registration populating segmentation n  Two key questions: prior generalisation and model

adaptation

Bot

tom

up

Mod

el d

riven

Outline n  Deformable Segmentation

n  Edge based 2D segmentation n  Extension to 3D

n  Direct sequential segmentation of temporal volumetric data n  Incorporating statistical shape prior

n  3D + T (4D) constrained segmentation n  Tracking using implicit representation

n  Implicit representation using RBF (region based) n  Hybrid approach

n  Level set intrinsic regularisation (initialisation invariance)

n  Integrated Reconstruction & Segmentation n  Combinatorial Optimisation

n  Minimum path: lymphatic membrane segmentation n  Optimal surface (minimum cut): coronary segmentation

Deformable Segmentation

Deformable Segmentation

Deformable Segmentation n  Design issues

n  Representation & numerical method n  Explicit Vs Implicit n  FEM, FDM, Spectral methods n  RBF-Level Set (Xie & Mirmehdi 07, Xie 11)

n  Boundary description & stopping function n  Gradient based (Caselles et al. 97, Xie & Mirmehdi 08, Xie 10

& 11, Yeo et al. 11 ) n  Region based (Paragios 02, Chan & Vese 01, Xie 09 & 11) n  Hybrid approach (Wang & Vemuri 04, Xie & Mirmehdi 04)

n  Initialisation and convergence n  Initialisation independency (Xie 10, Xie 11, Yeo et al. 11) n  Complex topology & shape (Xie & Mirmehdi 07, 08, Paiement

14)

n  These issues are often interdependent

Deformable model n  Active contour:

n  Dynamic curves within image domain to recover object shapes.

n  Deformable surface: n  Its extension to 3D.

n  Applications: n  Object localisation n  Motion tracking n  Segmentation (e.g. colour/texture)

n  Two general types: explicit and implicit models n  Kass et al. 1988, Caselles et al.1993, and many more…

Explicit model n  Parametric snake

n  Represented explicitly as parameterized curves, spline, polynormial function

n  Example classic parametric snake n  Snake evolves to minimize the internal and external forces (Let C(q)

be a parameterized planar curve);

n  Initialisation problem; n  Concavity convergence problem;

( ) ( ) ( ) ( )( ) .22

∫∫∫ ∇−ʹ′ʹ′+ʹ′= dqqCIdqqCdqqCCE λβα

Internal forces External force

Explicit model n  Point based on tracking

n  Resolution problem n  Addition and deletion n  And …

n  Topological problems n  Non-intrinsic, parameterisation dependent; n  Hard to detect multiple objects simultaneously.

n  Example:

?

Explicit model n  Advantages

n  Explicit control n  Point correspondence n  Probably easier to impose shape regularisation n  Computational efficiency

n  Should be considered when n  Known topology n  No (or predicated) topological changes n  Open curves n  …

n  Numerical method n  Finite element method (FEM) n  Discretise into sub-domain n  Cohen & Cohen, IEEE T-PAMI, 1993

Implicit model n  Popularly based on the Level Set technique n  Implicit snake models

n  Introduced by Caselles et al. and Malladi et al. (1993); n  Based on the theory of curve evolution n  Numerically implemented via level set methods; n  Snake evolves to minimize the weighted length in a Riemannian space

with a metric derived from the image content; n  Weighted length minimisation example:

A

B

A

B

Curve evolution n  Curvature flow

n  k is the curvature, N denotes the inward normal

n  The curvature measures how fast each point moves along its normal direction;

n  A simple closed curve will evolve toward a circular shape and disappear;

n  It smoothes the curve.

NCt!

κ=

Curve evolution n  Constant flow

n  c is a constant, N denotes the inward normal

n  Each point moves at a constant speed along its normal direction; n  It can cause a smooth curve to become a singular one; n  A.k.a. the balloon force.

NcCt!

=

Level set method n  A computational technique for tracking propagating interface n  Embed the curve into a surface, 2D scalar field n  Zero level set corresponds to the embedded curve n  Deforming the surface, instead of explicitly deforming the curve

Level set method n  A computational technique for tracking propagating interface n  Embed the curve into a surface, 2D scalar field n  Zero level set corresponds to the embedded curve n  Deforming the surface, instead of explicitly deforming the curve

Level set method n  Key ideas introduced by Dervieux & Thomasset

n  Lecture Notes in Physics 1980 n  Well-known after the seminal work by Osher & Sethian

n  Osher & Sethian, J. Computational Physics 1988 n  Fluid dynamics, computational geometry, material science, computer vision,

… n  Introduced to snake methods by Casselles et al. & Malladi et al.

n  Casselles et al., Nemuer. Math. 1993 n  Malladi et al., IEEE T-PAMI 1995

n  Advantages: n  Implicit, intrinsic, non-parametric n  Accurate modelling front propagation n  Capable of handling topological changes (almost!)

Level set method n  General curve evolution

n  is the level set function, F is the speed function

n  The curvature flow can be re-formulated as:

n  The constant flow can be re-formulated as:

0|| =Φ∇+Φ Ft

Φ

||||

Φ∇⎟⎟⎠

⎞⎜⎜⎝

⎛

Φ∇

Φ∇⋅∇=Φt

|| Φ∇=Φ ct

Level set method n  Numerical method

n  Finite difference method (FDM) n  A local method to estimate the partial derivatives n  Upwind scheme

n  Reinitialisation n  Reshaping the level set surface to retain smoothness n  Periodically performed n  Fast marching method, or n  By solving the following PDE:

|)|1)((sign Φ∇−Φ=Φt

Level set method n  Challenges:

n  Computational complexity n  FDM – a local method to estimate the partial derivatives n  Dense computational grid n  More expensive in 3D n  Fast marching, narrow band, additive operator splitting (AOS)

n  More sophisticated topological changes n  Signed distance function n  Reinitialisation is necessary to eliminate accumulated numerical error n  Prevent the level set developing new components n  Do not allow perturbations away from zero level set n  Can not create new contours n  E.g. fail to localise internal object boundaries

Level set method n  Challenges:

n  Computational complexity n  FDM – a local method to estimate the partial derivatives n  Dense computational grid n  More expensive in 3D n  Fast marching, narrow band, additive operator splitting (AOS)

n  More sophisticated topological changes n  Signed distance function n  Reinitialisation is necessary to eliminate accumulated numerical error n  Prevent the level set developing new components n  Do not allow perturbations away from zero level set n  Can not create new contours n  E.g. fail to localise internal object boundaries

Numerical solution n  The level set time derivative is approximated by forward

difference:

n  Force fields can be classified into three types n  Curvature flow n  Constant flow n  Advection force field

n  Each requires different differencing scheme

Numerical solution n  Weighted curvature flow:

Central differencing

Numerical solution n  Weighted constant flow:

Upwinding differencing

nji

nji Vcg ,0, |)|(|)|(.)( Φ∇=Φ∇

Numerical solution n  Advection flow:

n  Let denote the external velocity force field n  Check the sign of each component n  Construct one-sided upwind differences

n  E.g. GVF,

n  Edge Based 2D Segmentation

Xie & Mirmehdi, MAC, IEEE Trans. Pattern Analysis & Machine Intelligence 2008.

Motivation n  Convergence study – 4 disc problem

Geodesic DVF GGVF GeoGGVF CVF MAC

DVF: Cohen & Cohen, IEEE T-PAMI, 1993

Geodesic: Caselles et al., IJCV, 1997

GGVF: Xu & Prince, Signal Processing, 1998

GeoGGVF: Paragios et al., IEEE T-PAMI, 2004

CVF: Gil & Radeva, EMMCVPR 2003

CPM: Jalba et al., IEEE T-PAMI 2004


Motivation n  Convergence study – 4 disc problem


DVF: Cohen & Cohen, IEEE T-PAMI, 1993

Geodesic: Caselles et al., IJCV, 1997

GGVF: Xu & Prince, Signal Processing, 1998

GeoGGVF: Paragios et al., IEEE T-PAMI, 2004

CVF: Gil & Radeva, EMMCVPR 2003

CPM: Jalba et al., IEEE T-PAMI 2004


Motivation n  Objectives

n  Long range force interaction n  Dynamic force field, instead of static n  Bidirectional – allow cross boundary initialisation n  Efficiency

n  Region based or Edge based n  Prior knowledge n  Boundary assumptions

n  Discontinuity in regional statistics n  Discontinuity in image intensity

n  Application dependent n  Goal: improving edge based performance

n  Comparable to region based approaches n  Benefit from less prior knowledge, simpler assumption, and efficiency n  There are scenarios boundary description does not need region support

MAC model n  Proposed method

n  Novel external force field n  Based on hypothesised magnetic interactions between object boundary

and snake n  Significant improvements upon initialisation invariancy &

convergence ability n  Yet, a very simple model

n  Magnetostatics

MAC model n  Edge orientation

n  Analogy to current orientation n  Rotating image gradient vectors

= 1: anti-clockwise rotation; = 2: clockwise rotation. : normalised image gradient vectors. n  (actually, these are 3D vectors)

n  Current orientation on snake n  Similar to edge current orientation estimation n  Rotating level set gradient vectors

MAC model n  Magnetic force on snake

n  Derive the force on snake exerted from image gradients

: electric current unit vector on snake : current magnitude on snake, constant

: electric current vector on edges : current magnitude on edges

: unit vector between two point, x and s : permeability constant

n  Uniqueness n  The force on snake is dynamic n  Relies on both spatial position and evolving contour n  Always perpendicular to the snake n  Global force interaction

MAC model n  Snake formulation

: curvature : snake inward normal

n  Level set representation

n  Force field extension n  Snake is extended in a 2D scalar function n  Accordingly its forces upon it n  Fast marching n  In this case, simply compute forces for each level set

MAC model n  An example of dynamic force field

MAC model n  Edge preserving force diffusion

n  Minimise noise interference n  Nonlinear diffusion of magnetic flux density n  Similar to GGVF, but… n  Add edge weighting term in diffusion control

n  As little diffusion as possible at strong edges n  Homogeneous and noisy area which lack consistent support

from edges will have larger diffusion

MAC model n  Edge preserving force diffusion

n  Fast implementation n  Decompose the magnetic flux term n  Fast computation in the Fourier domain

Experimental results n  Comparative analysis on synthetic images


Experimental results n  Arbitrary initialisation

Experimental results n  Noise sensitivity

20% noise 30% noise 40% noise 50% noise

Experimental results n  Weak edges

Geodesic GGVF CPM MAC

Experimental results n  Weak edges

n  Brief comparison to Region Based

Region based (MoG) MAC

Geodesic GGVF CPM MAC

Experimental results n  On real images

Geodesic DVF GGVF

GeoGGVF CPM MAC

Experimental results n  On real images

Geodesic DVF GGVF

GeoGGVF CPM MAC

Experimental results n  On different types of images

Planar X-ray CT

Ultrasound MRI

Experimental results n  Dual level set


n  Extension to 3D

Yeo, Xie, Sazonov, Nithiarasu, GPF, IEEE Trans. Image Processing 2011.

GPF model n  Geometrical Potential Force

n  Suitable for 3D data n  Based on hypothesised geometrically induced force field between

deformable model and object boundary n  Generalisation of the MAC model

n  Unique bi-directionality n  Dynamic force interaction n  Global view of object boundary representation


GPF model n  Interaction force acting on due to is given as

q  – corresponding geometrically induced potential created by

GPF model n  Comparative Results

Target objects Initialisations Geodesic GGVF Proposed GPF


GPF model n  Medical 3D data segmentation

Aneurysm Aorta

GPF model n  Comparative analysis

Ground-truth GGVF Geodesic Proposed GPF

GPF model n  Further examples

GPF model n  Further examples

Human aorta (CT) Human carotid (CT)

GPF model n  Direct sequential segmentation of temporal volumetric data











GPF model n  Segmentation using statistical shape prior

GPF model n  Segmentation using statistical shape prior

Segmentation of corpus callosum from MRI

Image based energy Image and shape based energy

n  3D+T (4D) constrained SPECT segmentation

Yang, Mirmehdi, Xie, Hall, CI2BM, MICCAI workshop 2009. (MIA under-review)

n  Segmentation of LV borders allows quantitative analysis of perfusion defects and cardiac function.

4D SPECT Segmentation

SPECT slice of the LV A Doughnut Cardiac motion (mid-slice)

Yang, Mirmehdi, Xie, Hall, CI2BM, MICCAI workshop 2009. (MIA under-review)

Frontal view of opaque surface

Top view of opaque surface

Short-axis view

Frontal view

Correspondence between short-axis slice and 3D frontal view

Frontal view of opaque surface overlaid on orthogonal slice planes

Frontal view of transparent surface


Automatic Initialization

Snapshots

Input Slice CPM Geodesic

Snake Geodesic GVF CACE Ground

truth

Example 1

Example 2

Example 3

Problem Cases


Training Image and Shape Sequences

Gaussian Analysis

PCA

Gaussian Priors

Spatiotemporal Priors

Seg

men

tatio

n

Unseen Sequence

CACE Evolution

Constraint (based on Gaussian and Spatiotemporal Priors)‏

Update Level Sets and Spatiotemporal Parameters

Convergence

End

No

Yes

Trai

ning

GtL trans.


The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
















CCACE: 85.8%

SCMS: 63.1%

Results on input image

Results against ground truth

Results on input image

Results against ground truth


Slice 24

Slice 25

Slice 26

Slice 27

Slice 28

4D SPECT Segmentation – defect detection

n  Tracking using implicit representation

Chiverton, Xie & Mirmehdi, BMVC 2008 & 2009, IEEE TIP 2012

Tracking & Online Shape Learning n  Prior independent snake tracking n  Contour based tracking

n  Probably more difficult than box based tracking n  No prior knowledge n  Online shape learning and dynamic updating

60% random noise

Tracking & Online Shape Learning n  Prior independent snake tracking n  Contour based tracking

n  Probably more difficult than box based tracking n  No prior knowledge n  Online shape learning and dynamic updating

Tracking & Online Shape Learning n  Contour based object tracking n  Online shape learning n  Self-imposed shape regularisation

Without online shape self-regularisation

Proposed method

Chiverton, Xie & Mirmehdi, BMVC 2008 & 2009.

Automatic Bootstrapping & Tracking

Chiverton, Xie & Mirmehdi, IEEE T-IP 2012.

n  Online shape learning coupled with automatic bootstrapping n  Finite size shape memory n  Statistical shape modelling n  Level set based tracking – similar to previous approach

n  RBF-Level Set based Active Contouring

Xie & Mirmehdi, Image and Vision Computing 2011 & BMVC07.

Conventional level set technique n  Problems

n  Computational complexity n  Dense computation grid, particularly expensive in 3D n  Some solutions: fast marching, narrow band, AoS schemes, …

n  Can’t handle more sophisticated topological changes n  Usually requires re-initialisation to maintain a smooth surface to prevent

numerical artefacts contaminating the solution n  Perturbations away from the zero level set are missed

Conventional level set: The hole is missed!

RBF-Level Set n  RBF-Level Set

n  Use radial basis function to interpolate level set n  Updating expansion coefficients to deform level set n  Transfer PDE to ODE: efficient n  Much coarser computational grid, even irregular n  More complex topological changes readily achievable

Conventional level set Proposed method

RBF-Level Set n  RBF-Level Set

n  Use radial basis function to interpolate level set n  Updating expansion coefficients to deform level set n  Transfer PDE to ODE: efficient n  Much coarser computational grid, even irregular n  More complex topological changes readily achievable

Conventional level set Proposed method

RBF-Level Set

n  RBF interpolation n  Level set function, : a scalar function, usually obtained from the signed

distance transform n  Interpolate using a linear combination of a radial basis function,

where p(x) is a first degree polynomial and are the expansion coefficients n  The interpolation can be expressed as:

where

6

RBF-Level Set n  Updating RBF level set

n  Original level set evolution:

where F is the speed function in the normal direction n  Transferred evolution:

n  The spatial derivatives can be solved analytically n  First order Euler’s method n  Iteratively updating the expansion coefficients to evolve level set

n  Benefits: n  Coarse computational grid, could be irregular n  No need for re-initialisation n  More complex topological changes achievable

RBF-Level Set snake n  Prevent self-flattening

Non-normalised Normalised





RBF-Level Set n  Active modelling using RBF level set

n  A region based approach n  Texem based modelling n  Active contour formulation:

n  m is the number of classes n  1/m is the average expectation of a class n  u is the posterior of the class of interest

n  Level set representation:

I

RBF-Level Set n  Texems are image representations at various sizes that

retain the texture or visual primitives of a given image. n  A two-layer generative model n  Each texem represented by mean and variance: m={µ,ω} n  A bottom-up learning procedure

learning

{ } … M

Z

Xie-Mirmehdi, IEEE T-PAMI, 29(8), 2007.

RBF-Level Set n  Example learnt texems (7x7)

n  Multiscale branch based texems n  Texem grouping for multi-modal regions

Xie-Mirmehdi, IEEE T-PAMI, 29(8), 2007.

RBF-Level Set C

onve

ntio

nal

Pro

pose

d

n  On real images

RBF-Level Set

Proposed method

Conventional level set Proposed method Conventional level set Proposed method

RBF-Level Set n  Deformable modelling in 3D

Recover a hollow sphere

Initialised outside the target object Complex geometry

Xie & Mirmehdi, Image & Vision Computing 2011 & BMVC 2007.

n  Hybrid Approach

Xie & Mirmehdi, IEEE Trans. Image Processing, 2004.

RAGS model n  Region-aided (RAGS) model

n  Bridge boundary and region-based techniques n  Fusing global information to local boundary description n  Improvements towards weak edges n  More resilient to noise interference

Geodesic snake Proposed method GGVF snake

Xie & Mirmehdi, IEEE Trans. Image Processing, 2004.

n  Integrated Reconstruction, Registration and Segmentation

A. Paiement et al., IEEE Transactions on Image Processing, January 2014.

Motivation n  Modelling from 3D/4D imaging data raises two intertwined issues:

n  segmentation

n  interpolation

n  Segmentation n  partition 3D space containing the object and to distinguish data points belonging to the

object from background points

n  e.g. 2D slices, ranging from simple stacks of parallel slices to more complicated spatial configurations

Motivation n  Segmentation

n  2D independent segmentation is often not desirable

n  all the slices are better segmented simultaneously in 3D/4D

n  However, interpolation is thus necessary

n  since data often does not span the whole 3D space

n  only partial support from data

n  Imaging conditions

n  some modalities require integration over a thick slice to improve signal to noise ratio

n  e.g. 1.5T cine cardiac MRI typical slice thickness 7mm; hence spacing is 7mm or bigger

n  large spacing is also desirable in order to reduce acquisition time (patient discomfort, motion artefact)

n  in the 4D case, data must also be interpolated between available time frames

Motivation n  Argument

n  “the success of one stage (segmentation or interpolation) depends on the accuracy of the other”

n  Approaches

n  two sequential approaches: perform these two stages in opposing order

n  some first segment slices independently then interpolate from 2D contours

n  shape interpolation, notable work: Liu et al., surface reconstruction from Non-parallel curve network, CGF 27(2) 2008.

n  combining registration and segmentation

n  segment sparse volumes made up of 2D slices by registering and deforming a model on images (e.g. ASM): prior is often necessary

n  level set based method: foundation (earlier work) for what presented here

n  integrate segmentation and interpolation into a new RBF interpolated level set framework

n  simplicity and flexibility of level set

n  stability of RBF

n  inherent interpolation provided by RBF

Proposed Method n  Interpolate level set function using Strictly Positive Definitive (SPD) RBF

n  combining registration and segmentation

n  segment sparse volumes made up of 2D slices by registering and deforming a model on images (e.g. ASM): prior is often necessary

n  level set based method: foundation (earlier work) for what presented here

n  integrate segmentation and interpolation into a new RBF interpolated level set framework

n  simplicity and flexibility of level set

n  stability of RBF

n  inherent interpolation provided by RBF

Proposed Method n  Instead of evolve \phi through expansion coefficients (which involves

inverting a large matrix), evolve \alpha by minimising an energy functional E:

n  F may be any functional and is defined by the chosen segmentation method.

n  Conventional variational level set method:

n  using chain rule, a gradient descent method yields:

Proposed Method n  Rename as .

n  S is the speed of the moving front and is generally defined on the contour C only.

n  The \alpha evolution function can thus be simplified as

n  is an approximation of the Dirac function

n  this imposes a restriction of S to the contour C

n  practical choice of regularised Dirac function:

§  epsilon = 1 for sharp RBF; epsilon = 3 for flatter RBF.

Proposed Method n  RBF based interpolation methods usually define one control point per data

point

n  we define one control point per voxel of a discrete space,

n  thus allow rewrite as a convolution:

n  The initial expansion coefficients can be computed in the Fourier domain

Results

Chan-Veseinitialisation narrow-band PC

proposed PC initialisation proposed PC

Paiement et al., IEEE Trans. Image Processing 2014.

Results

initial slices (top left quadrant removed for visualisation)

central T1 weighted slice

modelled shape

central T2 weighted slice


Results

initial slices

central short axis slice

modelled shape

long axis slice


Example result

Example result

Combinatorial Optimisation n  Shortest path problem: segmentation

Combinatorial Optimisation n  Optimal surface (minimum cut)

n  Hard constraint n  Statistical shape constraint n  Temporal constraint: Kalman filter & HMM

Combinatorial Optimisation

IVUS s-t cut optimal surface texture RBF star graph proposed

n  Longitudinal cross section n  Yellow: frame-by-frame; red: proposed; green: ground truth

Combinatorial Optimisation

n  Team n  Dr. Feng Zhao, Mr. Ehab Essa, Mr. Jingjing Deng, Mr. Mike Edwards,

Mr. Robert Palmer, Mr. Yaxi Ye, Mr. David James, Mr. Jonathan Jones

n  Alumni n  Dr. Ben Daubney, Dr. Huazizhong Zhang, Dr. Dongbin Chen, Dr. Si

Yong Yeo, Dr. Cyril Charron, Dr. John Chiverton, Dr. Ronghua Yang, Mr. Liu Ren, Mr. Arron Lacey

n  Clinical collaborator n  Swansea Singleton Hospital ABM UHT at Morriston n  Bristol Royal Infirmary n  Cardiff Hospital n  Plymouth Hospital

Acknowledgement

csvision.swan.ac.uk