3D & 4D – Surfaces 3D slice by slice segmentation – 3D context missing

(spatial, temporal, both)

Extension of 2D dynamic programming to 3D

Combinatorial explosion

Optimal surface detection in 3D and above

Seemingly NP complete

Solutions were missing for decades

2002 – Chen & Wu - single surface graph-based solution in

low-order polynomial time and formal proof of optimality

While graph cut is used for optimization, this is

NOT Boykov’s graph cut segmentation

3/5/2012 1

LOGISMOS: Layered Optimal Graph

Image Segmentation of Multiple Objects

and Surfaces

Single surfaces

Multiple interacting surfaces

Cost functions

Complex and topology-changing surfaces

Multiple objects and multiple surfaces

Non-segmentation use – Image Resizing &

Stitching

3/5/2012 2

3/5/2012

1

5 -1

4

1

3

2

1

8

Example: Min s-t Cut approach to finding min-cost “path”Every pixel corresponds to a node in the graph, node costs usedThe path intersects with each column at exactly one nodeSmoothness constraint: max. vertical distance of neighbor-column nodes = 1

1

Min-cost path (cost = 2)

1

5 -1

4

1

3

2

1

8

Edge Construction:Connect each node to its bottom-most neighbor in the adjacent column.Build vertical edges along each column, pointing downwards.

3/5/2012

2

5 -1

4

1

3

2

1

8

Goal of the transform:transform the graph in a form that can be solved by finding a minimum-cost closed setefficient optimization exists for minimum-cost closed set

5 -1

4

1

3

2

1

8

Goal of the transform:transform the graph in a form that can be solved by finding a minimum-cost closed setefficient optimization exists for minimum-cost closed set

3/5/2012

3

5 -1

4

1

3

2

1

8

Cost Transformation:Along each column, subtract the cost of each node by the cost of the node immediately beneath it.The bottom-most two nodes are unchanged.

-1 – 2 = -35 – 4 = 1 -31

3

-2

1

-7

5 -1

4

1

3

2

1

8

Need to avoid empty zero-cost closed set solution:M: sum of costs of the bottommost nodesIf M >= 0:– Select ANY one of those nodes– Subtract (M + 1) from its cost

-31

3

-2

1

-7

M = 3 + 8 = 11 >= 0

3 – (11 + 1) = -9

-9

3/5/2012

4

5 -1

4

1

3

2

1

8

Compute the Minimum-Cost Closed Set:Closed set: a subset of the graph with no edges going outMinimum-cost closed set: A closed set with the minimum total node-cost.

-31

3

-2

1

-7

-9

A closed set

Not a closed set

The minimum cost closed set problem can be solved by a Maximum Flow algorithm…

3/5/2012

5

Add a source (s) and a sink (t) node…

We try to compute max amount of water that can flow from the source to the sink…

3/5/2012

6

And we start building the pipes in between…

s is connected to the negative-cost nodes. Pipe capacities are determined by…

3 = |-3|

9 = |-9|

3/5/2012

7

Similarly, t accepts pipes from the nonnegative-cost nodes...

Node costs are no longer needed…

3/5/2012

8

Try to find paths from s to t, and push the bottleneck amount of flow through it…

After we push a flow, some pipes may become saturated (allowing no more flow).

3/5/2012

9

The potential for reverse flow is increased, build pipes allowing water to flow back…

Saturated pipes can be removed.We continue to discover new s-t paths…

3/5/2012

10

Again, we push some flow, and one new pipe is saturated…

Remove all saturated pipes, and we cannot find any path from s to t now…

3/5/2012

11

Find the nodes that can be reached from the source, and the minimum-cost closed set is

identified.

s t

The upper envelope of the min-cost closed set is the solution.

5 -1

4

1

3

2

1

8

3/5/2012

12

Compute the Min-Cost Closed Set (cont’d):Can be solved by a Min s-t Cut (Max Flow) algorithm 2 auxiliary nodes – a start (s) & a terminal (t) are addedAn edge-weighted directed graph is built

s t

9

21

8

1 -3

3

-2

-9

1

-7

8

7

3

13

After applying the min s-t cut algorithm …Find all the nodes that can be reached from s, and the min-cost closed set is identified.The upper envelope of the min-cost closed set is the solution.

s t

3/5/2012

13

The upper envelope of the min-cost closed set is the solution.

5 -1

4

1

3

2

1

8

3D SurfaceThe surface intersects with exactly onevoxel of each column of voxels parallel to the z-axis

The difference in z-coordinates between neighboring voxels on a valid surface in x and y directions– smoothness constraint (∆x, ∆y)

3/5/2012

14

Terrain-like or Tubular Surfaces

3-D CasePrinciples presented in 2-D are applicable to 3-D

Detect a surface instead of a path

Construct Edges in both x-and y-directions

x- and y-direction may have different smoothness constraints

x

yz

3/5/2012

15

Airway Segmentation

Slice-by-slice Dynamic Programming

3-D Optimal Surface Detection

Segmentation of Pulmonary Fissures

3/5/2012

1

Interacting Surfaces

Translation to avoid empty set

Multiple (2) Interacting Surfaces

3/5/2012

2

A

B

CMinimum distance of 2 pixels, maximum of 3 pixels

Arc (A,B): If A is in the closure, B must also be in the closure (minimum distance)

Arc (C,A): If C is in the closure, A must also be … (maximum distance)

Relations between surfaces modeled by “inter-surface” arcs

Multiple Interacting Surfaces

Minimum distance of 1 pixel, maximum of 3 pixels

Multiple Interacting Surfaces

3/5/2012

3

Multiple Interacting Surfaces

Interacting Surfaces3D Phantom

Original Optimal graph search

2 interactingsurfaces

(no shape term)

Optimal graph search

single surface(no shape term)

3/5/2012

4

Pulmonary Airway Inner/Outer Surface

Detection

DEMO

IVUSSimultaneous IEL + EEL

IVUS

3/5/2012

5

Excised Iliac Artery - MR

Manual tracing Fully automated4 simultaneoussurfaces in 3D

Semi-automated2-D DP

2.4 clicks/slice

Original

3/5/2012

1

Surface set feasibility

A surface set { f1(x,y), …, fn(x,y) }

is considered feasible if:

Each surface in the set satisfies surface smoothness

constraints

Each pair of surfaces satisfies surface interaction

constraints.

Cost Function – Surface Costs One-to-one correspondence between each feasible

surface set and each closed set in the graph Graph arcs reflect surface feasibility constraints

Node costs defined so that cost of each closed set corresponds to cost of a set of surfaces

Minimum-cost closed set corresponds to optimal set of surfaces

Edge-based costs are a logical option Strong edge low node cost

Edge costs not always the best/only option regional costs

3/5/2012

2

Multi-Surface Regional Costs

Edge based costs: Each voxel corresponding local edge/gradient cost

n “on-surface” costs – unlikeliness of belonging to each of the n surfaces

Edge + region costs: each voxel assigned 2n + 1 cost values:

n + 1 “in-region” cost values, reflecting the unlikeliness of belonging to each of the n + 1 regions

n “on-surface” costs – unlikeliness of belonging to each of the n surfaces

Edge + Region Costs

Summation of “on-surface” cost terms “in-region” cost terms

3/5/2012

3

In-regions & on-surface costs

in-region cost functions

on-surface cost functions

3/5/20126

3/5/2012

4

3/5/20127

Node-based Region Costs

3/5/20128

+

+

+

+

+

+

-

-

-

-

-

-

Sub

grap

hS

urfa

ce 2

Sub

grap

hS

urfa

ce 1

0112Desired cost of closed set:

R0cost0 + R1cost

1 + R2cost2

Cost of closed set

R0cost0 - R1cost

0 + R1cost1

+ R1cost0 - R2cost

0 - R2cost1

NOT quite correct

R0cost0

R1cost1

R2cost2

3/5/2012

5

NOT quite correct Making it right

3/5/2012

9

Cost of closed set as defined so far: R0cost

0 + R1cost1 + R1cost

0 - R1cost0 - R2cost

0 - R2cost1

Correct cost to be minimized: R0cost

0 + R1cost1 + R2cost

2

R0cost0 - R1cost

0 + R1cost0 + R1cost

1 - R2cost0 - R2cost

1 + + R2cost

0 + R2cost1 + R2cost

2

Adding K= R2cost0 + R2cost

1 + R2cost2 solves the problem

K is a constant OK to add a constant

R0cost0

R1cost1

R2cost2

+

+

+

+

+

+

-

-

-

-

-

-

3/5/201210

+

+

+

+

+

+

-

-

-

-

-

-

Sub

grap

hS

urfa

ce 2

Sub

grap

hS

urfa

ce 1

0112

+

+

+

2

3/5/2012

6

In-region Costs Vertex Costs

3/5/201212

3/5/2012

7

3/5/201213

3/5/201214

3/5/2012

8

Retinal OCT

nerve fiber layer - NFL, ganglion cell layer -GCL, inner plexiform layer IPL, inner nuclear layer - INL, outer plexiform layer - OPL, outer nuclear layer - ONL, outer limiting membrane - OLM, inner segment layer - ISL,

connecting cilia - CL, outer segment layer - OSL, Verhoeff’s membrane - VM, and retinal pigment epithelium - RPE

Early Diabetes vs. Normal Controls – Layer Thickness

Van Dijk, IOVS 2009

3/5/2012

1

More than Rectangles and Tubes:Generalization to complex shapes Step 1: Pre-segmentation

Derive topology of objects of interest from image data approximate segmentation

Step 2: Mesh Generation Specify structure of a base graph defining neighboring relations among voxels

on the sought surfaces

Step 3: Image Resampling Resample along a ray intersecting every vertex of the mesh forming graph

columns.

Step 4: Graph Construction Weighted directed graph G built using from columns, with neighboring

relations, smoothness constraints, and inter-surface separation.

Step 5: Graph Search Searching for optimal closed set.

3/5/20121

Problems of Step 3 – Colliding Columns

Pre-segmented surface

True surface

Bifurcation Detection

3/5/2012

2

An electric field theory motivated search direction

Non-intersecting

Easy to compute

Expanding to any positions

Non-ELF medial-surface approach also possible

3

Electric Lines of Force (ELF)

rr

QEi ˆ

4

12

0

i iEE

Point charges

Iso-electric

fieldElectric lines of

force (ELF)

3D Airway Double Surface

Inner surface Double surfaces

3/5/2012

3

5

Aortic Thrombus from MDCT

6

Preliminary aortic/iliac lumen segmentation

3/5/2012

4

7

8

3/5/2012

1

1

Cross-Object Interactions

Multi-object 3D segmentation Regions of object-to-object interactions reflected in inter-graph arcs

Steps: Identify regions of pairwise interaction

Link interacting surfaces = create inter-graph arcs

Build/solve resulting graph

Example Prostate – Bladder – (Rectum)

Knee-joint cartilage segmentation Femur/Tibia/Patella cartilage thickness

3/5/2012

2

3/5/20123

Prostate – Bladder Segmentation

Inpu

t Im

age

Pre

-seg

men

ted

Sur

face

sC

onst

ruct

edG

raph

Fin

al r

esul

t

Pre-segmentation

Graph Optimization

3/5/2012

3

Healthy Knee

3/5/201210

Severe Osteoarthritis

3/5/201211

DEMO

3/5/2012

4

Simultaneous segmentation of an object and up to two surfaces (MVCBCT)

3/5/201212

Mutually interacting terrain-like surfaces

and regions of arbitrary topology

13

A lung tumor (orange) in megavoltage cone-beam CT

A lymph node (red) in X-ray CT data.

3/5/2012

1

Image & Video Resizing

Removal of seam lines based on specified image properties

Original Conventional Graph-based

3/5/20121

4D CT – Separate Phasic Stacks

4 stacks of CT, same phase, different respiratory cycles

LOGISMOS

graph-based stitching

Commercially available stitching

3/5/20122

3/5/2012

2

3/5/20123

LOGISMOS Commercial

Specific respiratory phase

3/5/20124

LOGISMOS Commercial

Another respiratory phase