Intensity-Based Registration CAMP II – IN2022far.in.tum.de/static/files/teaching/09ss/camp2/Intensity... · 2013. 7. 4. · Intensity-Based Registration - C. Wachinger, N. Navab

Intensity-Based Registration

CAMP II – IN2022

Christian Wachinger, Nassir Navab

Computer Aided Medical Procedures (CAMP)

Technische Universität München, Germany

Intensity-Based Registration - C. Wachinger, N. Navab 2

Agenda

Introduction

Classification

Dimensionality / Modalities

Transformation

Registration Basis


Method

Similarity Measures

Interpolation, Multi-Resolution


Motivation

• Today, medical images are to a large extent digital

• Different images contain complementary information

• To use this information => Registration


Registration: Definition

Bring two images into spatial alignment, or establish a common geometric reference frame.

Intensity-Based Registration - C. Wachinger, N. Navab 5Source: Xenios Papademitris


Medical Imaging Modalities

MRICT PET 3D-US X-Ray US Video

2D3D

Registration

Visualization of fused data

Measurement of changes Computer aided diagnosis / therapy

Intraoperative Navigation …


Agenda

Introduction

Classification


Transformation

Registration Basis


Method

Similarity Measures



Classification Scheme

Dimensionality:

Registration Basis:

Transformation:

Modalities:

Subject/Object:

3D-3D, 2D-3D, 3D-4D, …

Extrinsic (e.g. markers) vs. Intrinsic

Rigid, affine, projective, deformable

mono-modal / multi-modal

Intra- vs. Intersubject, which anatomy

Number of Images: Pair-wise, Group-wise


Dimensionality: 2D-2D / 3D-3D

• Digital Subtraction Angiography:X-Ray or CT scan image is subtracted from a second image with contrast agent >> highlighting of blood vessels

3D CT DSA:blood vessels

in the brain

2D X-Ray DSA

• Even if the same imaging device is acquiring the two images, registration is necessary in order to compensate for small shift of the patient position and movement of organs.


Dimensionality / Modalities: 3D-3D

PET-MRMR-CT

CT Scan

MR Scan



MR Scan

C-Arm Fluoroscopy

Registration →intraop. Navigation



CT Scan

Linear Accelerator

Registration →Patient Positioning


Dimensionality / Modality : 2D – 3D

CT Scan

Ultrasound

Registration →intraop. Navigation


US segmentation with registration to segmented CT

Ultrasound CT US-Segmentation


Registration Transformation

Image Y

T(Y)

Rigid Affine Projective Deformable

⎥⎦

⎤⎢⎣

⎡vTvtA

⎥⎦

⎤⎢⎣

⎡10tA

T⎥⎦

⎤⎢⎣

⎡10tR

T ?H4x4


Registration Basis - Extrinsic

Stereotactic Frame>> invasive method!

Markerse.g. invasive bone fiducials,noninvasive skin markers

Results in 3D point sets available for registration


Registration Basis - Intrinsic

anatomical landmarks>> 3D point sets to register

segmented objects>> 3D surfaces to register

full image content>> pixel/voxel intensities


Number of Images

• Pair-wise– Aligning two images

• Group-wise– Mosaicing

– Population studies, Atlas construction


Number of Images

• Pair-wise– Aligning two images

• Group-wise– Mosaicing

– Population studies, Atlas construction

Source: S. Joshi

Before Alignment

After Alignment

Source: L. Zöllei

Atlas

Atlas


Agenda

Introduction

Classification


Transformation

Registration Basis


Method

Similarity Measures



Intensity- vs. Feature-Based Registration

• Next class: Registration based on Features

• This class: Registration based on image intensities

Corresponding Points Point Cloud ↔ Shape


Intensity-Based 3D-3D Registration

X Y X,T(Y)• Define Transformation T on one of the images

• Compare X, T(Y) using the full image content• Refine T and repeat, until convergence criteria reached.

MRI fMRI



Similarity MeasurementTransformation T

Volume Y Volume X

T(Y)

Initial T



Interpolation

Similarity Measurement

Optimization

Transformation T

Volume Y Volume X

T(Y)

Value

Iterative Execution

Initial T


Source: Xenios Papademitris

Interpolation


Optimization

Transformation T

Volume Y Volume X

T(Y)

Value

Iterative Execution

Initial T



• In each iteration, compute a similarity measure sim(X, T(Y)) using the full image content of X and Y, i.e. all voxels!

• Using T(Y) requires voxel interpolation. Consideration of speed / quality!

• Maximize sim(X, T(Y)) by varying transformation→ optimization algorithm on transformation parameters!

• 3D Rigid transformation has 6 DOF

– 3 Translation Tt

– 3 Rotation Tr

Interpolation


Optimization

Transformation T

Volume Y Volume X

T(Y)

Value

Iterative Execution

Initial T

How to represent the rotation?

T = Tt * Tr

T̂ = argmaxTsim(X,T(Y ))


Representations of 3D Rotations

• Rotation group SO(3) (special orthogonal group)

• 3 degrees of freedom

• Rotation matrix

• Axis angle

• Quaternion

• Euler angles

• …

SO(3) = { Q ∈ R3×3 : QQ> = I, det(Q) = 1}

Source: www.euclideanspace.com

Tr =

⎡⎣cos γ − sin γ 0sin γ cos γ 00 0 1

⎤⎦⎡⎣1 0 00 cosβ − sinβ0 sinβ cosβ

⎤⎦⎡⎣cosα − sinα 0sinα cosα 00 0 1

⎤⎦



• In each iteration, compute a similarity measure sim(X,T(Y)) using the full image content of X and Y, i.e. all voxels!

• Using T(Y) requires voxel interpolation. Consideration of speed / quality!

• Minimize sim(X, T(Y)) by varying transformation→ optimization algorithm on transformation parameters!


Similarity Measures: SSD / SAD

Volume X Volume T(Y)

∑ −=i

ii yxN

SSD 2)(1

Sum of Squared Differences:Simple measure, can be optimized

with special algorithms

∑ −=i

ii yxN

SAD 1

Sum of Absolute Differences:Less sensitive on large intensity

differences than SSD

→ Only for intra-modal Registration, e.g. CT-CT



Similarity Measures: SSD / SAD

• Images slightly tilted:

• However: We encounter problems already if the images have different contrast or window/level values

- =

- =


Correlation Coefficient (Normalized Cross Correlation)


∑ −−=i

iiyx

yyxxNCC ))((1σσ

Normalized Cross Correlation:Expresses the linear relationship between voxel

intensities in the two volumes

→ Already looser dependence on intensities!


Correlation Coefficient – Example

• Two random variables X and Y

• What is their Correlation Coefficient CC(X,Y) ?

• X uniformly distributed in [-1;1], Y = X2, CC(X,Y) ?

X 3 5 7 9 11Y 11 19 27 35 43

Elements

X -1 -0,9 -0,8 -0,7 -0,6 -0,5 -0,4 -0,3 -0,2 -0,1 0Y 1 0,81 0,64 0,49 0,36 0,25 0,16 0,09 0,04 0,01 0

Elements

0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 10,01 0,04 0,09 0,16 0,25 0,36 0,49 0,64 0,81 1

…


Correlation Coefficient - Example

• Four datasets, two variables X,Y, NCC(X,Y) = 0.81 for each.

Source: wikipedia


Correlation Coefficient - Example

• Point sets (x,y), with correlation coefficient of x and y.

Source: wikipedia


Multi-modal 3D-3D Registration

No simple relationship between X and Y intensities anymore:

?

CT MR

Two approaches:

- Simulate one image from the other → reduce it to the mono-modal case

- Use more sophisticated similarity measures


• How to measure the relationship between X and Y ?

• Measure the structure of the joint distribution

• How to measure the structure?

• Shannon Entropy

• Optimum at a highly structured (clustered) histogram

Information Theoretic Approach

X

Y


Shannon Entropy, developed in the 1940s(communication theory)

∑−=i

ii ppH log

i

pi

i

piuniform distribution→ maximum entropy

any other distribution→ less entropy

Information Theoretic Approach - Entropy


• Images X and Y are treated as random variables

• px(i) = probability of a pixel of X having the intensity value i

• px(i) and py(i) can be estimated from histograms

i=0…255

px(i)

i=0…255

py(i)

Information Theoretic Approach - Entropy


Joint Entropy

• At each voxel location, we have a pair of intensity values, representing the combined image.

∑∑−=i j

xyxy jipjipYXH ),(log),(),(


Joint Entropy

px(i)

py(i)

pxy(i,j)

px(i)

py(i)

pxy(i,j)

X and Y identical X and Y misaligned


Joint Histogram

Intensity of Reference X

Intensity of TransformedTarget Y

SSD OptimumY = X

NCC OptimumY = a*X + b


Joint Entropy

• Histogram for images from different Modalities

Joint HistogramTarget ImageSource Image

Not Aligned

Aligned

Source: PhD Thesis, L. Zöllei [7]


Mutual Information

• Maximized if X and Y are perfectly aligned

• Any permutation on the intensity values of X or Y does not affect the measure → great for multimodal registration!

• H(X) and H(Y) help to make the measure more robust

∑∑=

=−+=

i j yx

xyxy jpip

jipjip

YXHYHXHYXMI

)()(),(

log),(

),()()(),(


Agenda

Introduction

Classification


Transformation

Registration Basis


Method

Similarity Measures



Image Grid

Origin

Spacing in y

Spacing in x

An Image is sampling of a continuous field using a discrete grid

Source: ITK registration methods


Image Grid

Pixel Value

Pixel Region

[0,0] [1,0] [2,0]

[0,1]

[0,2]

[0,3]

Partial Volume Effect: a single voxel contains a mixture of multiple tissue values



Fixed Image Grid

j

i

y

xFixed Image

Physical Coordinates

y’

x’Moving Image

Physical Coordinates

Moving Image Grid

j

i

Space Transform



Things I will not do …

I will not register images in pixel spaceI will not register images in pixel spaceI will not register images in pixel spaceI will not register images in pixel spaceI will not register images in pixel spaceI will not register images in pix



Quiz

256 x 256 pixels

MRI-T2

128 x 128 pixels

PET

Scaling Transform

What scale factor ?a) 2.0b) 1.0c) 0.5



Things I will not do …

I will not register images in pixel spaceI will not register images in pixel spaceI will not register images in pixel spaceI will not register images in pixel spaceI will not register images in pixel spaceI will not register images in pix



Interpolation Methods

• Essential Problem: Transforming an Image while keeping the rasterization grid

→ →

I1 T(I1) I2=T(I1)


Interpolation – Forward Warping

I2(u’,v’) I2(u’+1,v’)

I2(u’,v’+1) I2(u’+1,v’+1)

→ Holes possible


Interpolation – Backward Warping

⎟⎟⎠

⎞⎜⎜⎝

⎛′′

=⎟⎟⎠

⎞⎜⎜⎝

⎛ −

vu

Tyx 1

))(),((),( 12 yroundxroundIvuI =′′

I2(u’,v’)

I1(u,v)

I1(u+1,v)

I1(u+1,v+1)

I1(u,v+1)

(x,y) Nearest Neighbor Interpolation:

Bilinear Interpolation:

)1)(1)(1,1()1)()(1,())(1)(,1(

))()(,(),(

1

1

1

12

yvxuvuIyvuxvuIvyxuvuI

vyuxvuIvuI

−+−++++−+−++−−++

+−−=′′


Interpolation by Convolution

• Expression as convolution with a filter kernel h

Nearest:

Bilinear:

-2 -1 0 1 2-0.2

0

0.2

0.4

0.6

0.8

1 nearestlinearcubic

Bicubic:


Interpolation in 3D

• Extension to 3D: Trilinear Interpolation

Vxyz = V111 · (x− 1)(y − 1)(z − 1)+V110 · (x− 1)(y − 1)z+V101 · (x− 1)y(z − 1)+V011 · x(y − 1)(z − 1)+

V100 · (x− 1)yz+V010 · x(y − 1)z+V001 · xy(z − 1)+

V000 · xyz


Multi-Resolution (multi-scale) Approach

Registration

Registration

Registration

Fixed Image Moving Image



Student Project : Simultaneous Deformation

• Combination of registration of

– multiple images: simultaneously

– deformable registration


The end …

Thank you for your attention!

Documents

Intensity-Based Registration CAMP II – IN2022far.in.tum.de/static/files/teaching/09ss/camp2/Intensity... · 2013. 7. 4. · Intensity-Based Registration - C. Wachinger, N. Navab