Multispectral Imaging and Unmixing Jrgen Glatz Chair for
Biological Imaging www.cbi.ei.tum.de Munich, 06/06/12 Slide 2
Intraoperative Fluorescence Imaging Fluorescence Channel Color
Channel Slide 3 Outline Multispectral Imaging Unmixing Methods
Exercise: Implementation Slide 4 Multispectral Imaging
Multispectral Imaging Unmixing Methods Exercise: Implementation
Slide 5 Multispectral Imaging Nature Spatial Resolution
Magnification Spatial Resolution Magnification Spectral Resolution
Sensitivity Range Spectral Resolution Sensitivity Range Technology
Spatial Resolution and Magnification are significantly improved
Spectral Resolution has practically not improved since first camera
Slide 6 Color Vision Monochrome image of an apple treeColor image
of an apple tree Anyone feeling hungry? Color vision helps to
distinguish and identify objects against their background (here:
fruit and foliage) Color vision provides contrast based on optical
properties Slide 7 Color Vision redgreenblue redgreenblue Spectral
sensitivity of the human eye longmidshort low light wavelength
perception Color receptors (cone cells) with different spectral
sensitivity enable trichromatic vision Limited spectral range and
poor resolution redgreenblue Slide 8 Limited spectral range Visible
Ultraviolet Cleopatra butterflyEvening primrose Human eyes can only
see a portion of the light spectrum (ca. 400-750nm) Certain
patterns are invisible to the eye Slide 9 Limited spectral
resolution Different chemical composition Color vision is
insufficient to distinguish between two green objects Differences
in the spectra reveal different chemical composition plastic
chlorophyll redgreenblueredgreenblue Same color appearance Slide 10
Optical Spectroscopy Spectroscopy analyzes the interaction between
optical radiation and a sample (as a function of ) Provides
compositional and structural information Absorbance Fluorescence
Transmittance Emission Absorbance Fluorescence Transmittance
Emission Slide 11 Directions of optical Methods ImagingSpectroscopy
Currently there are two directions in optical analysis of an object
Camera Spectrometer A B Provides spatial information Provides
spectral information Reveals morphological features No information
about structure or composition / no spectral analysis Spectrum
reveals composition and structure No information about spatial
distribution Slide 12 Imaging Spectroscopy Spatial dimension y
Spatial dimension x Spectral dimension Spatial dimension y Spatial
dimension x Spectral dimension ImagingSpectroscopy Spatial
information Spectral information Imaging Spectroscopy Spectral Cube
Spatial and spectral information Slide 13 Spectral Cube 11 22 33 44
55 66 77 88 Acquisition of spatially coregistered images at
different wavelengths The maximum number of components that can be
distinguished equals the number of spectral bands The accuracy of
spectral unmixing increases with the number of bands Chemical
compound A Chemical compound B A Bchlorophyllplastic Pseudo-color
image representing the distribution of compounds A and B
(chlorophyll and plastic) Wavelength Slide 14 Multispectral Imaging
Modalities Camera + Filter Wheel Bayer Pattern Cameras + Prism
Multispectral Optoacoustic Tomography etc. Slide 15 Lets find those
apples Multispectral imaging alone is only one side of the medal
Appropriate data analysis techniques are required to extract
information from the measurements Slide 16 Unmixing Methods
Multispectral Imaging Unmixing Methods Exercise: Implementation
Slide 17 The Unmixing Problem Finding the sources that constitute
the measurements For multispectral imaging this means separating
image components of different, overlapping spectra Unmixing is a
general problem in (multivariate) data analysis Unmixing Slide 18
Multifluorescence Microscopy Disjoint spectra can be separated by
bandpass filtering Overlapping emission spectra create crosstalk
Slide 19 Autofluorescence Autofluorescence exhibits a broadband
spectrum Only mixed observations of the components can be measured
Post-processing to unmix them I Slide 20 Forward Modeling What
constitutes a multispectral measurement at a certain point and
wavelength? Principle of superposition: Sum of individual component
emission A components emission over different wavelengths is
denoted by its spectrum, its spatial distribution is still to be
defined. Slide 21 Setting up a simple forward problem (1) Two
fluorochromes on a homogeneous background Note: We define images as
row vectors of length n All components are merged in the (n x k)
source matrix O n: Number of image pixels k: Number of spectral
components Slide 22 Setting up a simple forward problem (2)
Defining the emission spectra for all components at the measurement
points Combining them into the (k x m) spectral matrix k: Number of
spectral components m: Number of multispectral measurements m k
Wavelength [nm] Relative Absorption [%] Slide 23 Setting up a
simple forward problem (3) Two fluorochromes on a homogeneous
background Heavily overlapping spectra 25 equidistant measurements
under ideal conditions Wavelength [nm] Relative Absorption [%]
Slide 24 Mathematical Formulation Multispectral measurement matrix
(n x m) Original component matrix (n x k) Spectral mixing matrix (k
x m) (+N) Noise, artefacts, etc. (n x m) Slide 25 Multispectral
Dataset Slide 26 Mathematical Formulation 10000x2510000x33x25 =
Slide 27 Linear Regression: Spectral Fitting Reconstructing O
System generally overdetermined: No direct inverse S -1 Generalized
inverse: Moore-Penrose Pseudoinvere S + Spectral Fitting: Finding
the components that best explain the measurements given the spectra
Minimizing the error: Slide 28 Spectral Fitting Orthogonality
principle: optimal estimation (in a least squares sense) is
orthogonal to observation space Slide 29 Spectral Fitting Slide 30
Given full spectral information (i.e. about all source components)
the data can be unmixed Spectral Fitting Slide 31 Blood oxygenation
in tumors Slide 32 Multifluorescence Imaging RGB imageFITC
TRITCCy3.5Food AutofluorescenceComposite Nude mice with two
different species of autofluorescence and three subcutaneous
fluorophore signals: FITC, TRITC and Cy3.5. (Totally 5 signals)
Slide 33 Spectral Fitting Fast, easy and computationally stable
Known order and number of unmixed components Quantitative Requires
complete spectral information Crucially depends on accuracy of
spectra (systematic errors) Suitable for detection and localization
of known compositions Slide 34 ? ? Still no apples Slide 35
Principal Component Analysis Blind source separation (BSS)
technique Requires no a priori spectral information Estimates both
O and S from M Assumption: Sources are uncorrelated, while mixed
measurements are not Slide 36 Principal Component Analysis Unmixing
by decorrelation: Orthogonal linear transformation Transforms the
data into a space spanned by the orthogonal PCs Maximum variance
along first PC, maximum remaining variance along second PC, etc.
Slide 37 Unmixing multispectral data with PCA 25 multispectral
measurements are correlated Their entire variance can (ideally) be
expressed by only 3 PCs Dimension reduction Those 3 PCs are the
unmixed sources Note that matrix orientations may vary between
different implementations Slide 38 Computing PCA Subtract mean from
multispectral observations Covariance Matrix: Diagonalizing C M :
Eigenvalue Decomposition Eigenvectors of C M are the principal
components, roots of the eigenvalues are the singular values
Projecting M onto the PCs: Method 1 (preferred for computational
reasons) Slide 39 Computing PCA with the SVD Method 2 (not suitable
for implementation) Subtract mean from multispectral observations
Singular Value Decomposition: M = UV T U is a (m x m) matrix of
orthonormal (uncorrelated!) vectors Projecting M onto those
decorrelates the measurements Singular values in denote how much
variance is explained by the respective PC Slide 40 PCA does more
than just unmix U is a (non-quantitative) approximation of the PCs
spectra These can be used to verify a components identity is the
singular value matrix Relatively small singular values indicate
irrelevant components Multispectral data space Original data space
PCA UTUT Mixing S (U T ) -1 = U S Slide 41 PCA Spectra Slide 42
Slide 43 Principal Component Analysis (PCA) Needs no a priori
spectral information Also reconstructs spectral properties
Significance measurement through singular values Unknown order and
number of components Generally not quantitative Crucially depends
on uncorrelatedness of the sources Suitable for many compounds and
identification of unknown components Slide 44 Advanced Blind Source
Separation Independent Component Analysis (ICA): assumes
statistically independent source components, which is a stronger
condition than PCAs orthogonality Non-negative Matrix Factorization
(NNMF): constraint that all elements must be positive Commonly
computed by iterative optimization of cost functions, gradient
descent, etc. Slide 45 Independent Component Analysis Assumes and
requires independent sources: Independence is stronger than
uncorrelatedness Slide 46 Independent Component Analysis Central
limit theorem: Sum of non-gaussian variables is more gaussian than
the individual variables Kurtosis measures non-gaussianity:
Maximize kurtosis to find IC Reconstruction: Slide 47 Practical
Considerations Noise Artifacts (from reconstruction, reflections,
measurement,) Systematic errors (spectra, laser tuning,
illumination,) Unknown and unwanted components Slide 48 Exercise:
Implementation Multispectral Imaging Unmixing Methods Exercise:
Implementation Slide 49 Forward Problem / Mixing Define at least 3
non-constant images representing the original components Plot them
and store them in the matrix O Define an emission spectrum for
every component at an appropriate number of measurment points Plot
them and store them in the matrix S Calculate the measurement
matrix as M = OS (and save everything) Slide 50 Forward Problem /
Mixing Wavelength [nm] Relative Absorption [%] OS Slide 51 Forward
Problem / Mixing Change matrices into vectors: y=reshape(X,) or
y=X(:) Plot image from a matrix: imagesc(X) or imshow(X) Useful
MatLab functions Slide 52 Spectral Fitting Create an m-file and
write a function that Has M and S as input variables Calculates the
pseudoinverse S + Returns the unmixing R pinv Test it on your data
Slide 53 Spectral Fitting Functions: function [out] = name([input])
Regular matrix inverse: y = inv(x) Useful MatLab functions Slide 54
Principal Component Analysis Create an m-file and write a function
that Has M as an input variable Subtracts the mean from the
measurements in M Computes the covariance matrix C M Performs an
eigenvalue decomposition on C M Sorts the eigenvalues (and
corresponding vectors) by size Projects M onto the eigenvectors
Returns the projected unmixing, the principal components and their
loadings Slide 55 Principal Component Analysis Mean: y = mean(x)
Eigenvalue Decomposition: [e_vec e_val] = eig(X) Useful MatLab
functions Slide 56 Testing your code Try fitting and PCA on your
mixed data Try adding different types and amounts of noise to M
(e.g. using imnoise) Simulate systematic errors in your spectra
(noise, changing values, offset,) Slide 57 Independent Component
Analysis (voluntary) You can download the FastICA MatLab code from
http://research.ics.tkk.fi/ica/fastica/
http://research.ics.tkk.fi/ica/fastica/ Type doc fastica for
function description Use the fastica function to unmix your
simulated data Compare the result to PCA. What are advantages and
disadvantages of ICA? Slide 58 Recommended Reading Shlens, J. A
Tutorial on Principal Component Analysis
http://www.cfm.brown.edu/people/gk/APMA2821F/PCA-Tutorial-
Intuition_jp.pdf Garini, Y., Young, I.T. and McNamara, G. Spectral
Imaging: Principles and Applications; Cytometry Part A 69A:
p.735-747 (2006) http://dx.doi.org/10.1002/cyto.a.20311
http://dx.doi.org/10.1002/cyto.a.20311 Stone, J.V. A brief
Introduction to ICA; Encyclopedia of Statistics in Behavioral
Science, Vol. 2, p. 907-912 http://jim-
stone.staff.shef.ac.uk/papers/ica_encyc_jvs4everrit2005.pdf
