High Content 2016 September 12th-14th 3rd Annual Conference

Joseph B. Martin Conference Center at Harvard Medical School, Boston, MA

Introduction to multispectral imaging

Bartek Rajwa, PhD

Bindley Bioscience Center Purdue University

Outline • 

– – – 

• • 

– – – 

• – – 

What is (multi)spectral imaging?

• 

– 

– – 

• 

– 

– • 

Origin of spectral imaging: LANDSAT system

• • 

• 

• 

Remote sensing

Hi-res imaging vs. spectral imaging

COLLECTING THE DATA

Spectral imaging in microscopy - web resources

• 

• 

• 

Collecting a lambda stack

Spectral imaging + microscopy

Spectral imaging – various approaches

Spectral microscopy hardware

• 

• 

Nikon spectral imaging system

Other multispectral arrangements in microscopy

PROCESSING THE DATA

210 bands unmixed into 7 endmembers

Multispectral small animal imaging

Problem: spectral overlap

TO-PRO-3MitoSOX RedCalcium GreenVybrant DyeCycleViolet stainMonobromobimane(mBBr)

• 

• 

Let’s revisit

Spectral unmixing overview

Mixing and unmixing

We have two detectors, but some cross-talk happens

Mixing , sorting and sieving…

a = 100

50⎡

⎣⎢

⎤

⎦⎥;M = 0.8 0.3

0.2 0.7⎡

⎣⎢

⎤

⎦⎥

Matrix notation

⎡ ⎤= ⎢ ⎥⎣ ⎦

0.8 0.30.2 0.7

M

What is the definition of “true” signal?

r1 = pa1 + (1− q)a2

r2 = (1− p)a1 + qa2

⎧⎨⎪

⎩⎪

r1r2

⎡

⎣⎢⎢

⎤

⎦⎥⎥=

p 1− q1− p q

⎡

⎣⎢⎢

⎤

⎦⎥⎥

a1

a2

⎡

⎣⎢⎢

⎤

⎦⎥⎥

′ ′= − = +⎧ ⎧⇒⎨ ⎨′ ′= − = +⎩ ⎩

′⎡ ⎤ ⎡ ⎤⎡ ⎤=⎢ ⎥ ⎢ ⎥⎢ ⎥′⎣ ⎦⎣ ⎦ ⎣ ⎦

1 1 2 1 1 2

2 2 1 2 1 2

1 1

2 2

11

s r q s r s q ss r p s r p s s

r sqr sp

The a1 and a2 (abundances) defined for unmixing are different than s1 and s2 (compensated signals) defined for compensation!

a1s1

Just a digression…

−−

⎛

⎝⎜

⎞

⎠⎟⎛

⎝⎜⎜

⎞

⎠⎟⎟=

′′

⎛

⎝⎜

⎞

⎠⎟⎛

⎝⎜⎜

⎞

⎠⎟⎟

−−

⎛

⎝⎜

⎞

⎠⎟⎛

⎝⎜⎜

⎞

⎠⎟⎟=

−

−

⎛

⎝

⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟

⎛

⎝⎜⎜

⎞

⎠⎟⎟

= =

= ÷ = ⋅

Unmixing requires inversion of the mixing matrix

MM−1 = 1= 1 0

0 1⎡

⎣⎢

⎤

⎦⎥

⎡ ⎤ ⎡ ⎤• =⎢ ⎥ ⎢ ⎥

⎣ ⎦ ⎣ ⎦

0.8 0.3 1 00.2 0.7 0 1

?

−⎡ ⎤ ⎡ ⎤ ⎡ ⎤• =⎢ ⎥ ⎢ ⎥ ⎢ ⎥−⎣ ⎦ ⎣ ⎦ ⎣ ⎦

0.8 0.3 1.4 0.6 1 00.2 0.7 0.4 1.6 0 1

Matrix inversion – simple example

a bc d

⎡

⎣⎢

⎤

⎦⎥

−1

= 1ad − bc

d −b−c a

⎡

⎣⎢

⎤

⎦⎥

4 72 6

⎡

⎣⎢

⎤

⎦⎥

−1

= 14 ⋅6 − 7 ⋅2

6 −7−2 4

⎡

⎣⎢

⎤

⎦⎥

= 110

6 −7−2 4

⎡

⎣⎢

⎤

⎦⎥

= 0.6 −0.7−0.2 0.4

⎡

⎣⎢

⎤

⎦⎥

Matrix inversion – cont.

4 72 6

⎡

⎣⎢

⎤

⎦⎥

0.6 −0.7−0.2 0.4

⎡

⎣⎢

⎤

⎦⎥ =

4 ⋅0.6 + 7 ⋅(−0.2) 4 ⋅(−0.7)+ 7 ⋅0.42 ⋅0.6 + 6 ⋅(−0.2) 2 ⋅(−0.7)+ 6 ⋅0.4

⎡

⎣⎢⎢

⎤

⎦⎥⎥

= 2.4 −1.4 −2.8 + 2.8

1.2−1.2 −1.4 + 2.4⎡

⎣⎢

⎤

⎦⎥ =

1 00 1

⎡

⎣⎢

⎤

⎦⎥

Sieved (yet, still mixed) result

• 

• 

Unmixing (or how to fix a bad sieve)

r = [95,55]; M = 0.8 0.3

0.2 0.7⎡

⎣⎢

⎤

⎦⎥; S = 1 0.4286

0.25 1⎡

⎣⎢

⎤

⎦⎥

aU = M-1r ⇒ aU = 1.4 −0.6−0.4 1.6

⎡

⎣⎢

⎤

⎦⎥ ⋅

9555

⎡

⎣⎢

⎤

⎦⎥ =

10050

⎡

⎣⎢

⎤

⎦⎥

sC = S-1r ⇒ sC = 1.12 −0.28−0.48 1.12

⎡

⎣⎢

⎤

⎦⎥ ⋅

9555

⎡

⎣⎢

⎤

⎦⎥ =

8035

⎡

⎣⎢

⎤

⎦⎥

Example: from n channels to two 2 colors

• • 

• • 

Generalization: linear mixing of fluorescence signals

r = Ma +n

• 

• 

OMG!More math!!

Mixing of three signals (overdetermined case)

M =

0.03267974 0.010416670.13071895 0.031250000.32679739 0.072916670.26143791 0.104166670.20915033 0.468750000.03921569 0.31250000

⎡

⎣

⎢⎢⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥⎥⎥

, a = 100 50⎡⎣ ⎤⎦

Overdetermined (multispectral) case

• 

• 

• 

Overdetermined case

aLS = MTM( )−1

MTr

aLS = M−1r

However, if the constraints are imposed…

• 

• 

( ) ( ){ } { }1min s.t. = 1pT

jJαα α

=∈Δ− − Δ =∑r Mα r Mα

( ) ( )

( ) ( )

11 1

11 1

ˆ ˆ ,whereT T TSCLS LS

T T TL L

P

P

−− −⊥

−− −⊥×

⎡ ⎤= + ⎢ ⎥⎣ ⎦

⎡ ⎤= + ⎢ ⎥⎣ ⎦

α α M M 1 M M 1

I M M 1 M M 1

• 

• 

( ) 1ˆ ˆ

ˆ( )

TNCLS LS

TNCLS

λ

λ

−⎧ = −⎪⎨⎪ = −⎩

α α M M

M r Mα

( ) ( ){ } { }min s.t. = 0Tjα

α α∈Δ

− − Δ ≥r Mα r Mα

EXPLORATORY ANALYSIS

Principal component analysis

• 

• 

• 

– – 

Principal component analysis – textbook explanation

• • 

=

=−1

1nT

Y

Y PX

C YY

• 

• 

−= 1A QLQ

−1

1T

nXX

So, let’s do that (Fisher’s data set)

• • 

4.5 5.5 6.5 7.5

4.5

5.5

6.5

7.5

Sepal.Length

2.0

2.5

3.0

3.5

4.0

Sepal.Width

12

34

56

7

Petal.Length

4.5 5.5 6.5 7.5

0.5

1.0

1.5

2.0

2.5

2.0 2.5 3.0 3.5 4.0 1 2 3 4 5 6 7 0.5 1.0 1.5 2.0 2.5

0.5

1.0

1.5

2.0

2.5

Petal.Width

Cx= XXT = QLQT

-0.10 0.00 0.05 0.10 0.15

-0.1

00.

000.

050.

100.

15

var 1

-0.2

-0.1

0.0

0.1

0.2

var 2

-0.10 0.00 0.05 0.10 0.15

-0.2

-0.1

0.0

0.1

0.2

-0.2 -0.1 0.0 0.1 0.2 -0.2 -0.1 0.0 0.1 0.2

-0.2

-0.1

0.0

0.1

0.2

var 3

Another approach to PCA: singular value decomposition

• 

• 

This could be used to compress images!

SVD and PCA

• 

• 

• = TX USV

( )( ) ( )( )= =

= = =

= 2

and

,

T T T

TT T T T T

T T

T

X USV XX QLQ

XX USV USV USV VSU V V 1

XX US U

To sum up…

• 

• 

• 

• 

-3 -2 -1 0 1 2 3 4

-3-2

-10

12

34

var 1

-1.0

-0.5

0.0

0.5

1.0

var 2

-3 -2 -1 0 1 2 3 4

-0.5

0.0

0.5

-1.0 -0.5 0.0 0.5 1.0 -0.5 0.0 0.5

-0.5

0.0

0.5

var 3

PCA and kernel PCA of lambda stack

-15 -10 -5 0 5 10 15

-15

-10

-50

510

151st Principal Component

2nd

Prin

cipa

l Com

pone

nt

•  We can compute PCA on spectral data •  Pure endmemebers form a simplex! •  Theory of convex sets •  Simplest endmember estimation requires

only PCA (or kPCA)

label 1

0

50

100

150

200

0 50 100 150 200

0 50 100 150 200

label 2

0

50

100

150

200

0

50

100

150

200

0 50 100 150 200

label 3

( )1SAM( , ) cosi j i j i js s −= ⋅ ⋅s s s s

PCA performed on a lambda stack

 

 

PCA performed on reflected light lambda stack

Example: Raman spectorscopy

• 

• 

Non-negative matrix factorization (NMF)

2( , )F

F = −W H X WH

Application example: autofluorescence removal

Woolfe, F. et al.., 2011. Autofluorescence Removal by Non-Negative Matrix Factorization. IEEE Transactions on Image Processing 20, 1085–1093. doi:10.1109/TIP.2010.2079810

NMF in histopathology

SCREEN QUALITY ASSESSMENT

Screen quality assessment in multiplexed HCS

• 

• 

– 

– 

• 

– 

…but first – what is Cohen’s d ?

• 

δ =

μpos − μneg

σ

d =

Ypos −Yneg

′s, ′s =

npos −1( )spos2 + nneg −1( )sneg

2

npos + nneg − 2

Cohen’s measures distance between normally distributed populations

Z’-factor and Cohen’s d

• 

• 

′Z = 1−

3 σ neg +σ pos( )μpos − μneg

, ′Z = 1−3 sneg + spos( )

Ypos −Yneg

′Z = 1− 6

d

Cohen’s d and Z’ in multiple dimensions

• 

– 

– 

– – 

• 

Why 2-D is better than 1-D?

Mahalanobis distance

D2 = dT R−1d = d1,d2 ,…,dm⎡⎣ ⎤⎦

1 r1,2 r1,m

r1,2 1

r1,m 1

⎡

⎣

⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥

−1

d1

d2

dm

⎡

⎣

⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥

.

D = dT R−1d• 

• • 

D = T 2 nposnneg

npos + nneg

Confidence interval for D

fD =

nposnneg npos + nneg − p −1( )p npos + nneg( ) npos + nneg − 2( ) D2

fD ~ Fp,m+n− p−1 λ( )

So, how can we use this information?

prob Fp,m+n− p−1 λL( )( ) = 1− α2

prob Fp,m+n− p−1 λU( )( ) = 1−α

Conclusions

• 

• 

• 

• 

• 

Thank you!