RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Residual Component Analysis ofHyperspectral Images for Joint NonlinearUnmixing and Nonlinearity Detection
Steve McLaughlin(1)
Joint work with Y. Altmann(1), N. Dobigeon(2), J.-Y. Tourneret(2) and M.Pereyra(3)
(1)Heriot-Watt University – School of Engineering of Physical Sciences, U.K.(2)University of Toulouse – IRIT/INP-ENSEEIHT/Tı¿ 1
2 SA, France(2)University of Bristol – School of Mathematics,U.K.
MAHI 2014 workshop, Nice, 15/12/2014
1 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Spectral unmixing
Features of interest
I pure spectral signatures:endmembers
I proportions of eachcomponent: abundances
Unsupervised unmixing
I Endmembers andabundances unknown
I Blind source separation
Supervised unmixing
I Endmembers assumed to beknown
I Abundance estimation:inverse problem
2 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Linear spectral unmixing
Linear mixing model (LMM)
yn =
R∑r=1
ar,nmr + en = Man + en
I yn nth observed pixel in L bandsI M = [m1, . . . ,mR] endmember matrix containing the spectra of
the R pure components of the sceneI an = [a1,n, . . . , aR,n]T abundance vector of the nth pixel
Abundance constraints
Abundance positivity: ar,n ≥ 0, r = 1, . . . , R n = 1, . . . , N.
Abundance sum-to-one:∑Rr=1 ar,n = 1 n = 1, . . . , N.
3 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Nonlinear spectral unmixing
yn = g (M,an) + en
I Possible interactions between the components of the scene
I Nonlinear terms included in the mixing model
I Several models depending on the nature of the scene (intimatemixtures, bilinear models, kernel-based models,...)
4 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Nonlinear spectral unmixing
Endmember variability
I Changes in observation conditionsI Intrinsic variability of the materials
Endmember Spectra measured using a handheld ASD spectrometer: Asphalt
(blue), Yellow Curb (black), Grass (red), Oak Leaves (green). (source: Master
Thesis of Xiaoxiao Du, University of Missouri-Columbia).
5 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Nonlinear spectral unmixing
Nonlinear unmixing to account for general deviations from the LMM
I Deformation of the classical data simplex
PCA-based representation of the real Villelongue image pixels.
6 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Outline
Kernel-based mixing models
Residual Component AnalysisBayesian modelResults
Generalised RCABayesian modelResults
Conclusion
7 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Outline
Kernel-based mixing models
Residual Component AnalysisBayesian modelResults
Generalised RCABayesian modelResults
Conclusion
8 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Kernel-based mixing models
Motivations
I Various sources of nonlinear effects
I Several possible parametric models to be considered (modelselection,. . . )
I Kernel-based models general and flexible tools to handle differentnonlinearities
I Often rely on few hyperparameters
9 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Kernel-based mixing models
Two main model classes
I Discriminative models (RBF1, Kernel-FCLS2,...)
an ≈ f(M,yn)
I Generative modelsyn ≈ f(M,an)
⇒ Generative models related to the actual acquisition process, moreintuitive
1Y. Altmann et al., “Non- linear unmixing of hyperspectral images using radial basis functions
and orthogonal least squares,” in Proc. IEEE IGARSS , Vancouver, Canada, 2011.2J. Broadwater et al., “Kernel fully constrained least squares abundance estimates,” in Proc.
IEEE IGARSS, Barcelona, Spain, 2007.
10 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Gaussian Processes for SU
Unsupervised unmixing problem
yn ≈ f(M,an)
I Blind nonlinear source separation problem: Estimation of M,anand f(·) unknown
I Challenging problem (even if f(·) is known)
Idea: Modeling f(·) by a Gaussian Process (GP)
11 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Gaussian Processes for SU
First Idea
I Consider a set of N pixels observed at L spectral bands gatheredin Y
I Abundance matrix: A
I Vectorized observation matrix Y: (NL× 1 vector)
Y: ∼ N (µM,A,KM,A)
I KM,A models correlations between pixels (through A) andspectral bands (through M)
⇒ Computationally intractable without particular structure for KM,A
Solution: Consideration of correlated pixels or bands only
12 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Gaussian Processes for SU
Second Idea: GP-LVM
I L spectral bands assumed to be i.i.d.
y:,` ∼ N (µA,KA) , ∀`
I M not explicit in the model
I Choice for KA: Gaussian Kernel3, Polynomial Kernel4
I Kernels possibly too general: regularization required (LLE,ISOMAP,. . . )
I More promising results obtained with the polynomial kernel(more suited for bilinear mixtures)
3Y. Altmann et al.,“Nonlinear unmixing of hyperspectral images using Gaussian processes,” in
Proc. ICASSP, Kyoto, Japan, 2012.4Y. Altmann et al., “Nonlinear spectral unmixing of hyperspectral images using Gaussian
processes,” IEEE Trans. Signal Processing, May 2013.
13 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Gaussian Processes for SU
Conclusion on GP-based unsupervised unmixing
I GPs flexible enough to handle correlations between pixels andbetween spectral bands
I Interesting results although the problem is highly ill-posed
I Model complexity possible prohibitive for large data sets
I Results highly dependent on the kernel choice (withoutadditional regularization)
I Limited to one mixing model per image...
⇒ GPs more adapted for more constrained problems, e.g., supervisedunmixing
14 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Outline
Kernel-based mixing models
Residual Component AnalysisBayesian modelResults
Generalised RCABayesian modelResults
Conclusion
15 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Residual Component Analysis (RCA)
RCA-based model
yn = Man + φn + en, n = 1, . . . , N
I φn: additive nonlinear effects in the nth pixel
I en ∼ N (0,Σ0) with Σ0 diagonal
Contraints
ar,n ≥ 0, ∀r, n and
(R∑r=1
ar,n = 1 ∀n
)
16 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Residual Component Analysis (RCA)
On the sum-to-one constraint
I Widely acknowledged for linear SU
I Accurate when illumination changes occur?
I Abundance interpretation in nonlinear mixture?I Relief: projected areas?I Multilayer scene: visible surfaces, volume?
⇒ Both cases investigated: with/without enforced sum-to-one
17 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Residual Component Analysis (RCA)
General nonlinear model
yn = Man + φn + en, n = 1, . . . , N
I φn = 0→ LMM
I φn =∑Ri=1
∑Rj=1 βi,j,nmi �mj → polynomial models5 6 7
(� denotes the Hadamard (termwise) product)
I φn = ψ (M)→ K-Hype8
I ...
5J. M. P. Nascimento and J. M. Bioucas-Dias, “Nonlinear mixture model for hyperspectral
unmixing,” Proc. SPIE Image and Signal Processing for Remote Sensing XV, 2009.6A. Halimi et al., “Nonlinear unmixing of hyperspectral images using a generalized bilinear
model,” IEEE Trans. Geoscience and Remote Sensing, vol. 49, no. 11, pp. 4153-4162, Nov. 2011.7Y. Altmann et al., “Supervised nonlinear spectral unmixing using a post-nonlinear mixing model
for hyperspectral imagery,” IEEE Trans. Image Process., 2012.8J. Chen et al., “Nonlinear unmixing of hyperspectral data based on a
linear-mixture/nonlinear-fluctuation model,” IEEE Trans. Signal Process., 2013.
18 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Bayesian model
Modeling nonlinearities
Nonlinearity-based image partition
I K classes {Ck}k=0,...,(K−1) corresponding to K levels ofnonlinearity
I zn: class label of the nth pixel
I yn ∈ Ck ⇔ zn = k
Label priorPotts Markov model for z = [z1, . . . , zN ]T
f(z) =1
G(β)exp[β
∑Nn=1
∑n′∈V(n) δ(zn−zn′ )]
I Spatial dependencies between nonlinearity levels
I β > 0: fixed granularity parameter
19 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Bayesian model
Modeling nonlinearities
Linearly mixed pixels (k = 0)
f(φn|zn = 0) =
L∏`=1
δ(φ`,n)
Nonlinearly mixed pixels (k 6= 0)
φn|M, zn = k, s2k ∼ N
(0L, s
2kKM
)I [KM]i,j =
(mTi,:mj,:
)2, i, j ∈ {1, . . . , L}
I s2k: controls the nonlinearity level of the kth class→ included in the Bayesian model: f(s2
k) non-informativeinverse-gamma distribution
I Tractable marginalization of φn
20 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Bayesian model
Bayesian model
Other model elements
I Likelihood: Gaussian distribution (zero-mean coloured noise)
I Other model parameters/hyperparameters: classical priordistributions
Joint posterior distributionAfter marginalization of Φ
f(θ|Y,M) ∝ f(Y|M,θ)f(θ)
with θ ={A, z,σ2, s2
}⇒ MCMC method (Metropolis-within-Gibbs) to sample from theposterior distribution...
21 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Results
Real data
Data set
I 180× 250 pixels image composed of R = 5 endmembers (L = 160)
I Endmembers extracted from the data using VCA
I K = 5 levels of nonlinearity, β = 1.6
22 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Results
Real data
Estimated Abundances (with sum-to-one constraint)
23 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Results
Real data
Estimated Abundances (without sum-to-one constraint)
Sum of the estimated abundances for each pixel of the image.
I Abundance maps similar to those obtained with sum-to-oneconstraint
I Sums of the estimated abundance “naturally” close to 1
24 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Results
Real data
Nonlinearity detection
25 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Results
Residual component analysis model
Summary
I RCA: flexible nonlinear model able to approximate differentmixing models
I Joint nonlinearity detection and abundance estimation
I Spatially correlated nonlinearities
I Image segmentation based on nonlinearity levels
I marginalisation of the nonlinearity spectra
Limitations
I Estimation of the Potts parameter β: efficiently solved bymarginal maximum likelihood estimation
I Positivity of the interaction spectra? (zero-mean GPs)
I Selecting the number of classes K...
26 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
RCA-based model
Results
Residual component analysis
Limitations
I Number of nonlinear classes K
I Often depending on the application (possibly infinite number ofclasses)
I Estimation of one nonlinearity level per pixel + spatialcorrelation...
27 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Outline
Kernel-based mixing models
Residual Component AnalysisBayesian modelResults
Generalised RCABayesian modelResults
Conclusion
28 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Bayesian model
Generalised Residual Component Analysis (GRCA)
RCA reformulation
yn = Man + φn + en, n = 1, . . . , N
yn = Man + φ(γn) + en, n = 1, . . . , N
φ(γn) =
R−1∑k=1
R∑k′=k+1
γ(k,k′)i,j
√2mk �mk′ +
R∑k=1
γ(k)i,j mk �mk.
I γn = [γ(1,2)n , . . . , γ
(R−1,R)n , γ
(1)n , . . . , γ
(R)n ]T of length
P = R(R+ 1)/2
I γn|s2n ∼ N
(0, s2
nIP)→ RCA
I Nonlinearity positivity: γn|s2n ∼ N(R+)P
(0, s2
nIP)
29 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Bayesian model
Generalised Residual Component Analysis (GRCA)
Introduction of spatial correlation
{γn|s2
n ∼ N(R+)P(0, s2
nIP)
s2n ∼ IG(α3, α3α4,n).
I α4,n: controls the mean/mode of the prior
I α3: controls the shape of the tails
I Correlation of the {s2n}: α4,n depending on the nonlinearity levels
in neighbouring pixels
⇒ Hidden gamma Markov random field (GMRF): α3 controls thecorrelation level
30 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Bayesian model
Bayesian model
Other model elements
I Likelihood: Gaussian distribution (zero-mean coloured noise)
I Other model parameters/hyperparameters: classical priordistributions
Bayesian inference
I MCMC method (stochastic gradient MCMC) to sample from theposterior distribution
I GMRF parameter estimation: marginal maximum likelihoodestimation
I More computationally efficient than samplingI Provide accurate unmixing results in practice
I Other model parameters estimated via Monte Carlo integration
31 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Bayesian model
Bayesian estimation
Parameter estimation
AMMSE = E [A|Y, α3](‖φn‖
2
2
)MMSE
= E[‖φn‖22 |Y, α3
]Bayesian nonlinearity detection
Tn(φn,an) =‖φn‖22∥∥∥yn −Man − φ(t)
n
∥∥∥2
2
Pn = E [Tn(φn,an) > η|Y, α3]
I η: user-defined threshold (η ∈ {2, 3} in practice)I Nonlinearity detection via probability thresholding
32 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Results
Synthetic data
Data set
I 100× 100 pixels image composed of R = 3 endmembers
I 6 mixing modelsI LMM (without sum-to-one constraint)I LMM (with sum-to-one constraint)I Generalized bilinear model (GBM)I Polynomial post-nonlinear mixing model (PPNMM) (b = 0.2)I Nascimento’s bilinear modelI RCA model (s2 = 0.1)
I SNR ' 30dB
33 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Results
Synthetic data
34 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Results
Unmixing performance
Abundance estimation
RNMSEk =
√1
NkR
∑n∈Ik
‖an − an‖2
I an: nth estimated abundance vector
C1 C2 C3 C4 C5 C6(LMM-WSTO) (LMM-STO) (GBM) (PPNM) (NM) (RCA)
NLCS 0.98 0.96 5.11 5.10 10.38 26.35FLCS 81.45 0.59 11.57 9.90 32.51 30.40GBM 80.68 0.60 4.64 5.01 32.54 29.25PPNM 70.33 1.11 1.85 0.97 28.13 23.08NM 81.06 0.98 11.53 9.77 2.73 29.43RCA 1.12 1.09 2.69 2.62 3.63 6.85
G-RCA 1.34 1.29 2.69 2.65 3.56 6.96G-RCA+ 1.21 1.14 2.11 2.86 2.88 19.63
Table: Abundance RNMSEs (×10−2).35 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Results
Unmixing performance
Reconstruction
REk =
√1
NkL
∑n∈Ik
‖yn − yn‖2
I yn: nth reconstructed pixel
C1 C2 C3 C4 C5 C6(LMM-WSTO) (LMM-STO) (GBM) (PPNM) (NM) (RCA)
NLCS 1.72 1.72 1.81 1.78 2.07 4.43FLCS 55.12 1.72 2.61 2.49 7.65 11.69GBM 54.54 1.72 1.92 2.01 7.65 11.28PPNM 9.47 1.72 1.73 1.72 3.26 4.00NM 55.09 1.73 2.62 2.62 1.71 9.97RCA 1.72 1.71 1.70 1.71 1.70 1.70
G-RCA 1.72 1.71 1.70 1.71 1.70 1.70G-RCA+ 1.72 1.72 1.72 1.72 1.72 3.60
Table: Reconstruction errors (×10−2).
36 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Results
Real data
Estimated abundances
I Similar reconstruction errors
I Model assessment?
37 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Generalised RCA
Results
Real data
Nonlinearity analysis
38 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Outline
Kernel-based mixing models
Residual Component AnalysisBayesian modelResults
Generalised RCABayesian modelResults
Conclusion
39 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Conclusion
Residual component analysis model
I RCA: flexible nonlinear model able to approximate differentmixing models
I Joint nonlinearity detection and abundance estimation
I Spatially correlated nonlinearitiesI Piece-wise constant: Potts model (finite number of classes)I Smoothness: GMRF (more flexible)
I Image segmentation based on nonlinearity levelsI RCA: via Potts modelI G-RCA: Bayesian hypothesis testing
I Estimation of the regularisation parameters (Potts, GMRF)
I GRCA+: positive nonlinear interactions
40 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Conclusion and future work
On the nonlinearity characterization
I Nonlinearity prior based on polynomial kernel KM well adaptedfrom polynomial mixtures
I The GP/prior could also depend on the abundance vectors
φn|M, zn = k, s2k,an ∼ N(0L, s
2kKM,a
)I The GP/prior covariance could also be directly learned from the
data
Unsupervised unmixing
I Extension of the proposed Bayesian model to estimate/correctthe endmember signatures
41 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Conclusion and future work
On the Endmember variability
I Bundles of possible spectral signatures
I Endmember distribution: e.g., Gaussian9 or Beta prior10
I Machine learning approaches (GPs, volatility models)
9O.Eches et al., “Bayesian Estimation of Linear Mixtures Using the Normal Compositional Model.
Application to Hyperspectral Imagery,” IEEE Trans. Image Processing, 2010.10
X. Du et al., “ Spatial and Spectral Unmixing Using the Beta Compositional Model ,” J. of Sel.Topics in Applied Earth Observ. and Rem. Sens., 2014.
42 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Thank you for your attention...
43 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Residual Component Analysis ofHyperspectral Images for Joint NonlinearUnmixing and Nonlinearity Detection
Steve McLaughlin(1)
Joint work with Y. Altmann(1), N. Dobigeon(2), J.-Y. Tourneret(2) and M.Pereyra(3)
(1)Heriot-Watt University – School of Engineering of Physical Sciences, U.K.(2)University of Toulouse – IRIT/INP-ENSEEIHT/Tı¿ 1
2 SA, France(2)University of Bristol – School of Mathematics,U.K.
MAHI 2014 workshop, Nice, 15/12/2014
44 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Synthetic data
Data set
I 60× 60 pixels image composed of R = 3 endmembersI 4 mixing models ⇒ K = 4 levels of nonlinearity
I LMMI Generalized bilinear model (GBM)I Polynomial post-nonlinear mixing model (PPNMM)I RCA model (s2 = 0.1)
I β = 1.6, SNR ' 30dB
Actual label map. log(‖φn‖2
). Detection map.
45 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Unmixing performance
Abundance estimation
RNMSEk =
√1
NkR
∑n∈Ik
‖an − an‖2
I an: nth estimated abundance vector
Unmixing algo.Class #0 Class #1 Class #2 Class #3
(LMM) (GBM) (PPNMM) (RCA)
FCLS 0.35 9.02 20.45 29.23
GBM 0.36 3.29 15.51 28.16
PPNMM 0.66 1.38 0.47 24.03
K-HYPE 3.36 3.26 3.13 3.57
RCA 0.44 1.55 2.13 3.52
Table: Abundance RNMSEs (×10−2).
46 /44
RCA of Hyperspectral Images for Joint Nonlinear Unmixing and Nonlinearity Detection
Conclusion
Unmixing performance
Reconstruction
REk =
√1
NkL
∑n∈Ik
‖yn − yn‖2
I yn: nth reconstructed pixel
Unmixing algo.Class #0 Class #1 Class #2 Class #3
(LMM) (GBM) (PPNMM) (RCA)
FCLS 0.99 2.17 1.33 3.10
GBM 1.00 1.15 4.38 11.94
PPNMM 1.00 1.01 0.99 3.79
K-HYPE 0.98 0.98 0.98 0.98
RCA-SU 1.00 0.99 0.98 0.98
Table: REs (×10−2).
47 /44