MultispectralMultispectral imaging : imaging : How many sensors do we need?How many sensors do we need?

Journal of Imaging Science and TechnologyJournal of Imaging Science and TechnologyVolume 50, Number 2, March and April 2006Volume 50, Number 2, March and April 2006

David David ConnahConnah, Ali , Ali AlsamAlsam and Jon Y. and Jon Y. HardebergHardeberg

School of Electrical Engineering and Computer Science School of Electrical Engineering and Computer Science KyungpookKyungpook National Univ.National Univ.

22 /25/25

AbstractAbstract

Purpose– Reconstruction of reflectance

• Using as few sensors as possible

– Find minimum number of sensors• Minimizing reconstruction error

Proposed method and procedure– Deriving different numbers of optimized sensors

• Transforming characteristic vectors of reflectance

– Simulating reflectance recovery in the presence of noise

33 /25/25

IntroductionIntroduction

Need of many sensors– Metamerism

• Spectrally different surface reflectanceSame camera response

– Reduction of metamerismBy increasing the number of channels in the

device

44 /25/25

Need of minimum number of sensors– Multispectral camera

• Cameras with more than three color channels• Restriction

– Increased cost – Increased memory requirements – Manufacturing limitations

– Need of minimum number of sensors in a multispectral imaging device

55 /25/25

Several studies– Discrepancies between several studies

• Adequate number of sensors : 3~17• Usage of different thresholds for the required similarity

between the original and reconstructed data

– Improvement of drawbacks in several studies• Basis functions physically not feasible sensors

Derivation of physically feasible sensors that optimisedto record and reproduce the spectral data

• Assumption of noise-free in dataQuantization and shot noise

66 /25/25

BackgroundBackground

Response of a digital camera

Sampling at discrete wavelength intervals– From 400 to 700 nm at 10 nm intervals,

λλλλ= ∫λ dREQqii )()()(

where : the response of the ith sensor (i=1, . . . ,P): the ith sensor response function: the spectral power distribution of the illuminant: the surface spectral reflectance function

iq)(λiQ)(λE)(λR

n

∑λ

λ∆λλλ= )()()( REQq ii

31=n

(1)

(2)

77 /25/25

Matrix-vector notation

– System of linear equations in unknowns– For an exact solution

• Number of knowns = number of unknowns=31

where : vector of sensor responses : reflectance vector: sensor matrix (product of and )

q

rQq T=)1( ×p

r )1( ×nQ )( qn×)(λiQ )(λE

np

p

(3)

88 /25/25

Possible to small number of sensors– Reflectance as the weighted linear sum

– Matrix vector notation

∑=

ω=m

iiib

1r (4)

where : the basis vectors: the respective weights

ibiω

nm pp

Bωr = (5)

where columns of B : the basis vectors

99 /25/25

– Replacing Eq. (5) into Eq. (3)

• To solve , setting sufficient

– Solving for ωqBQω 1)( −= T

BωQq T=ω mp =

(6)

(7)

1010 /25/25

Singular value decomposition

– Characteristic vectors• Columns of eigenvectors of the matrix • A set of basis vectors for the columns of • Increasing the number of characteristic vectors

– Improvement of the reflectance estimate

TVUΣR =where : a set of k reflectance spectra (nXk)

and : both orthonormal, i.e.,: diagonal contains the singular values of with zeros elsewhereΣ

U V IVVUU TT ==R

U TRRR

(8)

R

1111 /25/25

Find of a suitable point – with very small improvements in accuracy– Arbitrary threshold in improvement

Choice of sensors for optimised spectral recovery– Choice of as first m characteristic vectors

m

Q

1212 /25/25

NonNon--negative sensorsnegative sensors

Choice and transform of the sensors

– Existence of negative values in characteristic vectors– Real sensors non-negative– Transform into a non-negative vector space– Concentration in distinct regions of visible spectrum

BAQ =

where A : a linear transformation

(9)

1313 /25/25

Fig. 1. The first four characteristic vectorsof the Munsell reflectance data. The data are plotted at 10 nm intervals and interpolated linearly.

Fig. 2. The first four characteristic vectorsrotated by the varimax algorithm to be maximally positive.

1414 /25/25

Using varimax rotation– Varimax criterion

Columnwise variances of the squared elements of B• Optimal transform A

– Orthogonal rotation of B that maximizes the varimax criterion among all other orthogonal rotations

– Non-negative sensors that are as close as possible to the rotated sensors

])1(1[)( 224∑ ∑∑ −=k j

jkj

jk bn

bn

BV

0ˆˆˆmin2

≥− iiiiiggQtosubjectgQQ

i

Q̂

where : vector

: th column ofig 1×p

(10)

(11)

iQ̂ Q̂i

1515 /25/25

Fig. 3. Non-negative sensors formed by varimaxrotation with added positivity constraint.

Fig. 4. Non-negative sensors formed by Piché’s procedure.

1616 /25/25

Procedure

rQq T=

Bωr =

qBQω 1)( −= T

BAQ =

TVUΣR =

1717 /25/25

Using Piché– Generation of non-negative combinations of the

characteristic vectors – Maximization of mutual orthogonality of the sensors

• Measure by the ‘condition number’ of Q

– Determination solely by the condition number of A• Generation of transformations of the characteristic vectors

– Minimization of the condition number of A» Using iterative nonlinear optimization method

22)( †QQQcond =

where : norm of a matrix, which is given by its largest singular value† : pseudoinverse operation

2•

0min22 ≥= BAQtosubjectAA

A

†

(12)

(13)

minmax

1σ

×σ

1818 /25/25

MethodMethod

Synthetic camera responses

– Shot noise• Occurrence from inherent uncertainty in the generation,

reflection, and capture of light

quantshotT nnrQq ++=

where : shot noise

: quantization noiseshotnquantn

Tppshot qqqn ],,,[ 2211 ξξξ= L

where : pseudorandom variable taken from a Gaussian distribution with zero mean and variable standard deviation : sensor response

iξ

iq

(14)

(15)

1919 /25/25

– Quantization noise• Quantization of the simulated responses

Difference between original and estimated spectra

nrrrrrms

T )ˆ()ˆ( −−= (16)

where : reflectance estimate

: original

r̂r

2020 /25/25

Simulation resultsSimulation results

Simulation results

Fig. 6. Results for the Munsell eflectancedata with different levels of shot noise.

Fig. 5. Effect of increasing sensor numberwith 12 bit quantization and 1% shot noise on Munsell reflectance data.

2121 /25/25Fig. 7. Sensors derived by varimax rotation for four different reflectance datasets.

2222 /25/25

– Minimum number of sensors according to dataset• Munsell reflectance data: 11• Macbeth ColorChecker DC data : 13• Esser target : 13• Natural reflectance data : 17

Fig. 8. Effect of increasing sensor number with 12 bit quantization and 1% shot noise on Macbeth ColorChecker DC data.

2323 /25/25

Fig. 10. Natural reflectance data.Fig. 9. Esser target.

2424 /25/25

DiscussionDiscussion

Dependence of minimum number of sensors– Data set– Noise model and level

Not optimal in novel surfaces

Minimum number of sensors– Larger than theoretical case

2525 /25/25

ConclusionConclusion

Propose method– Generation of sensors for a multispectral imaging

device • Optimization for recovering reflectances • Maximally robust to noise

– Find the minimum number of sensors • Using generated sensors• Provision of minimal reconstruction error

Results– Dependence of data set and noise level