Multisampling Compressive Video Spectroscopy

Daniel S. Jeon Inchang Choi Min H. KimKorea Advanced Institute of Science and Technology (KAIST)

Fake or Real?

2

0

0.1

0.2

0.3

0.4

0.5

400 450 500 550 600 650 700

Radianceofpeppers

FakePepper

RealPepper

0

0.1

0.2

0.3

0.4

0.5

400 450 500 550 600 650 700

Radianceofapples

FakeApple

RealApple

Fake Real

[Yasum

aetal.20

08]

Hyperspectral Imaging

RGBImaging MultispectralImaging

HyperspectralImaging

3

3channels <~30channels ≥~30channels

Spectroscopy Imaging

4

Bandpass filter LCTF(liquidcrystaltunablefilter)

Pushbroom CASSI

[Mansouri etal.2007]

[Brusco etal.2006]

[Attas etal.2003]

[Wagadarikar etal.2008]

Multisampling CASSI

5

Maskshiftingusing piezotranslationstage[Kittleetal.2010]

DMD(digital-micromirror-device)

LCoS (liquidcrystalonsilicon)

[Linetal.2014]

[Wuetal.2011]

MultisamplingCASSIsystemsrequiremultiplecaptures

Goal

6

Singlecodedinput Hyperspectralvideo

• Multisamplingcompressiveimagingà Highspectralresolutionà Highspatialresolution

• Singlesnapshothyperspectralimagingà Videospectroscopy

Key Idea

8

• Codedaperturesnapshotspectralcamera

• Multisamplingè Kaleidoscope

System Setup

objective lensimage multiplier relay lens collimating lens relay lenscoded

aperturedetector

prism

9

diffuser

Examine Kaleidoscope

10

View Multiplication

11

1/ 1

bma a f

= =-

1 1 1a b f+ =a

f

b

al bl

: maginificationm

Lightdirection

Diffuser

Detector

Effect of Diffuser

12

Withdiffuser Withoutdiffuser

Raw Input Video

13

Geometric Calibration - Homography

1 2

3 4

2 1

4 3

2 1

4 3

3 4

1 2

3 4

1 2

4 3

2 1

4 3

2 1

4 3

2 1

4 3

2 1

14

Geometric Calibration - Optical Flow

15

Beforeapplyopticalflow Afterapplyopticalflow

Animated5views

Geometric Calibration - Dispersion Direction

Capturedimages Alignedimages

16

Dispersiondirectionflipped

Image Reconstruction

17

( ), ,f x y l

( )0 , ,kf x y l ( ),kT x y ( )2 , ,kf x y l( )1 , ,kf x y l

Prism

( ),kg x yA set of

first-surfacemirrors

⊙

⊙

⊙

( ), ,h x y l

0( , ) ( ( ), , , , ) ( , ) ( , , )k k k kg x y h x x y y T x y f x y dx dy df l l l lL

¢ ¢ ¢ ¢= -ò òòDispersion IncidentMask

Coded Aperture

18

Codedaperturespecs• Randombinarypatterns• correspondstotwo-by-twopixels

,( , ) rect ,k ijk

i j

x yT x y i jæ ö= - -ç ÷D Dè øåT

1pxonsensor

Each9viewpassthroughdifferentcodedaperturepatternsà Enablemultisampling

Prism Dispersion

19

0

10

20

30

40

50

60

450 550 650 750

Pixelshift[p

x]

Wavelength[nm]

Dispersioncalibration

2 1( , , ) ( ( ), , , , ) ( , , )k k kf x y h x x y y f x y dx dyl f l l l¢ ¢ ¢ ¢= -òòDispersion

500nm

600nm

700nm

Dispersedlight Codedlight

Reconstruction

20

• Minimizinganobjectivefunctionwithtotalvariation[Bioucas-DiasandFigueiredo 2007]

2

2Detectorinput

(3D)Lightmodulation

(4D)Hyperspectal

Image(3D)

RegularizingSparsity(3D)

View Multiplication

21

1 view 2 views 3 views

reference9 views5 views(syntheticimages)

Dispersion Direction

22

PSNR:28.20SSIM:0.88

PSNR:30.45SSIM:0.91

5viewswithoutdispersioninversion

5viewswithdispersioninversion

(syntheticimages)

reference

Multiview Tradeoff

23

1(full)

PSNR:27.84SSIM:0.88

1 2 34 5 67 8 9

PSNR:23.42SSIM:0.77

1 2 34 5 67 8 9

PSNR:31.29SSIM:0.92

(syntheticimages)

reference

Comparison

24

0.00.10.20.30.40.50.60.70.80.91.0

450 500 550 600 650

refle

ctan

ce

wavelength [nm]

red patchReference

CASSI

Ours

TraditionalCASSI

OurmultisamplingCASSI

5views

1fullview

Results

25

Input Reconstructedhyperspectral video

sRGB video Wavelengthat600nm

Results

26

Input Reconstructedhyperspectral video

sRGB video Wavelengthat600nm

Results

27

Input Reconstructedhyperspectral video

sRGB video Wavelengthat600nm

Results

28

Input Reconstructedhyperspectral video

sRGB video Wavelengthat600nm

Discussion

• Tradeoff between spatial and spectral resolution

– Significantly enhance spectral resolution

– Sacrifice sensor resolution

• Misalignment of copied views gives a critical

reconstruction problem

• Alternatives for TV-L1 optimization

29

Conclusion

30

• Singlesnapshot-baseddesign

• Hyperspectral videoacquisition

• Highspectralresolution

• Bycouplingmultisamplingandcompressiveimaging

Acknowledgements

• Korea National Research Foundation (NRF) grants

(2013R1A1A1010165 and 2013-M3A6A6073718)

• Korea ICT R&D program of MSIP/IITP (10041313)

31

Thankyou