Medical imaging with multi-tap CMOS image sensors

Keiichiro Kagawa, Keita Yasutomi, Shoji Kawahito

Research Institute of Electronics, Shizuoka University

3-5-1 Johoku, Hamamatsu, 432-8011 Japan

E-mail: kagawa@idl.rie.shizuoka.ac.jp

Abstract Optical tissue imaging based on a multi-tap CMOS image sensor with lateral electric field charge modulator

(LEFM) is shown. The imaging schemes are categorized into three types: 1) highly-time-resolved imaging, 2) time-

division-multiplexed imaging, and 3) coded shutter imaging. In type-1, by taking advantage of sub-nano-second time

resolution, fluorescence lifetime imaging, and time-resolved spectroscopy are explored. In type-2, the multi-tap CMOS

image sensor is combined with synchronized illuminations to perform time division multiplexed imaging. Multi-

spectral imaging and spatial frequency domain tissue imaging are studied. Type-3 is applied to blood flow imaging.

Keywords: Multi-tap CMOS image sensor, optical tissue imaging

1. Multi-tap CMOS image sensor based on lateral

electric field charge modulator (LEFM)

Multi-tap image sensor pixels are equipped with a single

photodiode, multiple storage diodes, and a draining path. Our

group proposed a new charge modulator called lateral electric

field charge modulator (LEFM) which is suitable for high-speed

lossless charge transfer and multi-tap implementation[1]. The

number of taps started from 1[2]. Then, it increased to 2[3-4],

4[5-7], and 8[8]. In this talk, biomedical applications of LEFM-

based CMOS image sensors are explained.

2. Highly-time-resolved imaging

The main application of our time-resolving CMOS image

sensors is time-of-flight depth imaging[9]. However, they are

also promising for time-resolved biomedical imaging such as

fluorescence lifetime imaging[2] and time-resolved

spectroscopy like near-infrared spectroscopy (NIRS)[10]. LEFM

is suitable for these biomedical applications because true

correlated double sampling (CDS) enables to detect weak

optical signal from the tissue. For these applications, low-noise

time-resolving CMOS image sensors with sub-nano-second

temporal resolution have been developed[3-4, 6-7].

3. Time-division-multiplexed imaging

Multi-tap CMOS image sensors are suitable for time-division-

multiplexed imaging with switched illuminations or pattern

projection. For example, spatial frequency domain imaging

(SFDI)[11] provides two dimensional maps of tissue absorption

and reduced scattering coefficients by capturing three images for

three different structured illuminations. However, it suffers from

motion artifact and errors by ambient light. When a 4-tap CMOS

image sensor is used, three projected patterns and ambient light

are assigned to each tap, respectively. Thus, motion-artifact-free

SFDI with ambient light suppression is demonstrated[12].

4. Temporal coded shutter

Multi-tap pixels are also effective to bridge the gap between

the bandwidths of tissue optical parameters and optical signals.

For example, in multi-exposure laser speckle contrast imaging,

which provides a blood flow velocity map, a high-speed image

sensor operating at 1kfps is required because the speckle pattern

changes at such a high frequency[13]. However, blood flow

velocity itself changes at much lower frequency. For efficient

lower frame rate imaging, application of temporal coded shutters

to multi-tap CMOS image sensors is explored[14].

References [1] S. Kawahito, G. Baek, Z. Li, S. Han, M. Seo, K. Yasutomi, and K. Kagawa,

“CMOS lock-in pixel image sensors with lateral electric field control for

time-resolved imaging,” Int’l Image Sensor Workshop, pp. 1417-1429

(2013).

[2] Z. Li, S. Kawahito, K. Yasutomi, K. Kagawa, J. Ukon, M. Hashimoto, and

H. Niioka, “A time-resolved CMOS image sensor with draining-only

modulation pixels for fluorescence lifetime imaging,” IEEE Trans. Electron

Devices, Vol.59, No. 10, pp.2715-2722 (2012).

[3] M. –W. Seo, K. Kagawa, K. Yasutomi, T. Takasawa, Y. Kawata, N.

Teranishi, Z. Li, I. A. Halin, S. Kawahito, “A 10.8ps-time-resolution

256x512 image sensor with 2-tap true-CDS lock-in pixels for fluorescence

lifetime imaging,” ISSCC Dig. Tech. Papers, pp. 189-199 (2015).

[4] M. –W. Seo, K. Kagawa, K. Yasutomi, Y. Kawata, N. Teranishi, Z. Li, I.

Halin, and S. Kawahito, “A 10ps time-resolution CMOS image sensor with

two-tap true-CDS lock-in pixels for fluorescence lifetime imaging,” IEEE J

Solid-State Circuits 51, pp. 141-154 (2016).

[5] T. Kasugai, S. –M. Han, H. Trang, S. Aoyama, K. Yasutomi, K. Kagawa,

and S. Kawahito, “A time-of-flight CMOS range image sensor using 4-tap

output pixels with lateral-electric-field control,” in Proc. of IS&T Int’l

Symp. Electronic 2016, ImagingIMS-048.1 (2016).

[6] M. -W. Seo, Y. Shirakawa, Y. Masuda, Y. Kawata, K. Kagawa, K. Yasutomi,

S. Kawahito, “A programmable sub-nanosecond time-gated 4-tap lock-in

pixel CMOS image sensor for real-time fluorescence lifetime imaging

microscopy,” ISSCC Dig. Tech. Papers, pp. 70-71 (2017).

[7] M. -W. Seo, Y. Shirakawa, Y. Masuda, Y. Kawata, K. Kagawa, K. Yasutomi,

S. Kawahito, “A programmable sub-nanosecond time-gated 4-tap lock-in

pixel CMOS image sensor for real-time fluorescence lifetime imaging

microscopy,” ISSCC Dig. Tech. Papers, pp. 70-71 (2017).

[8] Y. Shirakawa, M-W. Seo, K. Yasutomi, K. Kagawa, N. Teranishi, S.

Kawahito, “Design of an 8-tap CMOS lock-in pixel with lateral electric

field charge modulator for highly time-resolved imaging, ” Photonics West

2017, Proc. SPIE 10108, Silicon Photonics XII, 101080N (2017).

[9] S-M. Han, T. Takasawa, T. Akahori, K. Yasutomi, K. Kagawa, and S.

Kawahito, “A 413×240-Pixel Sub-Centimeter Resolution Time-of-Flight

CMOS Image Sensor with In-Pixel Background Canceling Using Lateral-

Electric-Field Charge Modulators,” ISSCC Dig. Tech. Papers, San

Francisco, pp. 130-131 (2014).

[10] Z. Liu, D-X. Lioe, M-W. Seo, M. Niwayama, M. Hakamata, K. Kagawa, K.

Yasutomi, Y. Fukushi, S.Yamamoto, S. Kawahito, “A time-resolved NIRS

experiment using a CMOS lock-in pixel image sensor with highly time-

resolving capability”, The 39th Annual International Conference of the

IEEE Engineering in Medicine and Biology Society (EMBC’17) (2017).

[11] D. Cuccia, F. Bevilacqua, A. Durkin, F. Ayers, and B. Tromberg,

“Quantitation and mapping of tissue optical properties using modulated

imaging,” J. Biomed. Opt., Vol. 14, 024012 (2009).

[12] Y. Nishioka, K. Kagawa, C. Cao, N. Tsumura, T. Komuro, K. Nakamura, A.

Durkin, B. Tromberg, K. Yasutomi, and S. Kawahito, “Real-time vein

imaging using a 4-tap CMOS image sensor,” in Proc. Optics and Photonics

Japan, 1pA4 (2018, in Japanese).

[13] M. Hultman, I. Fredriksson, M. Larsson, A. Alvandpour, and T. Stromberg,

“A 15.6 frames per second 1-megapixel multiple exposure laser speckle

contrast imaging setup,” J. Biophotonics, e201700069 (2017).

[14] K. Kagawa, K. Yasutomi, and S. Kawahito, “Optical tissue imaging with

multi-tap CMOS image sensors –scattering, fluorescence, blood flow-,” in

Proc. Optics and Photonics Japan, 1aAS4 (2018, in Japanese).

1

Multi-tap CMOS image sensor

T. Kasugai et al., EI2016

Photodiode

Storage-1

Storage-2

Storage-3

Storage-4Signal transfer

control

Opticalsignal

1-frame period (~30ms)

Signal modulationperiod

Readoutperiod

Tap-1

Time4

2

3

4 imagesare put out

at the same time

Benefits of fast lock-in detection• High speed (high frame rate) camera is not necessary

– Cost efficient, less processing capability required• Suppression of motion artifact

– Flowing liquid, moving objects are observable(Of course, μa and μ’s maps become blurry, but no obvious artifacts appears)

• Suppression of ambient light– Camera cam be used under normal light condition

Ambient light→ unexpected bias → suppress

Time-multiplexedillumination

Multi-tap camera

Motion of subject→ motion artifact

→ suppressSynchronization

Motion-artifact-free backgroud-light-tolerant TOF range imaging

Intensity Image

Distance

Frame rate and modulation frequencyWhat is the difference?

• Frame rate– Change of tissue parameters– E.g. Blood flow speed

• Modulation frequency of shutter speed– Bandwidth of optical signal– E.g. Correlation time of speckle

Illumination

1-frame period (~30ms)Charge modulation period

composed of N cycles Readout period

Tap-1

Time

Illumination cycle

Exposure cycle

4

2

3

1 2 3 4 1 2 3 4Time slot

Time-division-multiplexed imaging• Shorter exposure/illumination cycle will enhance isochronism between the

taps, or the same motion blur will appear among the taps • Oversampling and repeated charge accumulation will increase S/N

Multi-spectral imaging

λ1 λ2 λ3

Slot-1

Ambi-entlight

Slot-2 Slot-3 Slot-4

Tap-1image

Tap-2image

Tap-3image

Tap-4image

IlluminationsNo

illumination

Read

Multi-spectral imagesAmbient

image

Photodiode

2

Experimental setup

LED illuminations

Capture

Examples of multi-spectral imaging

730nm

780nm

850nm

Ambient

Functional assignment to the taps in this work

Pattern projection imagesAmbient

image

Tap-1image

Tap-2image

Tap-3image

Tap-4image

Ambient light

Pattern projectionsNo

projection

Slot-1 Slot-2 Slot-3 Slot-4

Read Photodiode

4-tap pixelwith draining

Spatial frequency domain imaging (SFDI)

𝐷𝐶 component 𝑀A𝐶 component 𝑀

Absorption 𝜇Reduced scattering 𝜇′

ImagesensorDMD

Light source

Tissues

MakeLUT

ReferLUT

Sinusoidal pattern

Reflected pattern

Tissues𝜇

𝜇′

DMD

4-tapTOF camera

850nm LEDSpecimen

DMD

Experimental setupTube filled with 0.5% intralipid + indigo ink → water is sucked up,

Example of captured image@23fps, f= 0.1mm−1

1ms/time slot (limited by DMD), 4-times oversampling

Tap-2Tap-1 Tap-3 Tap-4

Frame-248

Frame-290

Frame-399

Ambient imageis visible

3

0.5% intralipid + ink → 0.5% intralipid𝜇 𝜇′log [/mm] [/mm]

Frame190

Frame380

Frame250

Only μa decreased,μ’s was constant

Coded shutter imaging

Tap-1 Tap-2 Tap-3 Tap-3

Blurriness is different, brightness is the same (same photon shot noise)

Tap-

Effective exposure time

2

3

4

Total accumulations are the same1

Still Blurry

Time

Photodiode

Fluorescence lifetime imagingwith phasor plot

Plotted on the linebetween 𝜏1 and 𝜏2

Time

Flu

ore

scen

ce in

tens

ity

0 Re

Im

2-component mixing (τ1 ,τ2)

f(t)

jjand

2

tan,21

1θ

τ1

𝜏20.5

f(t)= 𝑒𝑥𝑝 − + 𝑒𝑥𝑝 −

Phasor plotVisualize multi-component lifetime in the frequency domain

Widefield FLIM

Pulsed laser

L2L1

L3

L4

Time-resolvingcamera

Sample

[ns]10

5

0

Red and blue acrylic plates

Synthesized lifetime

Averagedintensity

Exp-1 results

Re

Im

1

0.5

00 0.5 1

Lifetimes of blue and red fluorescences are identified.They are almost single-component.

10ns 9ns8ns7ns6ns5ns4ns3ns2ns1ns

4

Exp-2 resultsStacked red and blue acrylic plates

[ns]10

5

0

Re

Im

0 0.5 11ns

1

0.5

0

3ns2ns

4ns5ns6ns7ns

8ns10ns

15ns

- Linear distribution shows two components are mixed- But their fundamental components are not obvious(should be on the line)

Synthesized lifetime

Averagedintensity

Limitation of SFDI and proposalSFDI Binary stripe pattern + time resolving

時間遅れ強度減衰

を確認

𝑡

𝐼

𝑡

𝐼 ReflectedInput

Tissues

Pattern Light DepthSFDI Sinusoidal CW ~mm

Time-resolving Binary stripe Impulse ~10mm

生体組織

Deep tissue reflectance isoverwhelmed by the shallow reflection

Tissues

𝑥

𝐼

Deep tissue reflectanceis detectable (no shallow reflection)

Experimental setup

Time-resolvingCMOS image sensor

DMD

BPF

Diffuser

Subject

Analyzer

Supercontinuum(optical fiber output)

Polarizer

ExperimentsSFDI

Time-resolved measurement

- Conventional SFDI method is confirmed- 𝑀 and𝑀 are measured- 2D maps of 𝜇 and 𝜇′ are

made by referring LUT

- Waveforms for simulation andexperiment are compared

- Influence of the 2nd layer phantomto the waveform is measured

10mm

Comparison of simulated and measured Waveforms

Trends are roughly similar(deconvolution is necessary for accurate comparison)

0.0 0.2 0.4 0.6 0.8 1.0 1.2Time [ns]

Nor

mal

ized

inte

nsity

[a.u

.]

1.0

0.5

0.0

Nor

mal

ized

inte

nsity

[a.u

.]

1.0

0.5

0.0

0.0 0.2 0.4 0.6Time [ns]

256×256 pixels

90×8pixels

30m

m

Measured(system response included)

Subject

Simulated(w/o system response)

Comparison of 2-layeredphantoms in the center of dark part

𝜇

＜

10mm

10mm

10mm

10mmPeak position

Rise

Fall

Light

Nor

mal

ized

inte

nsity

Time [ns]- Influence of the bottom phantom was measured(shorter delay, steeper rise and fall due to larger absorption)- Inverse problem solving is necessary to decompose the optical properties of each layer