ENHANCED DETECTION OF CELL PATHS IN SPATIOTEMPORAL PLOTS FOR NONINVASIVE MICROSCOPY OF THE HUMAN RETINA

Johnny Tam1 and Austin Roorda1,2

1Joint Graduate Group in Bioengineering, University of California, Berkeley, and University of

California, San Francisco 2School of Optometry, University of California, Berkeley

ABSTRACT

Spatiotemporal (ST) plots have been used for studying cell motion in the microcirculation. However, ST plots are typically applied to invasive imaging methods. Noninvasive video microscopy of the human retinal capillaries can be performed using an Adaptive Optics Scanning Laser Ophthalmoscope, but direct implementation of ST plots is difficult due to low contrast and high noise. We introduce motion contrast enhancement to enhance detection of cell paths, and enable ST plot analysis. Using features generated by the motion contrast enhancement, a method for automatic extraction of cell paths was developed. Our results show that motion contrast is an important precursor step to ST plot analysis for videos with low signal to noise ratios.

Index Terms— Spatiotemporal plots, motion contrast, tracking, optical microscopy, retinal capillaries

1. INTRODUCTION Spatiotemporal (ST) plots (also known as kymograms) are used to visualize motion in many custom video microscopy systems. Some examples include visualization of intracellular structures, such as vesicles [5] and microtubules [7]; and visualization of blood cells through small vessels, such as in the hamster muscle [1], rat mesentery [6], and cat retina [3]. This paper addresses the second application. In order to achieve good resolution and contrast, invasive procedures are commonly used. We use an Adaptive Optics Scanning Laser Ophthalmoscope (AOSLO) to image the human retina at the cellular level [11]. Although leukocytes (WBCs) can be resolved using AOSLO [4], the spatial contrast between WBCs and their surrounding tissue is very low (see Fig. 1). This is largely because we acquire images both noninvasively and without contrast agents.

ST plots are a representation of intensity fluctuations in space and time through a given vessel. In most vessels, complete information about individual cells in a video requires 4 dimensions: 3 spatial, and 1 temporal. ST plots reduce these to 2 dimensions without loss of information

Fig. 1. Retinal capillary and background photoreceptors imaged using noninvasive in vivo microscopy in a human subject. Two consecutive frames are shown. A single white blood cell, which appears to blend into the background, is circled. The scale bar is 100 m.

about individual cells when (i) the size of the vessels is small enough to feature single-file flow of cells, and (ii) the spatial location of the vessel can be determined. The intensities of the one dimensional line along the vessel centerline are plotted over time (see Fig. 2). Thus, ST plots provide a compact way to represent the movement of individual cells through small capillaries.

The capillaries in the retina have been characterized using histology studies in monkeys [8]. Retinal capillaries are planar and single-layered in the location immediately outside of the fovea, and are small enough to feature single-file flow. Thus, they are good candidates for ST plot analysis. However, the method assumes that there is prior information about the location of the vessels, which is not immediately obvious in AOSLO videos.

The main contribution of this paper is to describe a method to enhance ST plots that suffer from low signal to noise ratios (SNR), based on motion contrast. We implement ST plots for AOSLO videos and illustrate automated extraction of cell paths.

2. METHODS 2.1. Motion contrast enhancement Motion contrast can occur independently from spatial contrast, as in the case where the texture of the moving object is similar or identical to the texture of the background. WBCs in retinal capillaries imaged using AOSLO appear to blend into the background when viewing

584978-1-4244-4126-6/10/$25.00 ©2010 IEEE ISBI 2010

Fig. 2. Schematic showing ST plot generation. Two consecutive frames (j=1,2) are shown with a schematic black vessel and two white cells. To generate the ST plot, the intensities along the vessel are plotted vs. the frame number (s vs. j). A piece of an actual ST plot is shown below the schematic ST plot (rescaled for display due to high spatial sampling compared to temporal sampling). Purely horizontal and vertical lines are imaging artifacts. a single frame, but become distinguishable across multiple frames. Our assumptions are (i) WBCs move while the background tissue remains still, and (ii) brightness constancy of features in the video is satisfied locally across time. The goal is to use motion to separate cells from the background.

For a pair of frames j and j+1, with intensities Ij(x,y) and Ij+1(x,y), the difference image, Dj(x,y) = Ij(x,y) - Ij+1(x,y), identifies regions within an image that have changed intensities. Let Bj = {(x,y) | Dj(x,y) = 0}. Bj represents the set of coordinates corresponding to the stationary background tissue in frame j, and Bj

C represents the set of coordinates corresponding to moving cells in frames j and j+1, so long as the assumptions are satisfied. Thus, locations

of the vessels can be computed as N

j

CjB

1

, where N is the

number of frames. AOSLO retinal imaging introduces challenges that

contribute to the breakdown of some of the assumptions used for difference images. First, involuntary eye movements cause the entire frame to move around during the imaging session. In other words, both the background tissue and the cells move, relative to the imaging camera. Nonetheless, individual frames can be registered to a single reference frame, taking into consideration intraframe distortions, as previously described [10]. Even after good registration, there are still intensity fluctuations in the background tissue, challenging the notion of brightness constancy. In addition, the texture of moving cells is identical to that of the background. Thus, the SNR of the WBCs is extremely low. We have previously described an

improved method for motion contrast enhancement using multi-frame division videos, which allows for accurate vessel centerline localization using a standard deviation image [9] – a prerequisite for implementation of ST plots.

Multi-frame division videos improve the SNR through signal averaging. Consider three frames from a video with noise, Ij, Ij+1, and Ij+2. The average of consecutive difference images, (Dj + Dj+1) / 2 yields the expression Ij – Ij+2, which is not an improvement over the single frame difference image. However, using division images, defined as Dj = Ij / Ij+1, we can define a multi-frame division image as Mj = (Dj + Dj+1) / 2. The corresponding background is defined as Bj = {(x,y) | Dj(x,y) = Dj+1(x,y) = 1+ }, where can be adjusted depending on the level of the noise. Suppose each image Ij has a noise component proportional to the signal intensity,

jIj, such that Ij = Ij’ + jIj’, where Ij’ denotes the noise-free component of image intensities. Let Dj’ = Ij’ / Ij+1’ and Mj’ = (Dj’ + Dj+1’) / 2, which are the ideal responses of the single and multi-frame division images in the absence of noise. For (x,y) Bj, let Nj = Dj – Dj’, which is the deviation from the ideal response, and the noise component of the single-frame division image. For the multi-frame division images, Mj – Mj’ = (Nj + Nj+1) / 2. This shows that the noise is averaged in the background.

For (x,y) BjC, which are locations of moving cells, an

effect that is beneficial to ST plot analysis occurs. A moving object in frame j manifests in Mj as both a bright and a dark object: the bright object at the location of the object in frame j+1, and the dark object at the location of the object in frame j. When viewed on the ST plot, this provides local motion contrast by “framing” white cell paths with a black border. Thus, the motion contrast analysis generates a unique pattern on the ST plot due to the incorporation of multiple frames, which is beneficial for automated analysis.

2.2. Background suppression To further minimize the effects due to noise, we consider the properties of the cell paths in the ST plots. The goal is to enhance cell paths on the ST plot while suppressing other regions. We assume that (i) cell paths are one dimensional and monotonic, and (ii) cell paths do not cross each other. These assumptions are valid for the smaller retinal capillaries under consideration.

For background suppression, we use an approach based on Frangi “vesselness” [2], which detects tubular structures. Motion contrast enhanced cell paths satisfy “vesselness” due to their appearance (white lines with black borders). Locally, the elements of the Hessian matrix are given by

jiij xx

LH

2

, where L is a local image region. The

eigenvalues of H, 1 and 2, are ordered such that | 1| | 2|. Regions of “vesselness” are identified where | 1| 0 and | 1| << | 2|.

585

Fig. 3. ST plots are shown after each of the intermediate steps proposed for automated cell path extraction. Each pair of two columns is one continuous ST plot, split into two columns: (A) unprocessed; (B) after motion contrast enhancement; (C) after background suppression; (D) after three-step thresholding; (E) after morphological opening; (F) extracted cell paths overlaid in blue on the motion contrast enhanced result. 2.3. Segmentation and extraction Three-step thresholding is used to segment the cell paths, an extension of a previously proposed two-step thresholding method [6]. The assumptions are that (i) the cell paths are sparse (i.e. nonuniformly distributed), (ii) one dimensional, and (iii) monotonic.

The goal of the segmentation step is to identify the brightest cell paths (i.e. the cells that are most visible on the raw video). Segmentation is performed using thresholding. Selection of the threshold parameter presents a tradeoff. On

Fig. 4. Locations on the ST plot where automated extraction failed. 3 of the 8 locations show portions of extracted cell paths (blue lines) that happened to be adjacent to the problem locations.

the one hand, a high threshold value will exclude undesired image regions, but may also exclude portions of desired image regions (clipping), resulting in false negatives. On the other hand, a low threshold value will include desired image regions, but may also include undesired image regions, resulting in false positives. Multi-step thresholding addresses this tradeoff. High, medium, and low threshold values are applied to the ST plot image, and stored as ST_h, ST_m, and ST_l, respectively. ST_l contains both desired and undesired image regions that are not present in ST_h. The idea is to include the desired image regions of ST_l and exclude the undesired regions of ST_l, using both ST_m and ST_h. First, ST_m is compared against ST_h. Image points (x,y) ST_m are kept if they are connected to an identified region of ST_h, and deleted if they are not connected to any identified regions of ST_h. The result is compared against ST_l, and the procedure is repeated. Thus, three-step thresholding minimizes the number of false positives while reducing clipping of desired image regions.

Morphological opening is performed to connect pieces of the cell paths together. This step is important for low SNR ST plots, since cell paths are not well connected. A line structural element oriented horizontally is used, because the orientations of cell paths are approximately horizontal in pixel space when viewed locally (note that cell paths do not appear to be horizontal due to rescaling for display purposes; the aspect ratio for display purposes is 1:20).

Least-squares polynomial fits are used for extraction of segmented cell paths. Prior to extraction, the following conditions must be met: first, the length of the cell path must be sufficiently long; second, the slope of the cell path must be sufficiently different from horizontal. We also require that the fit results be monotonic.

3. RESULTS

We tested the algorithms on retinal videos, which were

collected on separate days from two subjects with no systemic or ocular disease, using an AOSLO. For the first data set, 40-second videos were acquired at 30 Hz using a 532 nm wavelength laser; for the alternate data set, 40 second videos were acquired at 60 Hz using an 840 nm wavelength laser. Videos were preprocessed offline. The steps in preprocessing were (i) correction for frame distortions [10], (ii) registration of frames to a single reference frame [10], (iii) cropping registered videos to eliminate inconsistent boundaries, and (iv) a median filter to reduce the effects of noise. We used the first dataset to

586

Fig. 5. Automated extraction results on the alternate data set. From left to right: capillary from unprocessed video; identified capillary using motion contrast enhancement, with vessel centerline shown in blue used for ST plot generation; ST plot split into 4 sections for display purposes; one false positive, shown in blue (A), three false negatives (B), and three examples of correctly identified cell paths, shown using blue lines (C).

calibrate the analysis parameters, and repeated the analysis on the alternate dataset using the same parameters. For the multi-frame division videos, instabilities in division were handled by trimming responses to lie within the bit depth of the images: [0, 255]. An example of division instability can be seen as black and white horizontal lines on the ST plots, which were due to the subject blinking during the imaging session, resulting in frames with near-zero intensity.

We performed manual cell path extraction for comparison with the automatic method. Manual extraction was performed using a graphical user interface, asking the user to select points along each cell path. The extracted cell paths were assumed to be ground truth. For the first dataset, the automatic approach correctly identified 42 out of 50 cell paths (see Figs. 3, 4). The false negatives were all cell paths that were not distinct enough, possibly due to (i) proximity to a neighboring cell path, violating the assumption of sparseness, or (ii) lack of connectivity among segmented image regions. For the alternate dataset, the automatic approach correctly identified 26 out of 29 cell paths, and incorrectly identified 1 cell path. The incorrectly identified cell path was due to a cell path that was too close to an imaging artifact. The alternate dataset had more noise compared to the first dataset, and also more imaging artifacts (see Fig. 5). Overall, the automatic method correctly identified 86% of all the cell paths in low SNR datasets cluttered with imaging artifacts.

4. CONCLUSIONS Motion contrast enhancement is an effective tool for detection of cell paths in ST plots, and an important precursor to automated analysis. We demonstrate that ST plot analysis can be successfully applied to high resolution retinal videos of the microcirculation, despite challenges such as low SNR, and similarities in texture between moving cells and the background tissue. The method is most promising for noninvasive, in vivo assessment of hemodynamics through small capillaries, where contrast and noise issues are prevalent.

5. ACKNOWLEDGEMENTS

The authors thank Scott Stevenson for suggesting ST plots. This work was supported by NIH EY014375; NSF AST-9876783; DoD NDSEG Fellowship; and NSF GRFP.

6. REFERENCES

[1] C.G. Ellis, M.L. Ellsworth, R.N. Pittman, and W.L. Burgess, “Application of Image Analysis for Evaluation of Red Blood Cell Dynamics in Capillaries,” Microvascular Res, vol. 44, pp. 214-225, 1992. [2] A.F. Frangi, W.J Niessen, K.L. Vincken, and M.A. Viergever, “Multiscale Vessel Enhancement Filtering,” MICCAI, 1998. [3] P.S. Jensen and M.R. Glucksberg, “Regional Variation in Capillary Hemodynamics in the Cat Retina,” Invest. Opthalmol. Vis. Sci., vol. 39, pp. 407-415, 1998. [4] J. A. Martin and A. Roorda, “Direct and Noninvasive Assessment of Parafoveal Capillary Leukocyte Velocity,” Ophthalmol., vol. 112, pp. 2219-2224, 2005. [5] V. Racine, M. Saschse, J. Salamero, V. Fraisier, A. Trubuil, and J.B. Sibarita, “Visualization and quantification of vesicle trafficking on a three-dimensional cytoskeleton network in living cells,” J. Microscopy, vol. 225, pp. 214–228, 2007. [6] Y. Sato, J. Chen, S. Yamamoto, S. Tamura, N. Harada, T. Shiga, S. Harino, and Y. Oshima, “Measuring Microcirculation Using Spatiotemporal Image Analysis,” MICCAI, 1995. [7] I. Smal, I. Grigoriev, A. Akhmanova, W.J. Niessen, and E. Meijering, "Accurate estimation of microtubule dynamics using kymographs and variable-rate particle filters", Conf Proc IEEE Eng Med Biol Soc., vol. 1, pp. 1012-1015, 2009. [8] D.M. Snodderly and R.S. Weinhaus, “Retinal Vasculature of the Fovea of the Squirrel Monkey, Saimiri sciureus: Three-Dimensional Architecture, Visual Screening, and Relationships to the Neuronal Layers,” J Comparative Neurology, vol. 297, pp. 145-163, 1990. [9] J. Tam, J.A. Martin, and A. Roorda, “Noninvasive visualization and analysis of parafoveal capillaries, Invest. Opthalmol. Vis. Sci. doi: 10.1167/iovs.09-4483 (in press). [10] C.R. Vogel, D.W. Arathorn, A.Roorda, and A. Parker, “Retinal motion estimation in adaptive optics scanning laser ophthalmoscopy,” Optics Express, vol. 14, pp. 487-497, 2006. [11] Y. Zhang, S. Poonja, and A. Roorda, “MEMS-based adaptive optics scanning laser ophthalmoscopy,” Optics Letters, vol. 31, pp. 1268-1270, 2006.

587