Color and the Cone Mosaiccolor.psych.upenn.edu/.../Brainard_2015_ColorAndMosaic.pdf · 2015-11-22 · as metamerism). Two physically different spectral irradiances are shown. These

VS01CH21-Brainard ARI 31 October 2015 13:1

Color and the Cone MosaicDavid H. BrainardDepartment of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104;email: [email protected]

Annu. Rev. Vis. Sci. 2015. 1:519–46

First published online as a Review in Advance onOctober 22, 2015

The Annual Review of Vision Science is online atvision.annualreviews.org

This article’s doi:10.1146/annurev-vision-082114-035341

Copyright c© 2015 by Annual Reviews.All rights reserved

Keywords

color vision, physiological optics

Abstract

Vision is limited by the measurements taken by the cone photoreceptors.To provide useful perceptual representations, the brain must go beyond themeasurements and make inferences about the scene being viewed. This arti-cle considers the first stages of spatiochromatic vision. We show how spatialand chromatic information become intertwined by the optics of the eye andbecause of the structure of the retinal cone mosaic, and we consider theconsequent implications for perception. Because there is at most one coneat each retinal location, the standard treatment of human trichromacy doesnot apply at fine spatial scales. Rather, trichromacy results from a perceptualinference based on measurements from cones of different classes at differentlocations. Our treatment emphasizes linking physics, biology, and compu-tation with the goal of providing a framework for a larger understanding ofhow the brain interprets photoreceptor excitations to see objects and theirproperties.

519

Click here to view this article'sonline features:

• Download figures as PPT slides• Navigate linked references• Download citations• Explore related articles• Search keywords

ANNUAL REVIEWS Further

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.

http://www.annualreviews.org/doi/full/10.1146/annurev-vision-082114-035341


INTRODUCTION

Vision provides a coherent perceptual representation of the world around us, based on biologicalmeasurements of the light entering the eyes. Construction of this representation requires thatthe brain go beyond the direct measurements and make inferences about what is present in thescene; the measurements themselves do not uniquely determine the scene configuration (Knill& Richards 1996, von Helmholtz 1896). In this article, we consider one of the clearest cases ofwhy and how the visual system makes such inferences, the first stages of spatiochromatic vision(for related reviews, see Brainard 2009, Carroll et al. 2009, Hofer & Williams 2014). This caseis likely to be particularly informative because we have a solid understanding of the physics andencoding of the relevant light signals. The principles of inference developed for this case outline aframework for using computation to link physics and biology with perception, which in turn mayprovide the foundation for a broader understanding of how the brain interprets photoreceptorexcitations to see objects and their properties.

The photoreceptor basis of human color vision is well established (Brainard & Stockman2010, Wandell 1995, Wyszecki & Stiles 1982). The typical human retina contains three classes ofcone photoreceptors that operate at daytime light levels, and the excitation of these cones yieldsa trichromatic code for the spectrum of the incident light (Figure 1). This trichromatic codelimits what information about the spectra is available to the visual system, and our quantitativeunderstanding of trichromacy both simplifies the study of postreceptoral color processing (becausewe need only consider the information captured by the cones) and allows for economical colorreproduction technologies (which need only match a target in terms of their effect on the cones;

Pow

er (a

rbit

rary

qua

ntal

uni

ts)

0.00

0.25

0.50

0.75

1.00

Wavelength (nm) Wavelength (nm)

Qua

ntal

effi

cien

cy

0.00

0.25

0.50 a b

400 500 600 700

L conesM conesS cones

400 500 600 700

Figure 1(a) Estimates of the spectral sensitivities of the human foveal long-wavelength (L), medium-wavelength (M), and short-wavelength (S)cones. The estimates are shown as quantal efficiencies, namely, probabilities that a quantum of a given wavelength will cause anisomerization. Data are taken from an article by Stockman & Sharpe (2000; see also CIE 2006). The low quantal efficiency of the Scones is caused primarily by wavelength-selective absorption of light by the lens and macular pigment. (b) Spectral aliasing (also knownas metamerism). Two physically different spectral irradiances are shown. These result in the same trichromatic code and are thusindistinguishable to a typical human. Calculation of spectral aliases is treated in detail elsewhere (Brainard & Stockman 2010,Koenderink 2010, Wyszecki & Stiles 1982).

520 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


thus, a mixture of three well-chosen primaries is generally sufficient to reproduce arbitrary lightspectra).

A feature of the standard trichromatic theory is that the spatial structure of the image playsno role. That is, trichromatic encoding and spatial variation of incident spectra are treated asseparable, the idea being that we can consider the retinal image as consisting of separate long-wavelength (L), medium-wavelength (M), and short-wavelength (S) cone planes, each of which isformed by pointwise computation of the L, M, or S cone excitations from the incident spectra.This simplification neglects a rich set of questions that arise when we consider the fine structureof the retinal cone mosaic. Rather than having three cones at each location, there is only one; thus,space and color are intertwined at the first stage of the neural processing of color. This articlereviews work, mostly done over the past 30 years, that clarifies the nature of this intertwining andits implications for visual processing.

CONE IMAGES

An implication of trichromacy is that, for purposes of postreceptoral human vision, we can describethe retinal image as consisting of three hypothetical retinal cone images: The L cone imagedescribes the isomerization rate that would be seen by an L cone at each spatial position, theM cone image describes the isomerization rate that would be seen by an M cone at each spatialposition, and the S cone image describes the isomerization rate that would be seen by an S cone ateach spatial position. These retinal cone images are hypothetical because the actual retinal image ischaracterized as the spectral irradiance incident from each direction at each location, whereas thecone isomerization rates are properties of discrete cones that do not exist in three continuous sheets.

Calculation of the L, M, and S cone retinal images from a description of the external physicalstimulus is not trivial. Doing so requires integration of knowledge from two fields: physicallybased computer graphics and physiological optics. Physically based computer graphics (Foley et al.1990, Larson & Shakespeare 1998, Pharr & Humphreys 2010) seeks tractable approximations forstarting with a three-dimensional scene description that specifies the positions and properties ofilluminants, objects, and an imaging device and then computing a description of the light enteringthe pupil of the imaging device. Physiological optics (see also Adelson & Bergen 1991, Charman2010) seeks to characterize the imaging properties of the human eye to allow for computation ofcone images from a description of the entering light.

Here we adopt a simplification and consider the retinal image to arise when a flat, two-dimensional stimulus image is viewed by an observer. We can characterize the stimulus imageby the spectral radiance emitted at each location. We then apply a model of blurring by the opticsof the eye to the stimulus image in order to obtain the retinal spectral irradiance and to calculate L,M, and S cone isomerization rates from this irradiance (see Figure 2). It is important, particularlyfor our current purposes, to calculate the retinal cone images from the full spectral radiance im-age, as doing so allows us to take wavelength dependencies in the optical quality of the human eye(chromatic aberrations) into account (Autrusseau et al. 2011, Marimont & Wandell 1994, Thibos1987, Thibos et al. 1992). Such dependencies are the first stage of human color vision at whichsubstantial spatiospectral interactions occur.

THE CONE MOSAIC

Spectral information becomes intertwined with spatial structure in the first stages of visual pro-cessing. One reason for this is wavelength-dependent optical effects, as introduced above. A secondreason is that there is at most a single cone sampling each location of the retinal image: The trichro-macy observed for spatially extended stimuli arises only because the visual system makes inferences

www.annualreviews.org • Color and the Cone Mosaic 521

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


400 nm 550 nm 700 nm

400 nm 550 nm 700 nm

S cone

M cone

L cone

a

b

c Cone planes

Figure 2(a) A red, green, blue (RGB) rendering of an idealized stimulus image. The rendering was made from a hyperspectral image (collectedby the author and available at http://color.psych.upenn.edu/hyperspectral) according to the sRGB standard (IEC 1999), withoutmodeling optical blurring by the eye. (b) The stimulus image is specified by the spectral radiance at each location and wavelength. As anexample, the top panel shows three narrowband planes from the underlying hyperspectral image rendered in panel a. The fullhyperspectral image contained 31 narrowband planes (400 to 700 nm in 10-nm steps). Note that the spatial structure of the imageplanes is similar across them; what differs is the intensity at each location across the planes, which varies according to the spectrum. Forrendering, each narrowband image was scaled to its own maximum. The bottom panel shows the same three spectral planes afterblurring by the optics of the eye. Note that chromatic aberration in the optics leads to wavelength-dependent image blurring, andmaximal blurring occurs at short wavelengths. The blurring was computed assuming that the width of the stimulus image was 5 deg ofvisual angle. The calculations illustrated here do not account for wavelength-dependent changes in magnification due to lateral(transverse) chromatic aberration (Thibos 1987). (c) The retinal cone images may be computed from the retinal image using estimatesof the human cone spectral sensitivities. The retinal cone images are shown for a portion of the image shown in panels a and b. Notethat the S cone image is more blurred than the L and M cone images are, again a consequence of chromatic aberration. Thecalculations illustrated here neglect optical waveguiding by the cones themselves, which results in differential directional sensitivity tolight (Enoch & Lakshminarayanan 2010, Stiles & Crawford 1933).

522 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.

http://color.psych.upenn.edu/hyperspectral


HS YY AP MD BS

L:M = 0.4:1 L:M = 1.1:1 L:M = 1.2:1 L:M = 1.9:1 L:M = 16.5:1

5'5'

LMS

Figure 3The arrangements of different cone classes in the retinal cone mosaics of five of the observers studied by Hofer and colleagues (Hoferet al. 2005a,b). Each image shows a small patch of mosaic in which cone locations and types were characterized using adaptive opticsimaging. L cones are shown in red, M cones are shown in green, and S cones are shown in blue. The L:M cone ratios for these mosaicsare as follows: HS, 0.4:1; YY, 1.1:1; AP, 1.2:1; MD, 1.9:1; and BS, 16.5:1. Reproduced with permission from figure 1 in Hofer et al.(2005a). Copyright c© 2005 Association for Research in Vision and Ophthalmology.

based on information from cones of different classes at different retinal locations. In this review,we explicitly consider the spatially interleaved nature of the retinal cone mosaic, and, togetherwith considerations of the optics of the eye, we connect features of the mosaic with perception.

The images shown in Figure 3 illustrate the cone mosaics for five human subjects. In eachimage, the L cones are indicated in red, the M cones are in green, and the S cones are in blue.Each image was obtained from a different subject, and all subjects had normal trichromatic colorvision according to standard tests. Several striking features of the mosaics are apparent.

First, there are considerably fewer S cones than there are either L or M cones. S cones makeup only approximately 5% of the cones in the human retina (Curcio et al. 1991, Hofer et al.2005b; see also Marc & Sperling 1976, Mollon & Bowmaker 1992, Roorda & Williams 1999,Williams et al. 1981b), and this proportion is fairly constant across individuals. Outside of thecentral fovea, where there is a small region of approximately 0.3 deg in diameter that contains noS cones (Curcio et al. 1991, Williams et al. 1981a), the arrangement of the S cones, although byno means crystalline, is more regular than would be expected by chance (Curcio et al. 1991, Hoferet al. 2005b).

Second, in contrast to the relatively constant proportion of S cones across individuals, there isstriking individual variation in the relative numbers of L and M cones. The L:M cone ratios inthe mosaics shown in Figure 3 vary from 0.4:1 to 16.5:1; the M cone–rich mosaic belongs to afemale carrier of protanopia who would be expected to have such a mosaic (Miyahara et al. 1998).This large variation among color-normal observers is consistent with results obtained using lessdirect estimates of the L:M ratio (e.g., Carroll et al. 2002; see below). Recent studies indicate thatthe variation rides around an average L:M cone ratio in humans of approximately 2:1 (Carrollet al. 2002) to 2.5:1 (Hofer et al. 2005b). This ratio appears to differ in other Old World primates(Dacey et al. 2000, Dobkins et al. 2000, Jacobs & Deegan 1999, Lindbloom-Brown et al. 2014,Mollon & Bowmaker 1992).

The L and M submosaics are arranged randomly or nearly so (Hofer et al. 2005b; see alsoMollon & Bowmaker 1992), although there appears to be slight nonrandom structure in someindividuals. Such structure is in the direction of regular spacing in some individuals and in thedirection of clumping of cones belonging to the same class together in others (Hofer et al. 2005b).Because of the individual variation in cone ratio and the random element to the packing of the cone


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


submosaics, the detailed structure of the cone mosaic differs in every individual: Postreceptoralcircuitry must therefore be tailored to the particular mosaic of each individual.

Features of the retinal mosaic vary systematically across the retina. First, the overall cone densityvaries tremendously as a function of eccentricity (Curcio et al. 1990, Østerberg 1935). Density ishighest in the central fovea, where there are no rods, and it falls off systematically with eccentricity.The falloff is accompanied by a change in the regularity and geometry of the cone packing (Curcio& Sloan 1992). In the fovea, cones are closely packed into an almost regular hexagonal grid. Aseccentricity increases, the cone packing becomes less regular as rods become interspersed. Thechanges in overall cone density are not homogeneous across the three cone classes. In particular,although overall cone density declines monotonically with eccentricity, S cone density rises fromthe central fovea and reaches a peak at 0.5 to 1 deg of eccentricity before falling off (Curcioet al. 1991). Less is known about variation in the L:M cone ratio with eccentricity. There isindirect evidence based on electroretinogram (ERG) flicker photometry (see below) that, within anindividual, the L:M cone ratio is roughly preserved across the central retina. However, analysis ofRNA expression suggests an increase in L:M ratio as eccentricity increases toward the far periphery(Hagstrom et al. 1998), as do data collected using the multifocal ERG (but see Challa et al. 2010,Kuchenbecker et al. 2008). With respect to the correlation in the L:M cone ratio between the twoeyes of a single individual, the difference in the L:M cone ratio between patches from the two eyes ofone subject is small relative to the overall range of individual variation (Roorda & Williams 1999).

SPATIOCHROMATIC ALIASING

S Cone Submosaic Aliasing

The cone mosaic spatially samples the image, and we now turn to understanding the consequencesof this sampling. It is convenient to begin with the S cone submosaic and to consider spatialmodulations of stimuli that only modulate the S cones. Such modulations may be produced bysuperimposition of short-wavelength lights on intense middle-to-long-wavelength backgrounds(Williams et al. 1981a) or by appropriate modulation of superimposed primaries around a rod-saturating background, also known as the method of silent substitution (Estevez & Spekreijse1982, Spitschan et al. 2015, Vienot & Brettel 2014). When only the S cones are modulated, all ofthe information available to the visual system about the spatial pattern of the image is carried bythe S cone submosaic.

Figure 4a illustrates how information is lost by spatial sampling, for one-dimensional signals.Because the cones sample the image discretely, variation in the image that occurs between thecones is not sensed directly. Two patterns for which the value in the retinal image is the same atthe locations at which there are cones, but for which that value differs at locations between thecones, will be spatial aliases of one another and will be indistinguishable to the visual system.1

Williams and colleagues (Williams & Collier 1983, Williams et al. 1983) demonstrated spatialaliasing by the S cone submosaic. They presented a short-wavelength grating stimulus against anintense longer-wavelength background. Such a stimulus produces negligible contrast for the Land M cones, so we may conceive of it as consisting of L and M cone image planes that are spatiallyuniform, together with an S cone image plane that varies sinusoidally across space. For low–spatialfrequency gratings, observers report seeing the grating percept. As the spatial frequency of the

1Note that even when cones are closely packed, spatial aliases of this type will continue to exist, as sampling by cones of finitesize is equivalent to blurring by the cone apertures followed by punctate sampling at the centers of their locations.

524 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Figure 4(a) Schematic of spatial aliasing for regular sampling in one spatial dimension. The two sinusoidal patterns have the same value at eachlocation at which there is a cone and differ only at the locations between cones: They produce the same isomerization rates for all of thecones in the mosaic and are spatial aliases of one another. Panel reproduced with permission from figure 1 in Brainard & Williams(1992). Copyright c© 1992 Elsevier. (b) S cone submosiac aliases. The left image shows a 1-deg patch of a 12 cycle/deg sinusoid thatmodulates only the S cones. The right image shows a low–spatial frequency alias for this sinusoid, computed for a model of humanfoveal cone mosaic. The splotchiness of the low–spatial frequency alias arises because the S cone submosaic does not sample the imagein a perfectly regular pattern. Effects of optical blurring were not accounted for in producing this alias pair. The electronic version ofthese images, available in the PDF version of this article, renders the intended spatiochromatic structure more faithfully than does theprinted version.

grating increases, however, it is no longer seen veridically. Observers can still detect the stimulus,but they report a splotchy percept from which they cannot identify the orientation of the grating.

The explanation for the observed phenomenology is illustrated by Figure 4b. For any grating,many physical S cone image planes are consistent with the array of S cone responses to the grating.It makes sense that the percept experienced by the observer is consistent with one such image;that is, the cone responses do constrain the percept. But which of the many stimuli in the set ofspatial aliases of the stimulus is perceived cannot be determined by the S cone responses alone.The percept is an inference, owing to postreceptoral processing. Apparently, such processingfavors a low–spatial frequency reconstruction of the stimulus, as low-frequency gratings are seenveridically. And, when the spatial frequency of the stimulus becomes high relative to the spatialsampling rate of the mosaic, the low–spatial frequency alias of a sinusoidal grating for the humanS cone mosaic is an irregular, splotchy, low-frequency pattern (Figure 4b) (Yellott 1983), whichis similar to what observers report seeing.

Williams et al.’s (Williams & Collier 1983, Williams et al. 1983) measurements show that Scone spatial aliasing artifacts begin to intrude at spatial frequencies that exceed approximately 10–14 cycles/deg for foveally viewed stimuli subtending 10 deg of visual angle. Williams and colleagues(Williams & Collier 1983, Williams et al. 1983) were able to use perceptual observations of this sortto estimate S cone packing densities that are in good agreement with anatomical data. Althoughthis finding is beyond the scope of this review, Williams’ lab was also able to demonstrate aliasingby the cone mosaic as a whole using isochromatic gratings that provided equal contrast to all threecone types and to exploit these effects to estimate the density and arrangement of the cone mosaicin the living human eye (Coletta & Williams 1987, Williams 1985).

Why aren’t aliasing artifacts of the sort illustrated by Figure 4b pervasive in natural view-ing? Because any image may be represented as a sum of sinusoidal gratings of different spatialfrequencies, orientations, and phases (Pratt 1978), one might expect that high–spatial frequency


a b

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


components of the S cone image would be subject to aliasing artifacts and that our percepts wouldbe blemished. For natural images, the amplitude of the spatial frequency components tends to falloff as the spatial frequency itself increases (Field 1987, Simoncelli 2005), reducing the likelihoodthat high-contrast image components with high spatial frequencies will be encountered. In addi-tion, axial chromatic aberrations differentially blur short-wavelength light and reduce the powerof the high–spatial frequency components of the S cone image plane. Thus chromatic aberrationprotects against aliasing artifacts at the cost of removing high–spatial frequency information atshort wavelengths. The protective effect of optical blur was bypassed in Williams’ experiments bypharmacologically paralyzing accommodation and bringing the monochromatic short-wavelengthstimulus into good focus. S cone aliasing may also be obtained if laser interferometry is used toproduce a high-contrast, high–spatial frequency stimulus that isolates the S cones (Brainard &Williams 1992).

The fact that chromatic aberration in the optics of the eye could protect human vision from Scone aliasing artifacts was first noted by Yellott and colleagues (1984). These authors suggestedthat the reason S cones can sample the retinal image so coarsely is in fact that the optics blurthe S cone retinal image plane sufficiently to prevent finer sampling from providing additionalinformation. This observation does not, however, address the issue of why the optics are soaberrated in the first place, nor does it address the issue of why the visual system typicallyaccommodates to longer wavelengths. We return to these issues later in this review.

A Camera Example

The implications of mosaicked sampling become richer if we consider sampling by more thanone cone submosaic at a time. We can gain an intuitive understanding of this phenomenon byexamining artifacts that appear in digital color images because most digital cameras use sensorswith interleaved R (red), G (green), and B (blue) sensors. Figure 5a shows an image obtained withsuch a camera. Image regions in which the high–spatial frequency herringbone pattern on the sub-ject’s jacket is rendered as a lower–spatial frequency chromatic grating are readily observed. Thisnonveridical rendering occurs because the interleaved camera sensors do not distinguish betweenhigh–spatial frequency achromatic patterns and lower–spatial frequency chromatic patterns; bothimages can produce the same set of sensor responses.

As an example, consider the reduced case of a two-pixel camera that has one R sensor andone G sensor, as shown in Figure 5b. If an achromatic grating that goes from light to darkacross the two sensors is imaged in this setup, the sensor responses will go from high to low.As with the cones, the sensors of the camera are univariate. The R sensor will respond stronglybecause the light achromatic stimulus contains power at all wavelengths and, in particular, in thelong-wavelength end of the spectrum to which the red sensor is responsive. The G sensor will notrespond strongly, because the dark achromatic stimulus contains little power at any wavelength.The RG sensor doublet for this two-pixel camera in response to the achromatic grating will bea large number paired with a small number.

Now consider a spatially uniform red stimulus imaged on the same sensor. The R sensor willalso respond strongly to this stimulus because it contains power in the long-wavelength end ofthe spectrum. The G sensor will respond only a little because the stimulus contains little powerin the middle wavelengths. The doublet of RG sensor responses to this second stimulus will alsobe a large number paired with a small number.

What this means is that an achromatic grating and a spatially uniform red stimulus can produceexactly the same set of responses in all of the sensors in this simple camera. Thus, these two stimuliare spatiochromatic aliases of one another: They produce identical camera responses, meaning

526 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


R G R Ga b

Figure 5(a) A digital image demonstrating artifacts due to interleaved sampling of the red (R), green (G), and blue (B) sensors of a digitalcamera. The red and green distortions seen on the jacket are the result of spatiochromatic aliasing. Note that the distortions on the faceof the subject were introduced separately to protect her identity. Panel reproduced with permission from figure 1 of Brainard et al.(2008). Copyright c© 2008 Association for Research in Vision and Ophthalmology. (b) Schematic illustration of spatiochromaticaliasing for a simple, two-pixel RG sensor. The two-pixel sensor is illustrated at the top of two pairs of stimuli. The achromatic gratingand uniform red patch produce the same responses in this sensor array and are aliases of one another. Similarly, the opposite-phaseachromatic grating and uniform green patch are also aliases of one another for the same sensor array.

that the information that distinguishes them has been lost. Similarly, an achromatic grating of theopposite spatial phase will be a spatiochromatic alias of a spatially uniform green stimulus.

Although our two-pixel camera example is highly simplified, it captures both the basic nature ofthe information loss produced by interleaved color sampling and the type of artifacts to which thisloss can lead. Given any set of responses from an interleaved color sensor, multiple spatiochromaticpatterns will exist that could have produced an identical set of responses. To produce an RGBimage for viewing, the camera’s processing pipeline must include a reconstruction algorithmthat promotes the mosaicked RGB responses to a full, nonmosaicked, RGB image for display.For each set of mosaicked RGB responses, such an algorithm can choose only one of the set ofcorresponding aliases as the output full RGB image, and, sometimes, this algorithm will producean output image that does not match the image impinging on the camera’s sensors. An algorithmicpreference for low–spatial frequency chromatic patterns over high–spatial frequency achromaticpatterns, implemented in the camera’s processing pipeline, leads to the image artifacts shown inFigure 5a.

L Cone and M Cone Submosaic Aliasing

Although the human retina employs mosaicked trichromatic sampling, we rarely perceive thesort of chromatic fringing observed in digital images. This raises two central questions: (a) Howdoes the visual system process the mosaicked cone responses to provide us with useful percepts ofthe spatiochromatic structure of the world, and (b) what aspects of visual information processing


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


can we understand as being consequences of the interleaved sampling and concomitant neuralprocessing? These questions, which were little treated in the literature until the 1980s, are nowreceiving increased attention. Although our knowledge remains incomplete, interesting progresshas been made.

In principle, production of submosaic aliasing should be possible for the L and M cone submo-saics. In practice, the overlap in spectral sensitivity between the L and M cones makes it difficultto produce stimuli seen with high contrast by one of these classes in isolation. Packer & Williams(described in Williams et al. 1991) used monochromatic interference fringes against monochro-matic backgrounds in an attempt to isolate the L cone and M cone submosiacs, but they wereunable to demonstrate differences in the resolution limit for these stimuli and for isochromaticstimuli that drove both cone classes together.

Williams et al. (1991) took another approach. They instructed observers to view high–spatialfrequency achromatic gratings. They reasoned that for high spatial frequencies, artifacts such asthose illustrated in Figure 5a should be observed. And indeed, their observers reported fleetingpercepts of red–green colored splotches superimposed on a percept of an achromatic grating. Thecolor appearance of the splotches changed with the accommodation level of the observer in amanner consistent with an expected switch between retinal L and M cone contrast in one case andwith S cone contrast in the other. Such splotches were originally noted by Brewster (1832), andthey are sometimes referred to as “Brewster’s colors.” Williams et al. (1991) concluded that thecause of the red–green splotches was spatiochromatic aliasing by the L and M cones.

Williams et al.’s (1991) conclusions were supported by a modeling exercise with submosaicaliasing at its core. Their model predicted qualitative features of the observations; however, thesalience of the red–green splotches in the model was much greater than that of the actual Brewster’scolors, which are quite subtle. The model treated each submosaic separately, so the predicted imagefor each cone class was obtained without using information from the other cone classes. Such aprocedure is optimal only if the spatial structure of the cone images is statistically independent. Fornatural images, such independence does not hold (Benson et al. 2014, Burton & Moorehead 1987,Chakrabarti & Zickler 2011, Ruderman et al. 1998, Tkacik et al. 2011). This idea may be readilyappreciated by examination of grayscale versions of the three color planes from essentially anynatural image—the same spatial structure is readily seen in each plane (see Figure 2 above). Thevisual system could reconstruct the three cone images more effectively from the full interleavedmosaic than would be possible using each submosaic separately.

Submosaic Interactions

Brainard & Williams (1992) pursued experimentally the question of whether the visual systemcombines information across submosiacs by asking whether the spatial reconstruction of signalsoriginating in the S cone submosaic varies with the information carried by L and M cones. Theyconsidered the phenomenon of filling-in at the central tritanopic region of the fovea in whichthere are no S cones. If a small (having a diameter of less than 20 arcmin), short-wavelengthincrement is viewed against an intense middle-wavelength background, the increment is visibleonly to the S cones. When the subject fixates on the increment, imaging it on the tritanopicarea, the increment fills in and vanishes (Williams et al. 1981a), much as occurs with filling-in atthe blind spot corresponding to the optic disc. Brainard & Williams (1992) showed that L andM cone signals can modulate the amount of S cone filling-in and thus that the visual system canreconstruct the spatial structure corresponding to one cone submosaic using information from theothers. They were not able to demonstrate interactions between the aliasing for moderate–spatialfrequency S cone gratings and L and M cone signals, however. Perhaps this is because, for natural

528 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


images, chromatic aberration eliminates cross–cone class correlations in the retinal image for suchgratings.

FUNCTIONAL CONSEQUENCES OF THE L:M CONE RATIO

A striking feature of the mosaics shown in Figure 3 is the large variation in the L:M cone ratio.Yet, each of the individuals whose mosaic is shown has normal color vision by standard tests. Arethere any measureable functional consequences of the variation in L:M cone ratio?

Brainard et al. (2000) studied two subjects with L:M ratios of 1.2:1 and 3.8:1. They measuredspectral sensitivities using the full-field photopic flicker ERG (31 Hz) and also measured thewavelength of unique yellow. The flicker ERG spectral sensitivities were well fit with the sum ofL and M cone spectral sensitivities, and the L:M weights of these fits were in good agreementwith the underlying ratios in the mosaic. This conclusion was subsequently confirmed by Hoferet al. (2005a), who studied a larger group of subjects with both adaptive optics imaging and flickerERG and who found good agreement between the ratios obtained with the two techniques. Thisagreement presumably occurs because the flicker ERG taps signals at an early retinal site andis therefore proportional to the summed activity of the cones. The agreement between ERGspectral sensitivity and retinal L:M cone ratio validates studies that estimate this ratio using theERG (Brainard et al. 1999; Carroll et al. 2000, 2002; Jacobs & Neitz 1993; Kremers et al. 2000),although Hofer et al. (2005a) did observe that the ERG may slightly overestimate the relative Mcone numerosity.2 The agreement also suggests that the variation in the L:M cone ratio acrossthe central portion of an individual’s retina is reasonably small, as the retinal imaging results arefor small patches near the fovea and the ERG was obtained using 60–70-deg fields.

The flicker ERG has a psychophysical analog, the flicker photometric measurement of luminousefficiency (Kaiser & Boynton 1996). Spectral luminous efficiency is well fit as a sum of L and Mcone spectral sensitivities, and it has long been known that there is a great deal of individualvariation in luminous efficiency. In psychophysical studies in which isolating the responses ofchromatic mechanisms is of central importance, investigators often make individual-subject flickerphotometric measurements and use these measurements to construct isoluminant stimuli. An earlyhypothesis was that variation in psychophysical flicker photometry reflected variation in individualL:M cone ratios (De Vries 1948, Pokorny et al. 1991, Rushton & Baker 1964). Although therehave not been studies that compare psychophysically determined luminous efficiency with L:Mcone ratios obtained via direct imaging, psychophysical luminous efficiency is highly correlatedwith ERG spectral sensitivity (Kremers et al. 2000), and this result provides an empirical chainlinking individual variation in the L:M cone ratio to individual variation in luminous efficiency.

In contrast to the results for the flicker ERG and for psychophysical flicker photometry,Brainard et al. (2000) found that the wavelength of unique yellow, a measure of the propertiesof a red–green color-opponent mechanism, was independent of the L:M cone ratio across theirtwo directly imaged subjects (see also Hofer et al. 2005b, Kremers et al. 2000, Neitz et al. 2002).Brainard et al. (2000) suggested that the stability of unique yellow was governed by an adaptiveprocess that normalized the L and M cone inputs to the red–green color-opponent mechanism,and the normalization signal might be provided by the temporal average of the light entering theeye.

2As a technical matter, estimation of the L:M cone ratio from an ERG or from psychophysically measured spectral sensitivityis sensitive to the estimates of L and M cone spectral sensitivity employed in the data analysis (Brainard et al. 2000, Carrollet al. 2000).


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


The idea that normalization to environmental signals might calibrate color processing hasbeen studied directly in measurements of long-term chromatic adaptation. In an initial report,Yamauchi et al. (2002) instructed observers to spend long periods of time each day in rooms withred illumination. Each morning, spectral unique yellow (the wavelength that appeared neither rednor green) was measured for each observer before the observer entered the chromatically biasedenvironment. A shift of unique yellow built up steadily over several weeks, and was reversedon roughly the same timescale by adaptation to a room with green illumination. These resultswere replicated and extended by Neitz et al. (2002). More recent work indicates that similarnormalization processes can account for individual variability in unique green (Schmidt et al.2014). It appears that the visual system exploits statistical regularities in the spectral content ofthe natural environment to calibrate its processing of color against individual variation in the L:Mcone ratio, but not its processing of luminance.

Another approach to demonstrating functional differences in the L:M cone ratio is to examinedifferences in spatial resolution for stimuli that target either L or M cones. As noted above, Packer& Williams (described in Williams et al. 1991) did not observe differences in the spatial frequencyresolution limit for extended gratings designed to favor one cone class (L or M) over the other.More recently, however, Danilova et al. (2013) found that relative acuity for peripheral LandoltC stimuli directed toward L and M cones was correlated with estimates of the L:M cone ratioacross subjects, indicating increased cone-specific acuity with increasing numbers of cones of agiven class.

A THEORY OF BIOLOGICAL DEMOSAICKING

The work described above raises the issue of how a visual system would optimally reconstruct thethree idealized cone images, given the mosaic responses. Brainard (1994) developed a Bayesianalgorithm to solve this problem, formulated as a digital camera reconstruction algorithm. Brainardet al. (2008) used the same formulation and applied it to parameters typical of the human opticsand cone mosaic.

The formal problem posed in this work is to estimate the idealized (nonblurred) cone imageplanes impinging on the retina, given the set of isomerization rates for the cones in the retinalmosaic. The idealized cone planes can be described as a vector x, where the vector contains alist of the expected L, M, and S cone isomerization rates for cones at each location in the retinalimage. If the cone planes are specified on 100 × 100 pixel grid, the vector x would have 30,000entries. The data available to estimate x may be represented as a vector y, which contains theisomerization rates of cones in response to the retinal image. For a ∼0.2-deg patch of humanretina located ∼1 deg from the fovea, the vector y would contain approximately 100 entries.

A good estimate of x should have the property that it is consistent with the cone isomerizationrates y. That is, a good estimate should be one that is reasonably likely to have led to the coneisomerization rates, had the estimate actually been physically present. In the Bayesian approach,this likelihood is formulated explicitly as the conditional probability distribution p(y|x).

Clearly, given the dimensions of x and y above, as well as our discussion of aliasing, theestimation problem is underdetermined. Many choices of x will be consistent with any vector ofisomerization rates y, and thus many choices of x will produce the same likelihood. The Bayesianapproach to handling this underdetermination is to add a description p(x) of the probability thatany idealized cone image occurs. For Bayesian estimation to work well, there must be enoughstatistical structure in the distribution of natural images to place reasonably strong constraints onthe estimates. In addition, p(x) must provide an adequate description of this statistical structure.

530 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Once a likelihood p(y|x) and prior p(x) have been formulated, the Bayesian approach is straight-forward. Bayes’ rule is used to compute a posterior distribution p(x|y) = C p(y|x)p(x), where Cis a normalizing constant that does not depend on x. An estimate x is obtained from the posteriordistribution, for example, as the value of x that maximizes the posterior or as the posterior mean.For an introduction to Bayesian approaches to color vision, the reader is referred to Brainard(2009).

Brainard et al. (2008) used this approach. They explicitly constructed likelihood functionsp(y|x) that took into account of the optics of the eye (including axial but not lateral chromaticaberrations) and trichromatic sampling. They combined these likelihoods with priors that capturedtwo important regularities in the spatiospectral structure of natural images. First, images varyslowly over space, so the isomerization rate within a cone plane is similar for nearby spatiallocations (Burton & Moorehead 1987, Field 1987, Ruderman et al. 1998, Simoncelli 2005, Tkaciket al. 2011). Second, there is a high correlation between the isomerization rates that would beexperienced by cones of different classes at the same location (Benson et al. 2014, Burton &Moorehead 1987, Ruderman et al. 1998). Using Bayes’ rule, Brainard et al. (2008) were able toestimate idealized cone planes from cone responses. Consistent with the insight provided by theearlier work of Williams et al. (1991) and Brainard & Williams (1992), the algorithm combinesinformation across cone classes in estimating each cone plane. This approach makes sense, giventhe high correlations in the signals from different cone classes observed in natural images andspecified in the prior.

Several qualitative features of the algorithm’s performance are also consistent with humanperception. First, the work shows that good reconstructions of low–spatial frequency luminanceand chromatic stimuli may be obtained across the range of individual variation in the mosaics shownin Figure 3. That is, once the reconstruction algorithm is tailored to the mosaic of a particularindividual, differences among individual mosaics make little difference in the reconstructions ofthese stimuli (see Figure 6). This explains how all of the observers whose mosaics are shown inFigure 3 were able to pass standard tests of color vision.

Second, in contrast to the reconstructions obtained by Williams et al. (1991), in which theimage seen by each cone class was considered separately, the Bayesian algorithm predicts moresubtle chromatic fringing in response to in-focus luminance gratings of moderate spatial frequency(compare figure 16 from Brainard et al. 2008 and plate 1 from Williams et al. 1991).

Third, consideration of how interleaved sampling interacts with the statistics of natural imagesprovides insight into why the spatial bandwidth revealed by psychophysical threshold experiments[via spatial contrast sensitivity functions (CSFs) measured for modulations in different color direc-tions (Mullen 1985, Sekiguchi et al. 1993)] varies with the color direction of the modulation. Thevisual system does not have access to a full spatiotrichromatic representation of the retinal image;roughly only one-third of that information is available because of mosaicked sampling. Given thestructure of natural images, high–spatial frequency luminance grating components are more likelystimuli than are high–spatial frequency chromatic grating components. Thus, it makes sense thatthe visual system reconstructs luminance gratings out to higher spatial frequencies than chromaticgratings and that the luminance CSF has a higher spatial frequency cutoff than the red–greenchromatic CSF (see figure 17 in Brainard et al. 2008).

Does the Bayesian algorithm actually describe in functional terms what the human visualsystem does? This question remains open, but an experiment by Hofer et al. (2005a) providesdata that can be parsimoniously accounted for by a model based on the Bayesian algorithm. Theirexperiment used adaptive optics to present small spots to the retina; the spots were so small thateach of them stimulated only a single cone. Observers were asked to report their color sensations.


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Stimulus Bayes reconstruction Sampling mosaic

AP

BS

LMS

Figure 6The left images show the same small (∼0.2-deg) image patch from a 6 cycle/deg sinusoidal grating. Themiddle patches show schematically two mosaics from Figure 3 with different L:M cone ratios. The rightimages show reconstructions for each of these mosaics, obtained using the Bayesian algorithm proposed byBrainard et al. (2008). The same prior was used in both reconstructions, but the likelihood for each individualobserver matched his or her optics and mosaic. The reconstructions at this relatively low spatial frequencyare similar for the two observers; only subtle differences are visible. Figure adapted with permission fromBrainard et al. (2008). Copyright c© 2008 Association for Research in Vision and Ophthalmology.

The diversity of responses was surprising; some observers needed as many as eight different colornames to report what they saw, contrary to what would be expected if cones in each class produce asingle sensation associated with that class. However, this result is explainable if the visual system isoptimally using the full set of responses from the mosaic, in some local retinal area, to reconstructthe full trichromatic image. In that case, the identities of the cones surrounding the one beingstimulated are predicted to have an effect, as the Bayesian reconstruction when a single cone isstimulated depends on the classes of the cones in the neighborhood of it. Brainard et al. (2008) wereable to account for the naming data of Hofer et al. (2005a) by using the Bayesian reconstructionalgorithm applied to the experimental stimuli (Figure 7), suggesting that the broad principlesincorporated into the Bayesian algorithm may describe what is accomplished by mechanisms inthe human visual pathways.

Sharper tests and refinements of this idea await experiments in which the identity of the conebeing stimulated on each trial is known—this was not possible in those conducted by Hofer et al.(2005a)—as well as refinements of the Bayesian reconstruction algorithm that employ both richerpriors over the set of natural images and constraints imposed by the known structure of earlyvisual processing. One important feature of early vision not considered by Brainard et al. (2008)that will be critical to incorporate is lateral (transverse) chromatic aberration (Thibos 1987). Thisfeature shifts the effective locations in the scene sampled by cones of different classes and suggeststhat the appropriate proximity relationships to take into account when making predictions are

532 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Colo

rs n

amed

(%)

Observer

100

HS YY AP MD BS

90

80

70

60

50

40

30

20

10

0

a Data, 550 nm

Colo

rs n

amed

(%)

Observer

100

HS YY AP MD BS

90

80

70

60

50

40

30

20

10

0

b Predictions, 550 nm

Figure 7Summary of data and Bayesian model predictions. (a) Color naming data from five observers for small, 550-nm spots delivered to theretina using adaptive optics, as measured by Hofer et al. (2005a). The colors in the bars indicate the color names used by each subject,and the size of each colored region provides the fraction of trials on which that color name was used. Note that the white areas of thebars represent the name “white.” (b) Predictions of the Bayesian model proposed by Brainard et al. (2008) for the data in panel a.Reproduced with permission from figure 11 of Brainard et al. (2008). Copyright c© 2008 Association for Research in Vision andOphthalmology.

those inferred after adjustment for wavelength-dependent optical displacements, not those directlyobserved in the mosaic.

UNDERSTANDING WHY THE MOSAIC IS THE WAY IT IS

As reviewed above, several features of the human cone mosaic are striking, including the following:� There are three (rather than, say, one, two, or four) classes of cones.� The spectral sensitivities of the L and M cones are highly overlapping.� The packing arrangement of the L and M cones is almost random.� There is high individual variability in the L:M cone ratio, and there are many fewer S cones

than L and M cones.

When one examines mosaics across different species, none of these features appears necessary.For example, other species have both fewer photoreceptor classes and more photoreceptor classesthan humans (e.g., Bowmaker 1991, Cronin & Marshall 1989, Hart 2001, Jacobs 2009, Jacobset al. 1993, Marshall & Oberwinkler 1999), much less overlapped cone spectral sensitivities (e.g.,Bowmaker 1977, 1991), and regular cone packing in stereotypical mosaics across individuals (e.g.,Bowmaker & Kunz 1987, Kram et al. 2010, Scholes 1975). In addition, although not part of themosaic per se, the optics of the human eye have strong chromatic aberrations. These variousfeatures beg the issue of why our mosaics are the way they are. This type of “why” question isdifficult, and perhaps ultimately impossible, to answer with certainty. Nonetheless, two approachesallow interesting speculation. One is to consider the particular details of how features of the mosaicevolved, treating some starting state on the evolutionary path as a given and asking what mutations


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


and selective advantages led from that point to the currently observed structure. The other is toconsider, as a theoretical matter, how visual performance depends on the structure of a particularfeature of the mosaic (e.g., the number of cone classes or the L:M cone ratio) and to then askwhether the observed structure supports near-optimal performance. Both approaches have yieldedinsight in the case of color vision.

Evolution and the Cone Mosaic

As an illustrative example of evolutionary natural history, we consider primate trichromacy. Themammalian ancestors of primates were dichromats and had mosaics that contained an S cone andone cone class sensitive to longer wavelengths, which we refer to as an L/M cone ( Jacobs 2009).Interestingly, these early mammals likely evolved from species that had at least three classes ofcones ( Jacobs & Rowe 2004), from which the third class of cones was lost because early mammalswere nocturnal. (We consider why mosaics with fewer cone classes may be preferable at low lightlevels below.)

Although most modern mammals remain dichromats, primates re-evolved trichromatic conemosaics. In Old World primates, this occurred quite recently, approximately 35 million yearsago (Nathans 1999), via a duplication of the gene that coded for the ancestral L/M cone opsin.Remarkably, changes at only one amino acid position in this opsin can produce an approximately15-nm shift in the wavelength of peak sensitivity for this photopigment, and only a few aminoacid substitutions (∼6) are required to produce the ∼30-nm spectral shift between the L and Mcone photopigments of modern Old World primates ( Jacobs & Neitz 1985). The initial step in theevolution of human trichromacy was gene duplication, followed by mutation, which produced dif-ferential spectral sensitivities in the photopigments for which opsins were coded by the two genes.3

The addition of a gene coding for a third photopigment opsin does not in and of itself conferbehavioral trichromacy. First, some mechanism must exist that causes each cone to express onlyone photopigment. In the case of Old World primates, the molecular mechanism that causes thephotopigment genes to be expressed was not duplicated, leading to expression of only one of theduplicated genes in any given cone (Smallwood et al. 2002). The gene that is expressed via thisprocess in each cone may be determined independently in each cone, which would explain thenear random arrangement of L and M cones in the primate mosaic. The factors that determine therelative likelihood that a cone will express L versus M pigment are not currently well understood(but see Knoblauch et al. 2006).

The following question then arises: How did the rest of the primate visual system manage totake advantage of the expression of an extra photopigment? There appears to be an evolutionary“chicken-and-egg” conundrum here: Trichromatic color vision requires a trichromatic mosaic,yet how does a mutation that produces a trichromatic cone mosaic confer a selective advantagewithout a postreceptoral visual system evolved to make use of it? This question is of interest notonly for the special case of trichromatic color vision, but also because it applies more generally forunderstanding the evolution of sensory systems: How does a brain know how to take advantageof and organize the information provided by novel classes of sensors?

One clue is provided by an experiment by Mancuso et al. (2009). They used gene therapy tointroduce a human L cone photopigment into the retina of an adult dichromatic monkey that had

3Some species of New World primates have also evolved a form of trichromatic color vision, although the evolutionary pathand mechanisms differ somewhat from those for Old World primates (Dulai et al. 1999, Jacobs & Nathans 2009, Jacobs et al.1996).

534 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


theretofore only had S and M cones. Prior to gene therapy, the monkey had been trained to makecolor discriminations, and it could only make discriminations supported by S and M cones. Oncethe L cone was introduced, the monkey was able to make trichromatic discriminations, indicatingthat the third cone photopigment, together with the postreceptoral visual system of the monkey,was able to rapidly support novel visual discriminations. A similar result had been obtained earlierin mice ( Jacobs et al. 2007).

Perhaps this result should not be surprising. By adding a third cone class, we change the set ofspatiochromatic aliases for the mosaic. After gene therapy, two different spatially uniform fields ofdifferent spectral composition that produced the same array of cone responses prior to gene therapywill no longer do so. Rather, the uniform fields will now produce a pattern of responses consistentwith a high–spatial frequency spatiochromatic alias of a uniform field, and the exact pattern ofresponses for each field will depend on the incident spectrum. As the monkey was rewarded simplyfor making discriminations, not for experiencing color per se, the monkey may have been makingdiscriminations on the basis of a perceived change in spatial structure (Makous 2007). That is, abrain that can respond to differences in the array of sensory input to make discriminations canmake novel discriminations anytime the sensory front end changes. We might call this a weakform of color vision, as it allows novel discriminations of stimuli of different spectral compositionand the same spatial structure. The ability to make this type of novel discrimination might wellprovide a selective foraging advantage, even if the perceptual effect was not experienced in a formthat humans would describe as seeing novel colors.

The computational work on demosaicking reviewed above suggests that the appropriate pro-cessing of the mosaic’s signals for veridical spatiochromatic vision requires circuitry that takes intoaccount the position and class of each cone. Although discriminations can be made without suchknowledge, useful representation (here, operationalized as accurately reconstructing the retinalimage) requires more. How can a postreceptoral visual system detect the presence of a novel coneclass and determine which cones belong to which class?

Benson et al. (2014; see also Wachtler et al. 2007) show that the correlations between coneexcitations in response to natural images provide enough information to identify the number ofcone types present in the mosaic and to classify each cone by type. The intuition for this resultis straightforward. If we consider two adjacent cones, they will have a higher correlation acrossresponses to natural images if they are of the same class than if they are of different classes, and thisrelation will hold for pairs of cones at any separation. Benson et al. (2014) simulated the responsesof the cone mosaic to natural images and used the simulated responses to compute the interconecorrelations for mosaics typical of human trichromatic vision. They then developed an algorithmthat successfully used these correlations to separate cones by class, as well as a clustering analysisto identify the number of classes. Their work indicates that a postreceptoral visual system thatseeks statistical structure in the relation between the responses of its individual sensors would, withenough input data, be able to discover novel features of its front end and take advantage of thesenew features. That is, true trichromatic color vision can in principle be learned from the responsesof a trichromatic primate retina to natural image input. Other work shows that the positions ofcones in the mosaic may be learned by comparing the array of cone responses across translationaleye movements (Ahumada 1991, Maloney & Ahumada 1989).

The computational work on cone-class learning is agnostic about mechanism, but one mightconjecture that early in the evolutionary history of the introduction of a new photopigment, thecortex would be the most plastic site. In primates, the high–spatial resolution foveal specialization,in which each foveal cone appears to have a “private line” that transmits its output to the cortexdirectly, with minimal mixing with signals from adjacent cones (Calkins et al. 1994, Kolb &Dekorver 1991), may have been critical for allowing color information from a novel cone class to


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


reach the cortex without needing specialized retinal circuitry to separate signals from L and Mcones (Lennie et al. 1991). The degree to which retinal circuitry explicitly keeps L and M conesignals separate, beyond the level that would be expected by random wiring of L and M cones totheir postreceptoral targets, remains controversial (Buzas et al. 2006, Dacey 2000, Dacey & Packer2003, Field et al. 2010, Hansen et al. 2009, Jusuf et al. 2006; see, for example, Lee et al. 1998, as wellas Mullen & Kingdom 2002, Mullen et al. 2005, Solomon et al. 2005, Vakrou et al. 2005). Thus,this observation may be key to understanding how the relatively recent evolution of trichromacycan be supported by postreceptoral processing. In contrast, the typical mammalian retina containsbipolar and ganglion cell types that are specialized for processing S cone signals, as does theprimate retina (Dacey & Packer 2003), suggesting that there has been sufficient evolutionary timefor this more ancient form of mammalian color vision ( Jacobs 2009) to have woven its way intothe structure of the early visual system.

Computations and the Mosaic

Another way to gain insight into why the mosaic has the structure it does is to consider how variousfeatures of mosaic design affect visual performance, particularly in the context of natural imageinput. We can then ask whether a particular aspect optimizes performance. An affirmative answerwould suggest that evolutionary pressure led to the near optimality observed, whereas a negativeanswer would suggest either that idiosyncrasies of a particular evolutionary path placed constraintson the mosaic or that the computational analysis is incomplete. As with our treatment of evolutionabove, this review considers illustrative examples. Although we restrict our attention to the sensoryfront end, the same general approach has also been widely used to provide insight about visualperformance (e.g., Barlow 1956; Geisler 1989, 2011; Williams et al. 1993) and postreceptoralvisual physiology (e.g., Atick 1992, Atick et al. 1992, Buchsbaum & Gottschalk 1983, Koch et al.2004).

Manning & Brainard (2009) built on the Bayesian model of image reconstruction developedby Brainard et al. (2008). They noted that once one specifies a prior p(x) and a particular optics/mosaic/cone spectral sensitivity structure p(y|x), it is possible to define the estimator x that min-imizes the expected squared estimation error, with the expectation taken over draws of imagesfrom the prior. Thus, given a prior, one can associate with any optics/mosaic/spectral sensitivitystructure a measure of performance, namely, the expected squared estimation error obtained whenresponses of that mosaic are processed optimally.

Manning & Brainard (2009) investigated how the mosaic packing arrangement and the coneratio affected optimal performance for a simple model visual system. The model system operatedalong 1 spatial dimension, had 12 cones spaced equally along that dimension, and was dichromatic.Given this restriction, Manning & Brainard were able to exhaustively explore all mosaic arrange-ments for each cone ratio, and they did so for a prior that captured the type of spatiospectralcorrelations found in natural images. They drew two key conclusions.

First, for a given cone ratio, regular mosaics, namely, those that come as close as possible toalternating between cones of the two classes, gave the best performance. This is consistent with theregular submosaics observed in fish and birds (Bowmaker & Kunz 1987, Kram et al. 2010, Scholes1975), as well as with the nonrandom packing of the human S cone submosaic. It is not consistentwith the near-random packing found for human L and M cones. One possibility that remainsopen is that the regularity of the optimal mosaic depends strongly on the prior. The priors used byManning & Brainard captured only the second-order correlational regularities in natural images.The use of richer priors that capture more of the edge and object structure in natural imagesmight lead to different conclusions. In essence, this is the intuition behind an earlier suggestion

536 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


that irregular mosaics might allow better performance for natural images (Yellott 1982, 1983), theidea being that for an irregular mosaic, the set of aliases corresponding to an image with the regularstructure of a natural image would be irregular images atypical of natural viewing. This intuitionremains to be confirmed through a formal analysis. Another possibility that remains open is thatdifferent results would be obtained for optimization of a metric with higher biological relevancethan the expected squared estimation error (see Brainard & Freeman 1997).

Second, Manning & Brainard (2009) found that the optimal ratio between the two classesof cones depended critically on the signal-to-noise ratio (SNR) with which the two cone classestransmitted information. If the cone classes transmitted information with the same SNR, then a1:1 cone ratio was optimal. As the SNR shifted in favor of one cone class, however, having morecones of that class became optimal. This observation, coupled with several other factors (discussedin Manning & Brainard 2009) that might lead the SNR of one cone class to be better than thatof the others provides a potential computational explanation of why the human mosaic containsonly one spectral class of photoreceptors (the rods) that operates at low light levels, as well aswhy some nocturnal mammals have only one class of cones (see also Land & Osorio 2003, vanHateren 1993). Manning & Brainard’s (2009) calculations did not take into account differentialoptical blurring with wavelength (but see below), nor did they explore variation of cone spectralsensitivity. That said, their method can in principle be used to study optimal mosaic design withthese factors included (see also Levin et al. 2008). An important challenge remains to make thecalculations feasible for the two-dimensional case and for richer models of natural image statistics.

In a complementary line of work, Garrigan et al. (2010) used an information theoretic approachto try to understand what ratio of L, M, and S cones would be optimal for the primate retina.They based their work on the Penn Natural Image Database (Tkacik et al. 2011) and consideredhigh, daytime light levels.4 They also incorporated a simple model of chromatic aberration intotheir calculations. They asked how much information about natural images would be carriedby mosaics with different L:M:S cone ratios. They found that the amount of information wasessentially constant across a wide range of L:M cone ratios, and a significant decrease was observedonly at extreme ratios. The intuition for this is that the high correlation between L and M coneresponses, conferred in part by their high overlap in spectral sensitivity, means that trading offa cone of one type for one of the other has minimal impact on the total information carried bythe mosaic. Although there is an optimal L:M cone ratio, the dependence of performance on thisratio is weak over a wide range. This may reduce pressure for the evolution of mechanisms thatcontrol the L:M cone ratio. Other species that have more regular and stereotypical trichromaticmosaics also have a larger separation between the spectral sensitivities of their cone classes, and,for these species, more may be gained via precise control of the cone ratio.

Garrigan et al. (2010) also found that the SNR for the images they studied was slightly betterfor L and M cones than for S cones. In the absence of chromatic aberrations, this situation favorsa small increase in the numerosity of L and M cones compared with S cones, but not to the extentobserved. Once aberrations were introduced into the model, however, the small SNR advantage ofL and M cones provided an informational advantage for a system that focused light in the middlewavelengths, leaving the short-wavelength retinal signal as the most blurred. This factor in turnamplified the advantage of L and M cones, and led to prediction of a low proportion of S cones.

4The work of Garrigan et al. (2010) used an early version of this image database, which contained some errors of cameracalibration that affected estimates of overall image light level and of the chromatic aberration of the camera. These errorswere corrected when the database was published by Tkacik et al. (2011) and are unlikely to affect the conclusions drawn byGarrigan et al. (2010) because their results are robust to overall light level and because the effect of the optics of the camerais small compared with the effect of the human optics incorporated into the calculations.


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


This result extends the earlier conjecture (Yellott et al. 1984) that chromatic aberrations led tothe low proportion of S cones in the primate retina by suggesting why it is better to have longerwavelengths in good focus.

Chromatic aberration has also been suggested as one reason why primate L and M conespectral sensitivities are so highly overlapping (D.R. Williams, personal communication). Becauseof aberration, a small separation in spectral sensitivity allows the images seen by both the L and Mcone submosaics to be in good focus and for these combined mosaics to support high-resolutionfoveal spatial vision. A related idea was formulated by Osorio et al. (1998), while in contrastcalculations based on optimizing color information transmission alone for natural images predictlarger spectral separations between primate M and L cones (Lewis & Zhaoping 2006).

A difficulty with the computational approach is that it is not currently feasible to consider theoptimization problem while accounting for all relevant factors simultaneously. For example, inthe work of Garrigan et al. (2010), the presence of chromatic aberration is taken as a given, andthe optimization is performed over other features. If, instead, the relative number of S cones hadbeen fixed, then chromatic aberration might emerge as a feature that protects the visual systemfrom spatial aliasing by the S cone submosaic (Yellott et al. 1984). The assumption of chromaticaberration might be justified by the observation that chromatic aberration provides a useful cuefor accommodation (Burge & Geisler 2011), but once accommodation is brought in as part of therelevant visual performance measure, the ideal approach would be to include the full performancemeasure explicitly.

This type of limitation applies to all current computational analyses of this general sort: Theanswer as to what is optimal depends on nontrivial side assumptions. This limitation does noteliminate the value of the approach, as clarifying the effects of varying features of the sensoryfront end, even within a limited domain, sharpens our understanding. But it is important tokeep in mind the limits of that understanding. The optimal mosaic design also likely dependson the visual task under consideration (e.g., Chittka & Menzel 1992; Cummings 2004; Osorio &Vorobyev 1996; Regan et al. 2001; Sumner & Mollon 2000a,b), assumptions about energetic andspace constraints on the computations (e.g., Balasubramanian & Berry 2002, Balasubramanianet al. 2001, Laughlin 2001, Laughlin & Sejnowski 2003), and what is assumed as given about thestructure of the postreceptoral visual system (e.g., Cummings 2004). With respect to task effects,it is worth noting that human dichromats outperform human trichromats on some visual tasks( Jagle et al. 2006, Morgan et al. 1992, Sharpe et al. 2006).

SUMMARY

To understand visual processing, we must establish what visual information is encoded and whatis discarded. For color, the nature and consequences of the information loss caused by biologicaltrichromacy, the fact that the retina contains three classes of univariate cone photoreceptors, haslong been understood and appreciated. This article has reviewed work that considers the additionalloss caused by chromatic aberrations in the optics of the eye, coupled with the fact that cones samplethe retinal image in three interleaved submosaics. As with trichromacy, this information losshas consequences for vision, revealed most notably in various forms of spatiochromatic aliasing.In addition, we argue that optimization of visual performance requires that the postreceptoralcircuitry combine information across cone classes as it seeks to make inferences about the spatialand chromatic structure of the visual environment. Thus, spatial and color vision are intimatelyintertwined from the first stages of vision. Moreover, variations in the mosaic across individualsraise questions about how the visual system of each person learns the positions and classes of itscones and about the functional consequences of individual variation in the mosaic. We reviewed

538 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


both experiments and theory that provide partial answers to these questions. Finally, we reviewedboth the evolution of primate color vision and computational studies that seek to understand thedegree to which primate mosaics support optimal visual performance.

Although impressive progress has been made over the past 30 years, our understanding of thestructure of the interleaved human cone mosaic and its implications for vision is certainly notcomplete. Four current developments suggest that progress will continue. They are as follows:

1. First, genetic techniques, although not reviewed in detail here, have provided and will con-tinue to provide important information about what mechanisms control key features of themosaic and how these mechanisms evolved (for reviews, see Jacobs & Nathans 2009, Neitz& Neitz 2011).

2. Second, advances in psychophysics with adaptive optics (Harmening et al. 2013, Harmeninget al. 2014, Tuten et al. 2012) now enable investigators to probe psychophysically the func-tion of individual targeted cones of known type (Sabesan et al. 2015). Data from this typeof experiment should soon enable tests of and place additional constraints on models of thesort reviewed here.

3. Third, an ability to study how individual cones of known classes provide input to retinalganglion cells (Field et al. 2010) should allow a much more precise understanding of howretinal circuitry is or is not specialized to integrate signals from L and M cones at differentspatial locations.

4. Finally, increases in computing power and software tools are now likely to allow more ex-tensive modeling, both in terms of understanding the theoretical implications of mosaickedsampling and in terms of relating the information conveyed by the mosaic and early reti-nal processing to visual performance. Particularly key will be software tools (e.g., Farrellet al. 2014) that encapsulate accurate models of human physiological optics (including chro-matic aberration), photoreceptor sampling and transduction (including noise properties),and properties of early retinal processing such as photoreceptor adaptation and that makethese models easy to deploy.

SUMMARY POINTS

1. Information about color and space is intertwined at the very first stages of visual process-ing. This is because of chromatic aberrations in the eye’s optics and because the retinalcone mosaic spatially interleaves three classes of cone photoreceptor.

2. Because there is only a single cone at each retinal location, the trichromacy of humanvision results from neural processing that combines information from different classes ofcones across space.

3. Behavioral consequences of sampling by the interleaved cone mosaic include spatiochro-matic aliasing.

4. There are large individual differences in the cone mosaic, particularly in the ratio of Lto M cones. This suggests that each person’s nervous system must learn the positionsand classes of its own cones and take these into account in order to provide normal colorvision.

5. A Bayesian model provides one account of how signals from different cone mosaics can beprocessed to infer an accurate trichromatic percept. This model accounts for nonintuitivefeatures of the color names given to small spots that stimulate single cones.


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


6. Evolutionary and computational considerations begin to provide insight about the degreeto which the human cone mosaic optimizes trade-offs between color and spatial vision.

FUTURE ISSUES

1. What mechanisms control the structure of the cone mosaic in each individual?

2. What will be revealed by experiments in which it is possible to obtain behavioral andphysiological responses to stimuli that stimulate targeted individual cones, either singlyor in combination?

3. When and how is the structure of the cone mosaic learned in each individual, and basedon what information? What sites of neural plasticity mediate this learning?

4. What principles govern the processing that resolves ambiguity introduced by the fact thatthere is only one cone at each retinal location, to provide a trichromatic color percept? Atwhat stages of visual processing and how do neural mechanisms implement computationsbased on these principles?

DISCLOSURE STATEMENT

The author is not aware of any affiliations, memberships, funding, or financial holdings that mightbe perceived as affecting the objectivity of this review.

ACKNOWLEDGMENTS

I thank G.H. Jacobs, H. Hofer, B.A. Wandell, and D.R. Williams for many interesting discussions.B.A. Wandell provided helpful comments on the manuscript. Construction of figures was aidedby use of open source software tools, in particular the Psychophysics Toolbox (http://www.psychtoolbox.org), the Image System Engineering Tools for Biology (http://www.isetbio.org),and the Silent Substitution Toolbox (http://www.silentsubstitutiontoolbox.org). This workwas supported by US National Institutes of Health grant RO1 EY10016.

LITERATURE CITED

Adelson EH, Bergen JR. 1991. The plenoptic function and the elements of early vision. In ComputationalModels of Visual Processing, ed. MS Landy, JA Movshon, pp. 3–20. Cambridge, MA: MIT Press

Ahumada AJJ. 1991. Learning receptor positions. In Computational Models of Visual Processing, ed. JA Movshon,MS Landy, pp. 23–34. Cambridge, MA: MIT Press

Atick JJ. 1992. Could information theory provide an ecological theory of sensory processing. Network 3:213–51Atick JJ, Li ZP, Redlich AN. 1992. Understanding retinal color coding from first principles. Neural Comput.

4:559–72Autrusseau F, Thibos L, Shevell SK. 2011. Chromatic and wavefront aberrations: L-, M- and S-cone stimu-

lation with typical and extreme retinal image quality. Vis. Res. 51:2282–94Balasubramanian V, Berry MJ 2nd. 2002. A test of metabolically efficient coding in the retina. Network 13:531–

52Balasubramanian V, Kimber D, Berry MJ 2nd. 2001. Metabolically efficient information processing. Neural

Comput. 13:799–815

540 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.

http://www.psychtoolbox.org

http://www.psychtoolbox.org

http://www.isetbio.org

http://www.silentsubstitutiontoolbox.org


Barlow HB. 1956. Retinal noise and absolute threshold. J. Opt. Soc. Am. 46:634–39Benson NC, Manning JR, Brainard DH. 2014. Unsupervised learning of cone spectral classes from natural

images. PLOS Comput. Biol. 10:e1003652Bowmaker JK. 1977. The visual pigments, oil droplets, and spectral sensitivity of the pigeon. Vis. Res. 17:1129–

38Bowmaker JK. 1991. The evolution of vertebrate visual pigments and photoreceptors. In Evolution of the Eye

and Visual System, ed. JR Cronly-Dillon, RL Gregory, pp. 63–81. Boca Raton, FL: CRCBowmaker JK, Kunz YW. 1987. Ultraviolet receptors, tetrachromatic colour vision and retinal mosaics in the

brown trout (Salmo trutta): age-dependent changes. Vis. Res. 27:2101–8Brainard DH. 1994. Bayesian method for reconstructing color images from trichromatic samples. Proc. IS&T

47th Annu. Meet., pp. 375–79. Springfield, VA: IS&TBrainard DH. 2009. Bayesian approaches to color vision. In The Cognitive Neurosciences, ed. MS Gazzaniga,

pp. 395–408. Cambridge, MA: MIT Press. 4th ed.Brainard DH, Calderone J, Nugent AK, Jacobs GH. 1999. Flicker ERG responses to stimuli parametrically

modulated in color space. Investig. Ophthalmol. Vis. Sci. 40:2840–47Brainard DH, Freeman WT. 1997. Bayesian color constancy. J. Opt. Soc. Am. A 14:1393–411Brainard DH, Roorda A, Yamauchi Y, Calderone JB, Metha A, et al. 2000. Functional consequences of the

relative numbers of L and M cones. J. Opt. Soc. Am. A 17:607–14Brainard DH, Stockman A. 2010. Colorimetry. In The Optical Society of America Handbook of Optics,

Vol. 3: Vision and Vision Optics, ed. M Bass, C DeCusatis, J Enoch, V Lakshminarayanan, G Li, et al.,pp. 10. 1–10.56. New York: McGraw-Hill. 3rd ed.

Brainard DH, Williams DR. 1992. Spatial reconstruction of signals from short-wavelength cones. Vis. Res.33:105–16

Brainard DH, Williams DR, Hofer H. 2008. Trichromatic reconstruction from the interleaved cone mosaic:Bayesian model and the color appearance of small spots. J. Vis. 8(5):15

Brewster D. 1832. On the undulations excited in the retina by the action of luminous points and lines. Philos.Mag. 1(3):169–74

Buchsbaum G, Gottschalk A. 1983. Trichromacy, opponent colours coding and optimum colour informationtransmission in the retina. Proc. R. Soc. Lond. B 220:89–113

Burge J, Geisler WS. 2011. Optimal defocus estimation in individual natural images. PNAS 108:16849–54Burton GJ, Moorehead IR. 1987. Color and spatial structure in natural images. Appl. Opt. 26:157–70Buzas P, Blessing EM, Szmajda BA, Martin PR. 2006. Specificity of M and L cone inputs to receptive fields

in the parvocellular pathway: random wiring with functional bias. J. Neurosci. 26:11148–61Calkins DJ, Schein SJ, Tsukamoto Y, Sterling P. 1994. M and L cones in macaque fovea connect to midget

ganglion cells by different numbers of excitatory synapses. Nature 371:70–72Carroll J, McMahon C, Neitz M, Neitz J. 2000. Flicker-photometric electroretinogram estimates of L:M

cone photoreceptor ratio in men with photopigment spectra derived from genetics. J. Opt. Soc. Am. A17:499–509

Carroll J, Neitz J, Neitz M. 2002. Estimates of L:M cone ratio from ERG flicker photometry and genetics.J. Vis. 2(8):1

Carroll J, Yoon G, Williams DR. 2009. The cone photoreceptor mosaic in normal and defective color vision.In The Cognitive Neurosciences, ed. MS Gazzaniga, pp. 383–94. Cambridge, MA: MIT Press. 4th ed.

Chakrabarti A, Zickler T. 2011. Statistics of real-world hyperspectral images. Proc. IEEE Comput. Soc. Conf.Comput. Vision Pattern Recognit. 2011, pp. 193–200. New York: IEEE

Challa NK, McKeefry D, Parry NRA, Kremers J, Murray IJ, Panorgias A. 2010. L- and M-cone input to12Hz and 30Hz flicker ERGs across the human retina. Ophthalmic Physiol. Opt. 30:503–10

Charman N. 2010. Optics of the eye. In The Optical Society of America Handbook of Optics, Vol. 3: Vision andVision Optics, ed. M Bass, C DeCusatis, J Enoch, V Lakshminarayanan, G Li, et al., pp. 1.1–1.65. NewYork: McGraw-Hill. 3rd ed.

Chittka L, Menzel R. 1992. The evolutionary adaptation of flower colours and the insect pollinators’ colourvision. J. Comp. Physiol. A 171:171–81

CIE. 2006. Fundamental chromaticity diagram with physiological axes—Part 1. Tech. Rep. 170-1, Central Bur.Comm. Int. Eclair., Vienna


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Coletta NJ, Williams DR. 1987. Psychophysical estimate of extrafoveal cone spacing. J. Opt. Soc. Am. A4:1503–13

Cronin TW, Marshall NJ. 1989. A retina with at least ten types of spectral photoreceptors in a mantis shrimp.Nature 339:137–40

Cummings ME. 2004. Modelling divergence in luminance and chromatic detection performance across mea-sured divergence in surfperch (Embiotocidae) habitats. Vis. Res. 44:1127–45

Curcio CA, Allen KA, Sloan KR, Lerea CL, Hurley JB, et al. 1991. Distribution and morphology of humancone photoreceptors stained with anti-blue opsin. J. Comp. Neurol. 312:610–24

Curcio CA, Sloan KR. 1992. Packing geometry of human cone photoreceptors: variation with eccentricityand evidence for local anisotropy. Vis. Neurosci. 9:169–80

Curcio CA, Sloan KR, Kalina RE, Hendrickson AE. 1990. Human photoreceptor topography. J. Comp. Neurol.292:497–523

Dacey DM. 2000. Parallel pathways for spectral coding in primate retina. Annu. Rev. Neurosci. 23:743–75Dacey DM, Diller LC, Verweij J, Williams DR. 2000. Physiology of L- and M-cone inputs to H1 horizontal

cells in the primate retina. J. Opt. Soc. Am. A 17:589–96Dacey DM, Packer OS. 2003. Colour coding in the primate retina: diverse cell types and cone-specific circuitry.

Curr. Opin. Neurobiol. 13:421–27Danilova MV, Chan CH, Mollon JD. 2013. Can spatial resolution reveal individual differences in the L:M

cone ratio? Vis. Res. 78:26–38De Vries H. 1948. The heredity of the relative numbers of red and green receptors in the human eye. Genetica

24:199–212Dobkins KR, Thiele A, Albright TD. 2000. Comparisons of red-green equiluminance points in humans and

macaques: evidence for different L:M cone ratios between species. J. Opt. Soc. Am. A 17:545–56Dulai KS, von Dornum M, Mollon JD, Hunt DM. 1999. The evolution of trichromatic color vision by opsin

duplication in New World and Old World primates. Genome Res. 9:629–38Enoch JM, Lakshminarayanan V. 2010. The problem of correction for the Stiles-Crawford effect of the

first kind in radiometry and photometry, a solution. In The Optical Society of America Handbook of Optics,Vol. 3: Vision and Vision Optics, ed. M Bass, C DeCusatis, J Enoch, V Lakshminarayanan, G Li, et al.,pp. 9.1–9.18. New York: McGraw-Hill. 3rd ed.

Estevez O, Spekreijse H. 1982. The “silent substitution” method in visual research. Vis. Res. 22:681–91Farrell JE, Jiang H, Winawer J, Brainard DH, Wandell BA. 2014. Modeling visible differences: the compu-

tational observer model. Proc. 2014 Soc. Inf. Disp. (SID) Int. Symp. Campbell, CA: Soc. Inf. Disp.Field DJ. 1987. Relations between the statistics of natural images and the response properties of cortical cells.

J. Opt. Soc. Am. A 4:2379–94Field GD, Gauthier JL, Sher A, Greschner M, Machado TA, et al. 2010. Functional connectivity in the retina

at the resolution of photoreceptors. Nature 467:673–77Foley JD, van Dam A, Feiner SK, Hughes JF. 1990. Computer Graphics: Principles and Practice. Reading, MA:

Addison-WesleyGarrigan P, Ratliff CP, Klein JM, Sterling P, Brainard DH, Balasubramanian V. 2010. Design of a trichromatic

cone array. PLOS Comput. Biol. 6:e1000677Geisler WS. 1989. Sequential ideal-observer analysis of visual discriminations. Psychol. Rev. 96:267–314Geisler WS. 2011. Contributions of ideal observer theory to vision research. Vis. Res. 51:771–81Hagstrom SA, Neitz J, Neitz M. 1998. Variations in cone populations for red-green color vision examined by

analysis of mRNA. NeuroReport 9:1963–67Hansen T, Pracejus L, Gegenfurtner KR. 2009. Color perception in the intermediate periphery of the visual

field. J. Vis. 9(4):26Harmening WM, Tuten WS, Bruce KS, Holz FG, Roorda A, Sincich LC. 2013. Visual function testing at

the single cone level with a multi-wavelength adaptive optics SLO. Ophthalmologica 230:13–14Harmening WM, Tuten WS, Roorda A, Sincich LC. 2014. Mapping the perceptual grain of the human retina.

J. Neurosci. 34:5667–77Hart NS. 2001. Variations in cone photoreceptor abundance and the visual ecology of birds. J. Comp. Physiol.

A 187:685–97

542 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Hofer H, Singer B, Williams DR. 2005a. Different sensations from cones with the same photopigment. J. Vis.5(5):5

Hofer HJ, Carroll J, Neitz J, Neitz M, Williams DR. 2005b. Organization of the human trichromatic conemosaic. J. Neurosci. 25:9669–79

Hofer HJ, Williams DR. 2014. Color vision and the retinal mosaic. In The New Visual Neurosciences, ed. LMChalupa, JS Werner, pp. 469–83. Cambridge, MA: MIT Press

IEC. 1999. sRGB Standard, International Electrotechnical Commission Standard 61966-2-1Jacobs GH. 2009. Evolution of colour vision in mammals. Philos. Trans. R. Soc. Lond. B 364:2957–67Jacobs GH, Deegan JF II. 1999. Uniformity of colour vision in Old World monkeys. Proc. R. Soc. Lond. B

266:2023–28Jacobs GH, Deegan JF II, Neitz J, Corgnale MA, Neitz M. 1993. Photopigments and color vision in the

nocturnal monkey, Aotus. Vis. Res. 33:1773–83Jacobs GH, Nathans J. 2009. The evolution of primate color vision. Sci. Am. 300:56–63Jacobs GH, Neitz J. 1985. Spectral positioning of mammalian cone pigments. J. Opt. Soc. Am. A 2:23Jacobs GH, Neitz J. 1993. Electrophysiological estimates of individual variation in L/M cone ratio. In Colour

Vision Deficiencies XI, ed. B Drum, pp. 107–12. Dordrecht: KluwerJacobs GH, Neitz M, Deegan JF, Neitz J. 1996. Trichromatic colour vision in New World monkeys. Nature

382:156–58Jacobs GH, Rowe MP. 2004. Evolution of vertebrate color vision. Clin. Exp. Optom. 87:206–16Jacobs GH, Williams GA, Cahill H, Nathans J. 2007. Emergence of novel color vision in mice engineered to

express a human cone photopigment. Science 315:1723–25Jagle H, de Luca E, Serey L, Bach M, Sharpe LT. 2006. Visual acuity and X-linked color blindness. Graefe’s

Arch. Clin. Exp. Ophthalmol. 244:447–53Jusuf PR, Martin PR, Grunert U. 2006. Random wiring in the midget pathway of primate retina. J. Neurosci.

26:3908–17Kaiser PK, Boynton RM. 1996. Human Color Vision. Washington, DC: Opt. Soc. Am. 2nd ed.Knill DC, Richards W. 1996. Perception as Bayesian Inference. Cambridge, UK: Cambridge Univ. PressKnoblauch K, Neitz J, Neitz M. 2006. An urn model of the development of L/M cone ratios in human and

macaque retinas. Vis. Neurosci. 23:387–94Koch K, McLean J, Berry M, Sterling P, Balasubramanian V, Freed MA. 2004. Efficiency of information

transmission by retinal ganglion cells. Curr. Biol. 14:1523–30Koenderink J. 2010. Color for the Sciences. Cambridge, MA: MIT PressKolb H, Dekorver L. 1991. Midget ganglion cells of the parafovea of the human retina: a study by electron

microscopy and serial reconstructions. J. Comp. Neurol. 303:617–36Kram YA, Mantey S, Corbo JC. 2010. Avian cone photoreceptors tile the retina as five independent, self-

organizing mosaics. PLOS ONE 5:e8992Kremers J, Scholl HPN, Knau H, Berendschot TTJM, Sharpe LT. 2000. L/M cone ratios in human trichro-

mats assessed by psychophysics, electroretinography and retinal densitometry. J. Opt. Soc. Am. A 17:517–26

Kuchenbecker JA, Sahay M, Tait DM, Neitz M, Neitz J. 2008. Topography of the long- to middle-wavelengthsensitive cone ratio in the human retina assessed with a wide-field color mutlifocal electroretinogram.Vis. Neurosci. 25:301–06

Land M, Osorio D. 2003. Colour vision: colouring the dark. Curr. Biol. 13:R83–R85Larson GW, Shakespeare R. 1998. Rendering with Radiance: The Art and Science of Lighting Visualization. San

Francisco: Morgan KaufmanLaughlin SB. 2001. Energy as a constraint on the coding and processing of sensory information. Curr. Opin.

Neurobiol. 11:475–80Laughlin SB, Sejnowski TJ. 2003. Communication in neuronal networks. Science 301:1870–74Lee BB, Kremers J, Yeh T. 1998. Receptive fields of primate retinal ganglion cells studied with a novel

technique. Vis. Neurosci. 15:161–75Lennie P, Haake PW, Williams DR. 1991. The design of chromatically opponent receptive fields. In Com-

putational Models of Visual Processing, ed. MS Landy, JA Movshon, pp. 71–82. Cambridge, MA: MITPress


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Levin A, Durand F, Freeman WT. 2008. Understanding Camera Trade-Offs through a Bayesian Analysis of LightField Projections. Cambridge, MA: MIT Press

Lewis A, Zhaoping L. 2006. Are cone sensitivities determined by natural color statistics? J. Vis. 6(3):8Lindbloom-Brown Z, Tait LJ, Horwitz GD. 2014. Spectral sensitivity differences between rhesus monkeys

and humans: implications for neurophysiology. J. Neurophysiol. 112:3164–72Makous W. 2007. Comment on “Emergence of novel color vision in mice engineered to express a human

cone photopigment.” Science 318:196Maloney LT, Ahumada AJJ. 1989. Learning by assertion: two methods for calibrating a linear visual system.

Neural Comput. 1:392–401Mancuso K, Hauswirth WW, Li QH, Connor TB, Kuchenbecker JA, et al. 2009. Gene therapy for red-green

colour blindness in adult primates. Nature 461:784–87Manning JR, Brainard DH. 2009. Optimal design of photoreceptor mosaics: why we do not see color at night.

Vis. Neurosci. 26:5–19Marc RE, Sperling HG. 1976. Chromatic organization of primate cones. Science 196:454–56Marimont DH, Wandell BA. 1994. Matching color images: the effects of axial chromatic aberration. J. Opt.

Soc. Am. A 11:3113–22Marshall J, Oberwinkler J. 1999. The colorful world of the mantis shrimp. Nature 401:873–74Miyahara E, Pokorny J, Smith VC, Baron R, Baron E. 1998. Color vision in two observers with highly biased

LWS/MWS cone ratios. Vis. Res. 38:601–12Mollon JD, Bowmaker JK. 1992. The spatial arrangement of cones in the primate fovea. Nature 360:677–79Morgan MJ, Adam A, Mollon JD. 1992. Dichromats detect colour-camouflaged objects that are not detected

by trichromats. Proc. R. Soc. Lond. B 248:291–95Mullen KT. 1985. The contrast sensitivity of human colour vision to red-green and blue-yellow gratings.

J. Physiol. 359:381–400Mullen KT, Kingdom FAA. 2002. Differential distributions of red-green and blue-yellow cone opponency

across the visual field. Vis. Neurosci. 19:109–18Mullen KT, Sakurai M, Chu W. 2005. Does L/M cone opponency disappear in human periphery? Perception

34:951–59Nathans J. 1999. The evolution and physiology of human color vision: insights from molecular genetic studies

of visual pigments. Neuron 24:299–312Neitz J, Carroll J, Yamauchi Y, Neitz M, Williams DR. 2002. Color perception is mediated by a plastic neural

mechanism that is adjustable in adults. Neuron 35:783–92Neitz J, Neitz M. 2011. The genetics of normal and defective color vision. Vis. Res. 51:633–51Osorio D, Ruderman DL, Cronin TW. 1998. Estimation of errors in luminance signals encoded by primate

retina resulting from sampling of natural images with red and green cones. J. Opt. Soc. Am. A 15:16–22Osorio D, Vorobyev M. 1996. Colour vision as an adaptation to frugivory in primates. Proc. R. Soc. Lond. B

263:593–99Østerberg GA. 1935. Topography of the layer of rods and cones in the human retina. Acta Ophthalmol. 6:1–102Pharr M, Humphreys G. 2010. Physically Based Rendering: From Theory to Implementation. San Francisco:

Morgan KaufmannPokorny J, Smith VC, Wesner MF. 1991. Variability in cone populations and implications. In From Pigments

to Perception, ed. A Valberg, BB Lee, pp. 23–34. New York: PlenumPratt WK. 1978. Digital Image Processing. New York: WileyRegan BC, Julliot C, Simmen B, Vienot F, Charles-Dominique P, Mollon JD. 2001. Fruits, foliage and the

evolution of primate color vision. Philos. Trans. R. Soc. Lond. B 356:229–83Roorda A, Williams DR. 1999. The arrangement of the three cone classes in the living human eye. Nature

397:520–22Ruderman DL, Cronin TW, Chiao CC. 1998. Statistics of cone responses to natural images: implications for

visual coding. J. Opt. Soc. Am. A 15:2036–45Rushton WAH, Baker HD. 1964. Red/green sensitivity in normal vision. Vis. Res. 4:75–85Sabesan R, Schmidt B, Tuten WS, Boehm A, Roorda A. 2015. Functional organization of color in the trichromatic

cone mosaic. Presented at ARVO Annu. Meet., Denver, CO

544 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Schmidt BP, Neitz M, Neitz J. 2014. Neurobiological hypothesis of color appearance and hue perception.J. Opt. Soc. Am. A 31:A195–207

Scholes JH. 1975. Colour receptors and their synaptic connexions in the retina of a Cyprinid fish. Philos. Trans.R. Soc. Lond. B 270:61–118

Sekiguchi N, Williams DR, Brainard DH. 1993. Aberration-free measurements of the visibility of isoluminantgratings. J. Opt. Soc. Am. A 10:2105–17

Sharpe LT, de Luca E, Hansen T, Jagle H, Gegenfurtner KR. 2006. Advantages and disadvantages of humandichromacy. J. Vis. 6(3):213–23

Simoncelli EP. 2005. Statistical modeling of photographic images. In Handbook of Image and Video Processing,ed. A Bovik, pp. 431–41. New York: Academic

Smallwood PM, Wang Y, Nathhans J. 2002. Role of a locus control region in the mutually exculsive expressionof human red and green cone pigment genes. PNAS 99:1006–11

Solomon SG, Lee BB, White AJ, Ruttiger L, Martin PR. 2005. Chromatic organization of ganglion cellreceptive fields in the peripheral retina. J. Neurosci. 25:4527–39

Spitschan M, Aguirre GK, Brainard DH. 2015. Selective stimulation of penumbral cones reveals perceptionin the shadow of retinal blood vessels. PLOS ONE 10:e0124328

Stiles WS, Crawford BH. 1933. The luminous efficiency of rays entering the eye pupil at different points.Proc. R. Soc. Lond. B 112:428–50

Stockman A, Sharpe LT. 2000. Spectral sensitivities of the middle- and long-wavelength sensitive conesderived from measurements in observers of known genotype. Vis. Res. 40:1711–37

Sumner P, Mollon JD. 2000a. Catarrhine photopigments are optimized for detecting targets against a foliagebackground. J. Exp. Biol. 203:1963–86

Sumner P, Mollon JD. 2000b. Chromaticity as a signal of ripeness in fruits taken by primates. J. Exp. Biol.203:1987–2000

Thibos LN. 1987. Calculation of the influence of lateral chromatic aberration on image quality across thevisual field. J. Opt. Soc. Am. A 4:1673–80

Thibos LN, Ye M, Zhang X, Bradley A. 1992. The chromatic eye: a new reduced-eye model of ocular chromaticaberration in humans. Appl. Opt. 31:3594–600

Tkacik G, Garrigan P, Ratliff C, Milcinski G, Klein JM, et al. 2011. Natural images from the birthplace ofthe human eye. PLOS ONE 6:e20409

Tuten WS, Tiruveedhula P, Roorda A. 2012. Adaptive optics scanning laser ophthalmoscope-based mi-croperimetry. Optom. Vis. Sci. 89:563–74

Vakrou C, Whitaker D, McGraw PV, McKeefry D. 2005. Functional evidence for cone-specific connectivityin the human retina. J. Physiol. 566:93–102

van Hateren JH. 1993. Spatial, temporal and spectral pre-processing for colour vision. Proc. Biol. Sci. 251:61–68Vienot F, Brettel H. 2014. The Verriest lecture: visual properties of metameric blacks beyond cone vision.

J. Opt. Soc. Am. A 31:A38–46von Helmholtz H. 1896. Handbuch der Physiologischen Optik. Hamburg: Leopold VossWachtler T, Doi E, Lee T-W, Sejnowski TJ. 2007. Cone selectivity derived from the responses of the retinal

cone mosaic to natural scenes. J. Vis. 7(8):6Wandell BA. 1995. Foundations of Vision. Sunderland, MA: SinauerWilliams DR. 1985. Aliasing in human foveal vision. Vis. Res. 25:195–205Williams DR, Collier RJ. 1983. Consequences of spatial sampling by a human photoreceptor mosaic. Science

221:385–87Williams DR, Collier RJ, Thompson BJ. 1983. Spatial resolution of the short-wavelength mechanism. In

Colour Vision, ed. JD Mollon, LT Sharpe, pp. 487–503. London: AcademicWilliams DR, MacLeod DIA, Hayhoe MM. 1981a. Foveal tritanopia. Vis. Res. 19:1341–56Williams DR, MacLeod DIA, Hayhoe MM. 1981b. Punctate sensitivity of the blue-sensitive mechanism. Vis.

Res. 19:1357–75Williams DR, Sekiguchi N, Brainard DH. 1993. Color, contrast sensitivity, and the cone mosaic. PNAS

90:9770–77Williams DR, Sekiguchi N, Haake W, Brainard DH, Packer O. 1991. The cost of trichromacy for spatial

vision. In From Pigments to Perception, ed. A Valberg, BB Lee, pp. 11–22. New York: Plenum


Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.


Wyszecki G, Stiles WS. 1982. Color Science: Concepts and Methods, Quantitative Data and Formulae. New York:Wiley

Yamauchi Y, Williams DR, Brainard DH, Roorda A, Carroll J, et al. 2002. What determines unique yellow,L/M cone ratio or visual experience? Proc. SPIE 4421:275

Yellott JI Jr. 1982. Spectral analysis of spatial sampling by photoreceptors: Topological disorder preventsaliasing. Vis. Res. 22:1205–10

Yellott JI Jr. 1983. Spectral consequences of photoreceptor sampling in the rhesus retina. Science 221:382–85Yellott JI Jr, Wandell BA, Cornsweet TN. 1984. The beginnings of visual perception: the retinal image and

its initial encoding. In Handbook of Physiology: The Nervous System, ed. I Darien-Smith, pp. 257–316. NewYork: Easton

546 Brainard

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.

VS01-FrontMatter ARI 30 October 2015 19:57

Annual Review ofVision Science

Volume 1, 2015ContentsAn autobiographical article by Horace Barlow is available online atwww.annualreviews.org/r/horacebarlow.

Image Formation in the Living Human EyePablo Artal � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 1

Adaptive Optics OphthalmoscopyAustin Roorda and Jacque L. Duncan � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �19

Imaging GlaucomaDonald C. Hood � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �51

What Does Genetics Tell Us About Age-Related Macular Degeneration?Felix Grassmann, Thomas Ach, Caroline Brandl, Iris M. Heid,

and Bernhard H.F. Weber � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �73

Mitochondrial Genetics and Optic NeuropathyJaney L. Wiggs � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �97

Zebrafish Models of Retinal DiseaseBrian A. Link and Ross F. Collery � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 125

Angiogenesis and Eye DiseaseYoshihiko Usui, Peter D. Westenskow, Salome Murinello, Michael I. Dorrell,

Leah Scheppke, Felicitas Bucher, Susumu Sakimoto, Liliana P. Paris,Edith Aguilar, and Martin Friedlander � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 155

Optogenetic Approaches to Restoring VisionZhuo-Hua Pan, Qi Lu, Anding Bi, Alexander M. Dizhoor, and Gary W. Abrams � � � � 185

The Determination of Rod and Cone Photoreceptor FateConstance L. Cepko � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 211

Ribbon Synapses and Visual Processing in the RetinaLeon Lagnado and Frank Schmitz � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 235

Functional Circuitry of the RetinaJonathan B. Demb and Joshua H. Singer � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 263

vii

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.

VS01-FrontMatter ARI 30 October 2015 19:57

Contributions of Retinal Ganglion Cells to Subcortical Visual Processingand BehaviorsOnkar S. Dhande, Benjamin K. Stafford, Jung-Hwan A. Lim,

and Andrew D. Huberman � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 291

Organization of the Central Visual Pathways Following Field DefectsArising from Congenital, Inherited, and Acquired Eye DiseaseAntony B. Morland � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 329

Visual Functions of the ThalamusW. Martin Usrey and Henry J. Alitto � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 351

Neuronal Mechanisms of Visual AttentionJohn Maunsell � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 373

A Revised Neural Framework for Face ProcessingBrad Duchaine and Galit Yovel � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 393

Deep Neural Networks: A New Framework for Modeling BiologicalVision and Brain Information ProcessingNikolaus Kriegeskorte � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 417

Visual Guidance of Smooth Pursuit Eye MovementsStephen G. Lisberger � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 447

Visuomotor Functions in the Frontal LobeJeffrey D. Schall � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 469

Control and Functions of Fixational Eye MovementsMichele Rucci and Martina Poletti � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 499

Color and the Cone MosaicDavid H. Brainard � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 519

Visual AdaptationMichael A. Webster � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 547

Development of Three-Dimensional Perception in Human InfantsAnthony M. Norcia and Holly E. Gerhard � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � 569

ErrataAn online log of corrections to Annual Review of Vision Science articles may be found athttp://www.annualreviews.org/errata/vision

viii Contents

Ann

u. R

ev. V

is. S

ci. 2

015.

1:51

9-54

6. D

ownl

oade

d fr

om w

ww

.ann

ualr

evie

ws.

org

by

${in

divi

dual

Use

r.di

spla

yNam

e} o

n 11

/22/

15. F

or p

erso

nal u

se o

nly.

Documents

Color and the Cone Mosaiccolor.psych.upenn.edu/.../Brainard_2015_ColorAndMosaic.pdf · 2015-11-22 · as metamerism). Two physically different spectral irradiances are shown. These