Object correspondence across brief occlusion is established on the basis of both spatiotemporal and surface feature cues

Cognition 113 (2009) 150–166

Contents lists available at ScienceDirect

Cognition

journal homepage: www.elsevier .com/ locate/COGNIT

Object correspondence across brief occlusion is established on the basisof both spatiotemporal and surface feature cues

Andrew Hollingworth a,*, Steven L. Franconeri b

a Department of Psychology, The University of Iowa, 11 Seashore Hall E, Iowa City, IA 52242-1407, USAb Department of Psychology, Northwestern University, 2029 Sheridan Road, Evanston, IL 60208-2710, USA

a r t i c l e i n f o a b s t r a c t

Article history:Received 16 January 2009Revised 12 July 2009Accepted 1 August 2009

Keywords:Object perceptionObject persistenceObject correspondenceOcclusionDynamic vision

0010-0277/$ - see front matter � 2009 Elsevier B.Vdoi:10.1016/j.cognition.2009.08.004

* Corresponding author. Tel.: +1 319 335 2964; faE-mail address: [email protected]

The correspondence problem is a classic issue in vision and cognition. Frequent perceptualdisruptions, such as saccades and brief occlusion, create gaps in perceptual input. Howdoes the visual system establish correspondence between objects visible before and afterthe disruption? Current theories hold that object correspondence is established solely onthe basis of an object’s spatiotemporal properties and that an object’s surface feature prop-erties (such as color or shape) are not consulted in correspondence operations. In fiveexperiments, we tested the relative contributions of spatiotemporal and surface featureproperties to establishing object correspondence across brief occlusion. Correspondenceoperations were strongly influenced both by the consistency of an object’s spatiotemporalproperties across occlusion and by the consistency of an object’s surface feature propertiesacross occlusion. These data argue against the claim that spatiotemporal cues dominate thecomputation of object correspondence. Instead, the visual system consults multiplesources of relevant information to establish continuity across perceptual disruption.

� 2009 Elsevier B.V. All rights reserved.

1. Introduction

Although our visual perception of the world seems sta-ble and continuous, the input to vision is in constant flux.Objects move, the observer moves, objects become oc-cluded, and visual input is interrupted by frequent sac-cades and blinks. Such disruptions cause gaps in theavailability of perceptual information. A classic issue in vi-sual cognition is the means by which the visual systemestablishes correspondence between objects viewed acrosssuch disruptions. Consider the following scenario. As youare driving, there are three cars immediately in the roadahead, and you are following one of them to a new restau-rant. Your attention is distracted briefly by a nearby pedes-trian, and then you switch attention and gaze back to theview ahead. As before, three cars are visible. How doesthe visual system determine that these are indeed thesame three cars visible a moment ago? And, in particular,

. All rights reserved.

x: +1 319 335 0191.u (A. Hollingworth).

how does the visual system establish correspondence be-tween the particular ‘‘car to be followed” before and afterthe disruption? An error in correspondence might meanfollowing the wrong car to the wrong destination. A similarcorrespondence problem is generated every time onemakes an eye movement: how does the visual systemdetermine that an object at one retinal location before asaccade is the same object appearing at a different retinallocation after the saccade? Understanding how correspon-dence problems are solved in vision is central to under-standing how we perceive continuity across visualdisruptions, such as saccades, and how we track and mon-itor goal-relevant objects in the service of intelligentbehavior.

The visual system could solve correspondence problemsusing two broad categories of information. First, corre-spondence could be computed on the basis of spatiotempo-ral continuity across a disruption (Kahneman, Treisman, &Gibbs, 1992; for a review, see Scholl (2007)), with corre-spondence established if an object’s position over time isconsistent with the interpretation of a single, persisting

http://dx.doi.org/10.1016/j.cognition.2009.08.004

mailto:[email protected]

http://www.sciencedirect.com/science/journal/00100277

http://www.elsevier.com/locate/COGNIT

A. Hollingworth, S.L. Franconeri / Cognition 113 (2009) 150–166 151

entity. For example, before being distracted by the pedes-trian, the ‘‘to be followed” car’s, position, speed, and trajec-tory would be encoded. After the disruption, if there werean object present at the appropriate location given the pre-vious spatiotemporal properties of the tracked car, corre-spondence would be established, and that object wouldbe treated as a continuation of the tracked car. Second,the correspondence problem could be solved by surfacefeature continuity across a disruption (Hollingworth, Rich-ard, & Luck, 2008; Richard, Luck, & Hollingworth, 2008).When following the car, its perceptual features (white, 4-door, boxy) would be encoded. After a disruption, an objectmatching the remembered surface features would be trea-ted as a continuation of the previously tracked car. Ofcourse, both spatiotemporal and surface feature informa-tion could be consulted and perhaps weighted accordingto informativeness. If there were many cars in the roadahead moving rapidly through traffic, one might dependmore on surface feature properties than on spatiotemporalproperties, given difficultly in reliably tracking the posi-tions of the cars. In contrast, if there happened to be multi-ple white, boxy cars visible, one might depend more onspatiotemporal properties of the cars than on surface fea-ture properties, as surface feature properties would notefficiently distinguish between them.

Although either form of correspondence operation iscomputationally feasible, researchers have generally con-cluded that it is the spatiotemporal properties of an ob-ject that are used to establish correspondence across thebrief disruptions characteristic of dynamic vision (Flom-baum, Scholl, & Santos, 2009; Kahneman et al., 1992;Mitroff & Alvarez, 2007; Scholl, 2007). Across longer de-lays, surface feature information is undoubtedly used toestablish object mapping. One’s ability to determine thatthe dog greeting you in the evening is the same dogyou left in the morning clearly depends on memory forthe surface feature properties of the dog, a straightfor-ward example of object recognition. In the literature ondynamic object perception, however, the claim is thatwithin a single perceptual event, when one is mappingobjects across disruptions such as saccades or brief occlu-sion, spatiotemporal properties dominate in the computa-tion of object correspondence.

This conclusion has been driven by theory and byempirical evidence. The claim of spatiotemporal domi-nance follows directly from the most prominent theoryof object correspondence, the object-file framework ofKahneman et al. (1992). In this view, when an object ispresent in the visual field, a position marker is assignedto the location occupied by the object (similar to the spa-tial indexes proposed by Pylyshyn, 2000). If the objectmoves, the position marker ‘‘sticks” to the object, trackingthe object’s position over time (i.e., tracking its spatiotem-poral properties). Non-spatiotemporal properties of theobject (such as color, shape, identity, and so on) can bebound to the position marker, forming the content of theobject file.

In the object-file framework, object continuity acrosschange (such as motion) or disruption (such as a saccadeor brief occlusion) is established by a correspondence oper-ation that consults only the spatiotemporal properties of

the object. If an object’s spatiotemporal properties are con-sistent across disruption, the same object file assigned tothe object before the disruption will be assigned to the ob-ject after the disruption, thereby establishing continuity.However, the content of the file, including object surfacefeatures and identity, cannot be used to establish corre-spondence, because these properties can be retrieved fromthe file only after spatiotemporal correspondence has beenestablished (a process of content reviewing that occursafter the correspondence operation). In sum, brief objectmemory is proposed to be addressed by spatiotemporalproperties and not addressed by content (see also Flom-baum et al., 2009). Correspondence operations are there-fore limited to an analysis of the spatiotemporalproperties of objects.

Empirical evidence that object correspondence mecha-nisms consult only spatiotemporal information comesfrom the original object-reviewing paradigm developedby Kahneman et al. (1992). In this paradigm, participantssee two boxes. Preview letters appear briefly inside eachbox. The letters are removed, and the empty boxes moveto new positions. One test letter appears in a box, and par-ticipants name the letter. Naming latency is typically lowerwhen the letter appears in the same box as it had appearedoriginally versus when it appears in the other box. This ob-ject-specific benefit has been interpreted as indicating thatthe box in the test display was treated as the same objectthat had appeared at a different position at the beginningof the trial (i.e., that object correspondence was estab-lished). Specifically, the object-specific benefit suggeststhat a feature associated with the box at its original loca-tion (the preview letter) remains associated with the boxat its new location, facilitating letter naming when the testletter matches the letter associated with that box. In theterminology of Kahneman et al., letter naming is facilitatedwhen the test letter matches the content of the object fileat that location. The presence or absence of an object-spe-cific benefit across change or disruption therefore providesa measure of the extent to which object correspondencehas been established and an object has been treated bythe visual system as a single, persisting entity.

The object-specific benefit has been observed consis-tently when there is spatiotemporal information availableto establish object correspondence, such as when partici-pants observe linking motion between two locations ofan object (Kahneman et al., 1992). However, when surfacefeature properties are the only cues to object correspon-dence, several studies have found no indication of a sameobject benefit, suggesting that surface feature propertiesare not consulted in correspondence operations. In Exper-iment 6 of Kahneman et al. (1992), the letters that ap-peared in the boxes were drawn in different colors. Twoboxes were then moved, in a single step, to new locationsequidistant from the original box locations, rendering spa-tiotemporal properties non-informative. One letter ap-peared in a box, and the color consistency between thepreview and test letter was manipulated. Naming latencywas no faster when the preview and test color of the letterwere consistent than when they were inconsistent, sug-gesting that color was not used to establish correspon-dence between the preview letter and test letter.

T P

T

Spatiotemporal Condition

Surface Feature Condition

TT

P

Fig. 1. Illustration of the method used by Mitroff and Alvarez (2007).

152 A. Hollingworth, S.L. Franconeri / Cognition 113 (2009) 150–166

Mitroff and Alvarez (2007) modified this paradigm tomaximize the possibility of observing correspondence onthe basis of a surface feature match. Their method is illus-trated in Fig. 1. Participants first saw two objects. In thespatiotemporal condition, the objects were identical boxes,as in the basic Kahneman et al. (1992) paradigm. In the sur-face feature condition, the objects differed on a number ofsalient surface feature attributes, including color, size,shape, and the presence of a hole. Two letters were pre-sented briefly in the two objects, and the objects were thenmoved to new locations. In the spatiotemporal condition,linking motion between the two positions provided spatio-temporal cues to object correspondence. In the surface fea-ture condition, there was no linking motion; the two boxeswere moved in a single step to the new positions across ablank ISI, making spatial correspondence ambiguous. Thus,in the surface feature condition, surface feature informa-tion was the only available cue to object correspondence.A test letter was then presented in one of the two objects,and participants reported whether it was or was not one ofthe two letters appearing at the beginning of the trial.1

In the spatiotemporal condition of Mitroff and Alvarez(2007), when the test letter matched one of the previewletters, the test letter could appear either in the same ob-ject as it appeared in the preview display (as defined bythe motion history of the objects) or in the different object.Similarly, in the surface feature condition, the test lettercould appear in the same object in which that letterappeared in the preview display (as defined by the surfacefeature properties of the objects) or in the different object.An object-specific benefit was observed only in thespatiotemporal condition. That is, only spatiotemporalconsistency at test influenced recognition memory perfor-

1 This type of recognition memory task has replaced the original letternaming task of Kahneman et al. (1992) and is now standard in the object-file literature (Kruschke & Fragassi, 1996; Mitroff & Alvarez, 2007; Mitroff,Scholl, & Wynn, 2004; Mitroff, Scholl, & Wynn, 2005). The logic of therecognition memory task is essentially the same as that for the letternaming task. On trials when a test letter is presented within the sameobject as that letter appeared originally, comparison operations will berelatively efficient, generating an object-specific benefit on elapsed time torespond ‘‘same”.

mance; surface feature consistency produced no reliableeffect. Mitroff and Alvarez argued that a surface featurematch was not sufficient for the visual system to treat atest object as a continuation of one of the preview objectsand that the original object files (containing the previewletter information) were not systematically assigned to testobjects on the basis of a surface feature match. Conse-quently, there was no object-specific effect on recognitionmemory performance. Given the strength of their surfacefeature manipulation (the two objects differed on multiplesurface feature attributes and were highly distinguishableon the basis of these surface feature differences), the Mitr-off and Alvarez data provide the strongest evidence to datethat correspondence operations are limited to the spatio-temporal features of an object and do not consult an ob-ject’s surface features.

The assumption of spatiotemporal dominance is centralto understanding how brief forms of object representationare structured in memory and how the visual system isable to maintain coherent representations of dynamicallychanging environments. In addition, the assumption ofspatiotemporal dominance has been central to the transla-tion of the object file framework to other domains of per-ception and cognition, such as infant object perception(Feigenson & Carey, 2005; Xu, Carey, & Welch, 1999), mul-tiple-object tracking (Horowitz et al., 2007; Pylyshyn,2004; Pylyshyn & Storm, 1988; Scholl & Pylyshyn, 1999),and apparent motion (for a review, see Scholl, 2007). Webriefly discuss each of these literatures in turn.

In the infant cognition literature, evidence that younginfants tend to rely on spatiotemporal and numerical infor-mation when interpreting perceptual scenes has been ex-plained as early dependence on an object-file system thatdoes not consult surface feature or identity information(Leslie, Xu, Tremoulet, & Scholl, 1998). However, by oneyear of age (Xu & Carey, 1996; Xu et al., 1999) or even ear-lier (Moore, Borton, & Darby, 1978), infants can solve ob-ject correspondence on the basis of surface featureproperties as well. Although spatiotemporal mechanismsmay be primary in their developmental appearance, it doesnot necessarily follow that they are primary in adult corre-spondence operations. That is, cognitive processes that


arise relatively late in development need not be relegatedto a secondary role in adult cognition (language, whichdevelops relatively late, is certainly a central feature ofadult, human cognition). Thus, the infant cognition litera-ture does not strongly constrain theorizing about the con-tribution of spatiotemporal and surface features tocorrespondence operations in adults.

In work on multiple-object tracking, participants track aset of moving objects among other moving objects (Pyly-shyn & Storm, 1988). Object correspondence does indeedappear to be established on the basis of spatiotemporalproperties in this paradigm. Participants can successfullytrack the locations of multiple-objects without necessarilykeeping track of their surface feature properties (Horowitzet al., 2007; Pylyshyn, 2004; Scholl, Pylyshyn, & Franconeri,1999). However, dependence on spatiotemporal propertiesis not particularly surprising in a task that requires partic-ipants to explicitly track the locations of objects over time.

In the literature on apparent motion, motion correspon-dence can be perceived between two objects that differ insurface feature properties (Kolers, 1972), and ambiguousmotion tends to be resolved on the basis of spatiotemporalfeatures rather than on the basis of surface features (for areview, see Dawson, 1991), providing some of the stron-gest evidence in support of the spatiotemporal dominancehypothesis. Further, in the tunnel effect, an object passingbehind an occluder is often perceived as a single objectin motion if its spatiotemporal properties are consistentwith that interpretation, even if its surface features changeduring occlusion (Flombaum et al., 2009). Despite this evi-dence of spatiotemporal dominance in motion correspon-dence, Moore, Mordkoff, and Enns (2007; see also Mooreand Enns, (2004)) recently found that an abrupt changein the surface features of an object (such as a change in sizeor color) during a series of apparent motion frames blockedthe interpretation of correspondence, disrupting the per-ception of continuous motion. This occured despite thepresence of spatiotemporal information consistent withobject continuity. Thus, under some circumstances, surfacefeature properties can trump spatiotemporal properties inthe perception of motion.

Finally, the object-file framework has been translated tothe domain of correspondence across saccadic eye move-ments (Henderson, 1994; Henderson & Siefert, 2001; Irwin,1992; Irwin & Andrews, 1996; Irwin & Gordon, 1998; Ze-linsky & Loschky, 2005), but it has been done so withouta strong commitment to the assumption of spatiotemporaldominance (Currie, McConkie, Carlson-Radvansky, & Irwin,2000; Gordon, Vollmer, & Frankl, 2008; Hollingworth et al.,2008; Richard et al., 2008). Recently, Richard et al. testedthe relative contributions of spatiotemporal and surfacefeature information to object correspondence across sac-cades. The weighting of the two sources of informationwas governed by the demands of saccade target selection.If a saccade target was selected on the basis of its position,an object in the appropriate position after the saccadetended to be treated as the original target object. However,if a saccade target was selected on the basis of its surfacefeatures (such as its color), an object with the appropriatesurface features after the saccade tended to be treated asthe original saccade target.

In summary, the evidence relevant to assessing the spa-tiotemporal dominance hypothesis is ambiguous. In theoriginal object-reviewing paradigm (Kahneman et al.,1992; Mitroff & Alvarez, 2007) and in the multiple-objecttracking literature, spatiotemporal properties appear todominate correspondence operations, with little or no con-sultation of surface features. In the apparent motion liter-ature, spatiotemporal properties tend to be weightedmore heavily than surface feature properties, but this canbe reversed if there is a salient change in surface features(Moore, Mordkoff, & Enns, 2007). Finally, in the eye move-ment literature, there is little evidence of spatiotemporaldominance: Surface and spatiotemporal features are con-sulted flexibly on the basis of the demands of saccade tar-get selection (Richard et al., 2008).

2. The present study

The strongest evidence for spatiotemporal dominancecomes from the object-reviewing paradigm that originallygenerated object-file theory (Kahneman et al., 1992). Anygeneral account of the role of spatiotemporal and surfacefeature information in object correspondence must addressthis paradigm as one of the central domains of study. As re-viewed above, direct tests of surface feature consistency inthe object-reviewing paradigm have found no evidencethat surface features play any role in correspondence(Kahneman et al., 1992; Mitroff & Alvarez, 2007). However,two aspects of the design of these studies may have limitedtheir sensitivity to effects of surface feature consistency.

First, the spatiotemporal and surface feature conditionswere not equated for discontinuity on the two dimensions.Consider the Mitroff and Alvarez method illustrated inFig. 1. In the spatiotemporal condition, there was no dis-continuity in the surface feature properties of the objects;the two boxes retained their original surface featuresthroughout the trial. However, in the surface feature condi-tion, there was a salient discontinuity in spatial informa-tion, because the two objects were shifted, in a singlestep, to new locations, without traversing a path fromthe original location to the new location. The salient spa-tiotemporal discontinuity in the surface feature conditioncould have masked an effect of surface feature consistency.A similar spatiotemporal discontinuity was present in theKahneman et al. experiment probing color consistency:When the two boxes appeared at new locations, boxes re-mained visible at the original locations for 87 ms, introduc-ing strong spatiotemporal cues against correspondence. Ingeneral, then, to place spatiotemporal and surface featureinformation on an equal footing, the object-reviewing par-adigm would need to be modified so that salient spatio-temporal discontinuity was eliminated from conditionstesting the effect of surface feature consistency.

A second limitation of previous experiments testingsurface feature correspondence is the use of letters at thetest stimuli. Letters certainly can be encoded visually, butthey also can be encoded verbally. The absence of a surfacefeature consistency effect in Kahneman et al. (1992) andMitroff and Alvarez (2007) might have arisen because theletters and the surface feature properties of the objects in


which they appeared were stored in separate workingmemory systems (verbal working memory for the letters;VWM for the surface feature properties of the objects)and were not strongly bound together within an object-based, VWM representation (Luck & Vogel, 1997). To pro-vide a stronger test of surface feature correspondence,the test stimuli should require visual memory and shouldnot be easily encodable as verbal labels.

In this study, we developed a paradigm that probed therelative contributions of surface feature and spatiotempo-ral information to object correspondence, and we did soin a manner that eliminated any salient discontinuity onboth of the two dimensions. This was achieved by intro-ducing object manipulations during a brief period of occlu-sion, when they were not directly visible. In addition, we

Fig. 2. (A) Sequence of events in a trial of Experiment 1. Participants reportedShape-position consistency was manipulated. Note that in all of the figures inimprove visibility. In the experiments themselves, the color disks and shapes wedisks could have fit behind the occluder without touching). (B) Mean Correct RT irefers to the consistency between the locations of the test shapes and the expectein which they appeared. The stimuli inset in the bars of the graph show the tes

used unfamiliar shapes as the test stimuli and requiredarticulatory suppression throughout each trial, minimizingthe possibility that participants encoded the stimuli ver-bally, and thus providing a more direct test of correspon-dence on the basis of visual object representations.

In the basic method (see Figs. 2 and 3), two coloreddisks were presented to the left and right of a central oc-cluder. Novel shapes appeared in the two disks. The shapeswere removed, and the empty color disks moved behindthe occluder. The occluder was removed to display thetwo color disks containing two shapes. Both shapes werethe same as in the preview, or one was replaced by anew shape. Participants reported ‘‘same” or ‘‘new”. Whenboth test shapes were the same as in the preview, spatio-temporal consistency was manipulated (the test shapes

whether the two test shapes were the same as the two preview shapes.this paper, the color disk and novel shape stimuli have been enlarged tore significantly smaller relative to the size of the occluder (a total of eightn each of the consistency conditions of Experiment 1. Position consistencyd locations of the test shapes given the spatiotemporal history of the diskst configuration in each condition for the illustrated sample trial.

Fig. 3. (A) Sequence of events in a trial of Experiment 2. (In the print version of this figure, the two disk colors are represented as light grey and dark grey.)The consistency between the binding of novel shape and colored disk in the preview display and in the test display was manipulated. (B) Mean Correct RT ineach of the consistency conditions of Experiment 2. The stimuli inset in the bars of the graph show the test configuration in each condition for the illustratedsample trial.


appeared in locations that were either consistent or incon-sistent with the motion of the disks), and surface featureconsistency was manipulated (the test shapes appearedin disks with colors that were either consistent or inconsis-tent with the colors of the disks in which they originallyappeared). If object correspondence operations consultonly spatiotemporal information, as held by the spatiotem-poral dominance hypothesis, significant effects of spatio-temporal consistency should be observed, but no effectsof color consistency should be observed.

The present study consisted of five experiments. InExperiment 1, we manipulated spatiotemporal consistencyto ensure that our paradigm was sensitive to the spatio-temporal effects observed in previous object file experi-ments. In Experiments 2–5, we manipulated surfacefeature consistency (color), and these experiments imple-

mented increasingly stringent conditions for observingsurface feature effects on object correspondence. To pre-view the results, we found robust effects of both spatio-temporal and color consistency on the mapping ofobjects across brief occlusion, demonstrating that bothsources of information contribute to object correspon-dence. Contrary to claims of spatiotemporal dominance,surface feature consistency influenced object correspon-dence under a wide range of circumstances. Surface fea-tures were used to establish object correspondence whenspatiotemporal information was ambiguous (Experiment2), when spatiotemporal information was perfectly predic-tive of correspondence (Experiment 3), when spatiotempo-ral information was inconsistent with the interpretation ofcorrespondence on the basis of a color match (Experiment4), and even when there was salient spatiotemporal dis-


continuity directly in conflict with an interpretation of ob-ject continuity (Experiment 5).

3. Experiment 1

Before testing the role of surface features in object cor-respondence, we first sought to confirm that our modifiedversion of the object-reviewing paradigm was sensitive tospatiotemporal consistency. The method, illustrated inFig. 2A, was similar to the classic object-reviewing para-digm of Kahneman et al. (1992), and we expected to repli-cate the basic spatiotemporal consistency effect: aperformance advantage when the test stimuli appearedin positions consistent with their spatiotemporal history.

Each trial began with two blue disks appearing to theleft and right of a central occluder. Two novel shapes ap-peared in the disks briefly. Then, the empty disks moveddiagonally (one up, one down), followed by horizontal mo-tion that caused them to disappear behind the occluder atdifferent vertical positions. The occluder was removed toreveal the two blue disks containing two test shapes (both‘‘same” or one ‘‘new”). The test shapes appeared in posi-tions that were either consistent or inconsistent with theexpected locations given the motion of the disks. Considerthe sample trial illustrated in Fig. 2. The stimuli inset in thebars of the graph show the test configuration on a ‘‘same”trial in each condition. In the consistent-position condition,the test shape that originally appeared in the left disk ispresented in the bottom position, which is consistent withthe movement of the left disk to the bottom position. Theposition of the other test shape is also consistent withthe motion of the disk in which it appeared. In the incon-sistent-position condition, the two test shapes appear inthe inappropriate positions given the motion of theirrespective disks. Note that surface feature correspondencewas ambiguous, because the two color disks had identicalsurface features.

As is standard in this literature, the data of interestcame primarily from ‘‘same” trials. On ‘‘new” trials, thecorrespondence between the new test shape and the pre-view display is undefined, complicating their interpreta-tion. Our task is equivalent to the recognition memorytask of recent object file studies (Kruschke & Fragassi,1996; Mitroff & Alvarez, 2007; Mitroff et al., 2004; Mitroffet al., 2005), except that we presented two test shapes in-stead of one test letter. Our method required two compar-isons of test shapes with memory to successfullydetermine ‘‘same”, each of which could have been influ-enced by the surface feature or spatiotemporal propertiesof the object in which the test shape appeared. This meth-od was designed to maximize consistency effects, whichwere more than twice as large in the present study thanin previous object-file experiments.

3.1. Method

3.1.1. ParticipantsParticipants in all experiments were recruited from the

University of Iowa community and were between the agesof 18 and 30. Participants reported normal or corrected-

to-normal vision. They either received course credit orwere paid. Twenty participants completed Experiment 1.

3.1.2. Stimuli and apparatusStimuli are shown in Fig. 2A. The background was gray.

Color disks subtended 1.59�. All disks were blue. Five novelshapes were created in a computer graphics program. Eachshape was white and was centered within the disk. Oneach trial, the two shapes that appeared in the preview dis-play were chosen randomly without replacement from theset of five. The transition between shape and disk wasabrupt (i.e., not antialiased). Shapes subtended an averageof 1.16� horizontally and 1.14� vertically. The central, blackoccluding rectangle subtended 4.26� � 6.82�.

The initial positions of the two color disks were cen-tered 7.1� to the left and right of the screen center. The mo-tion animation was composed of 51 frames at a rate of20 ms/frame. The two disks first moved on a diagonal path,with one moving upward and the other downward (hori-zontal displacement = 1.19�; vertical displacement =1.42�), assigned randomly. The diagonal motion was com-pleted in 200 ms. Thereafter, the two objects moved onhorizontal paths until they were completely occluded,which required an additional 820 ms. The vertical posi-tions of the two objects were separated by 2.84� (center-to-center).

In the test image, two color disks were presented at thecenter of the display, separated vertically by 2.84�. Twoshapes were presented within the disks. On ‘same’ trials,the two shapes at test were the same two shapes appearingin the preview display. On ‘new’ trials, one of the two pre-view shapes (randomly chosen) was replaced by a newshape (randomly chosen from the remaining three shapes).The locations of the two shapes were either consistent orinconsistent with the expected locations given the motionof the disks. Because the two disks always had the samecolor, color was non-informative with respect to the shapememory task.

Stimuli were displayed on a 17-in CRT monitor with a100 Hz refresh rate. A forehead rest was used to maintaina constant viewing distance of 80 cm. Responses were col-lected using a serial button box. The experiment was con-trolled by a computer running E-Prime software.

3.1.3. ProcedureOn each trial, participants first saw four randomly se-

lected digits and repeated the digits aloud throughoutthe trial to further minimize verbal encoding of the stimuli(the experimenter monitored digit repetition to ensurethat participants complied). The participant pressed a but-ton to begin the trial. There was a 700 ms blank delay fol-lowed by the sequence of events depicted in Fig. 2A.Because the color disks were visible before, during, andafter the appearance of the novel shapes, the experiencewhen viewing these displays was of two objects (the colordisks) that temporarily acquired a new feature (the novelshape) and then moved behind the occluder. During mo-tion and at the point of occlusion, the only means to dis-criminate the two objects was by virtue of theirspatiotemporal properties. After the appearance of the testdisplay, participants pressed one of two buttons to respond


that both shapes were the same as in the preview or thatone shape was new. Participants were instructed to re-spond as quickly and as accurately as possible. In eachexperiment, participants were also instructed that thepositions of the shapes and the colors of the disks in whichthey appeared were entirely irrelevant to the task, and thatthey should base their decision solely on the shapesthemselves.

Experiment 1 was a 2 (spatiotemporal consistency) � 2(same, new) design. Participants completed a practiceblock of 12 trials, drawn randomly from the full design, fol-lowed by an experiment block of 280 trials (70 in each con-dition). Trial order was determined randomly. The entiresession lasted approximately 45 min.

3.2. Results and discussion

As is conventional, analyses were conducted over‘‘same” trials, in which both test shapes had been presentin the preview. ‘‘New” trials introduced a new shape, andthe relationship between this test shape and the colordisks was undefined. For all experiments, analyses overall trials produced the same pattern of results as analysesover same trials alone. The complete data are reported inthe Appendix. RT analyses were limited to correct re-sponses. In addition, responses more than 2.5 SD from eachparticipant’s mean RT in each condition were eliminated asoutliers (no more than 2.7% of the data in any experiment).RT trimming did not alter the pattern of results in anyexperiment. Mean correct RT data are reported in Fig. 2B.

The purpose of Experiment 1 was to ensure that wecould observe the standard effect of spatiotemporal consis-tency found originally by Kahneman et al. (1992). Indeed, arobust effect of spatiotemporal consistency was obtained.Mean RT was 73 ms faster in the consistent-position condi-tion than in the inconsistent-position condition, t(19) =7.25, p < .001. In addition, there was a complementary2.2% effect on accuracy, t(19) = 2.25, p = .04. An effect onaccuracy, in addition to the relatively large effect on RT,indicates that the present method is a highly efficientmeans to assess object correspondence operations.

Before proceeding to examine the role of surface fea-tures in object correspondence, it is important to note thatthere exists some ambiguity in the object-reviewing para-digm developed by Kahneman et al. (1992) and adaptedhere. Correspondence is measured by the response to a teststimulus (typically a letter; in the present case, novelshapes) that is associated with the object whose continuityis being manipulated (typically a box; in the present case,colored disks). The assessment of correspondence is there-fore indirect. It is possible that participants track the ob-jects on the basis of the test feature (the letters or novelshapes) and use a test feature match to establish corre-spondence. The association of the test stimulus to a loca-tion could then influence the response to the teststimulus, generating the consistency effect, but withoutnecessarily influencing the computation of object corre-spondence per se. The same issue applies to manipulationsof surface feature consistency (Kahneman et al., 1992;Mitroff & Alvarez, 2007; present Experiments 2–5). Theassociation of the test stimulus to a particular set of surface

features could potentially generate a consistency effectindependently of the computation of correspondence.

Although possible, we think this alternative account isunlikely to explain the spatiotemporal consistency effectsobserved here and in earlier experiments or the surfacefeature consistency effects reported in Experiments 2–5.The novel shapes were clearly perceived as temporary fea-tures of otherwise stable objects. The colored disks werevisible for a full second before the appearance of the pre-view shapes. The preview shapes disappeared 500 ms be-fore the objects started to move. And, critically, duringthe motion sequence and disappearance behind the occlu-der (the events that created the need to establish corre-spondence), the shapes were not visible. It therefore doesnot seem likely that participants ‘‘tracked” the objectsand established correspondence on the basis of a previ-ously visible feature (the preview shape) rather than onthe basis of features directly visible throughout the trialuntil the critical occlusion event (the spatiotemporal andsurface feature properties of the objects).

4. Experiment 2

Experiment 2 constituted our first test of a surface fea-ture contribution to object correspondence and thus thefirst test of the spatiotemporal dominance hypothesis. Inthe standard object-reviewing paradigm (Kahnemanet al., 1992), replicated in Experiment 1, spatiotemporalconsistency is manipulated in the context of ambiguoussurface feature information, because the two objects haveidentical surface features. Spatiotemporal information istherefore the only available cue to correspondence. InExperiment 2, we developed an analogous test of surfacefeature contributions to object correspondence: Surfacefeature (color) consistency was manipulated in the pres-ence of ambiguous spatiotemporal information, and sur-face feature information was the only available cue tocorrespondence. The method is illustrated in Fig. 3A.

In the preview display, the two disks differed in color.After the presentation of the preview shapes, the two colordisks moved initially on quasi-random paths. Then, bothobjects disappeared behind the occluder while travelinghorizontally at the same vertical position. In the test dis-play, the two disks were separated vertically, as in Experi-ment 1. The presentation of vertically separated items attest was consistent with the history of the display, becausethe objects had a history of moving up or down erratically.However, spatiotemporal correspondence was ambiguous,because the spatiotemporal history of the display did notspecify the relative locations of the two objects in the testdisplay. Of central importance, the colors of the disks inwhich the test shapes appeared were either consistent orinconsistent with the color-shape pairings in the preview.Consider the sample trial in Fig. 3. In the color-consistentcondition, the shape that originally appeared within thered disk (depicted in light gray in the print version of thefigure) is presented within the red disk at test. Color is alsoconsistent for the shape that originally appeared in thepurple disk (depicted in dark grey in the print version ofthe figure). In the color-inconsistent condition, however,


the pairings between test shapes and color discs are re-versed, and both test shapes appear within colors thatare inconsistent with their preview colors.

If surface features can serve to establish object corre-spondence across brief occlusion, a performance advantageshould be observed for the color-consistent condition overcolor-inconsistent condition. Such evidence would demon-strate that correspondence operations are not limited to spa-tiotemporal features and that surface features are consulted,at least under conditions of spatiotemporal ambiguity.

4.1. Participants

Twenty new participants completed Experiment 2.

4.2. Stimuli, apparatus, and procedure

The method in Experiment 2 was the same as that inExperiment 1, with the following exceptions. In Experi-ment 2, the colors of the two disks were different. Each col-or was chosen randomly without replacement from a set offive colors: blue, green, red, purple, and yellow. In the testdisplay, the colors of the disks in which the test shapes ap-peared were either consistent or inconsistent with the pre-view colors. In the latter case, the pairings of novel shapeand color were swapped from preview to test.

The motion sequence consisted of a series of diagonaland horizontal motion components. Each diagonal compo-nent displaced the object 0.91� horizontally and 2.3� verti-cally over 200 ms of animation. To smooth the transitionbetween diagonal components, each diagonal componentwas separated by 0.23� of horizontal motion over 40 ms.The motion sequence started with the first diagonal compo-nent, with the vertical direction (up or down) randomlychosen for each object. The second diagonal componentbrought the object back to the vertical midline. The direc-tion of the third diagonal component (up or down) wasagain randomly chosen for each object. And the fourth diag-onal component again brought the objects back to the ver-tical midline. The last component was horizontal motionof 1.59� over 180 ms, which ended with the full occlusionof both objects at the vertical midline. This entire motion se-quence required 1100 ms. What the observer perceived wasthe unpredictable upward and downward diagonal motionof the objects, followed by horizontal motion that causedthe objects to be occluded at the same vertical position.After the 1100 ms motion sequence, there was a 400 msperiod of occlusion before the test display. This periodwas easily sufficient to support the possibility of anotherdiagonal motion behind the occluder, consistent with theappearance of the test objects at different vertical positions.

The test positions of the two color disks were randomlychosen on each trial. The pairings of test shapes and colordisks were either consistent with the shape-color pairingsin the preview display or inconsistent with those pairings.


Mean correct RT data are reported in Fig. 3B. Mean RTwas 32 ms faster in the color-consistent condition than inthe color-inconsistent condition, t(19) = 3.25, p = .004.

There was a complementary, but nonsignificant, 0.9% effecton accuracy, t(19) = 0.83, p = .42.

The results of Experiment 2 demonstrate that surfacefeature consistency can be used to establish object corre-spondence when spatiotemporal correspondence is ambig-uous. They stand in contrast with the results of Mitroff andAlvarez (2007) and Kahneman et al. (1992), in which no ef-fect of surface feature consistency was observed. The po-tential causes of the difference between our results andthose of previous studies are examined further in Experi-ment 5.

The surface feature consistency effect observed inExperiment 2 eliminates a strong version of the spatiotem-poral dominance hypothesis holding that only spatiotem-poral information is consulted by correspondenceoperations (Kahneman et al., 1992). However, the resultsof Experiment 2 are potentially consistent with a weakerversion of the spatiotemporal dominance hypothesis. Inthis view (Flombaum et al., 2009; Scholl, 2007), surfacefeatures can be consulted in the computation of object cor-respondence, but only when spatiotemporal information isambiguous or non-informative. That is, surface featuresmay be used only as a ‘‘last resort” if correspondence can-not be established on the basis of spatiotemporal cues.When spatiotemporal and surface feature information areboth potentially informative, spatiotemporal informationwill ‘‘trump” surface feature information in correspon-dence operations, and no effect of surface feature consis-tency should be observed. In Experiments 3 and 4, wetested this weaker version of the spatiotemporal domi-nance hypothesis by manipulating surface feature consis-tency in the presence of predictive, rather thanambiguous, spatiotemporal information.

5. Experiment 3

In Experiment 3, the color manipulation of Experiment2 was combined with the unambiguous motion used inExperiment 1 (see Fig. 4). Two differently colored disks ap-peared to the left and right of the occluder. They thenmoved behind the occluder at different vertical positions(as in Experiment 1). Spatiotemporal history served to dis-tinguish the two objects and could have been used toestablish correspondence after occlusion. In addition, thetest shapes always appeared in locations consistent withthe motion history of the two disks, so spatiotemporalinformation also perfectly predicted the locations of thetest shapes. Color consistency was manipulated within thispredictive spatiotemporal context. In the color-consistentcondition, the test shapes appeared in disks that matchedthe colors of the preview disks in which the shapes origi-nally appeared. In the color-inconsistent condition, bothtest shapes appear within colors that were inconsistentwith their preview colors.

In this method, correspondence could have been estab-lished on the basis of spatiotemporal information alone.Thus, the hypothesis that surface features are consultedonly when spatiotemporal information is ambiguous(Flombaum et al., 2009; Scholl, 2007) predicts that spatio-temporal information should dominate correspondence

Fig. 4. Experimental design and mean correct RT data for Experiments 3 (A) and 4 (B). (In the print version of this figure, the two disk colors are representedas light grey and dark grey.) For each experiment, the left-hand portion of the figure shows the key events in a sample trial. Color consistency refers to theconsistency between the binding of novel shape and colored disk in the preview display and in the test display. Position consistency refers to theconsistency between the locations of the test shapes and the expected locations of the test shapes given the spatiotemporal history of the disks in whichthey appeared. The stimuli inset in the bars of the graph show the test configuration in each condition for the illustrated sample trial.


operations, and color should not be consulted. If so, thenthe consistent-color advantage observed in Experiment 2should be eliminated in Experiment 3.

5.1. Method

5.1.1. ParticipantsTwenty new participants completed Experiment 3.

5.1.2. Stimuli, apparatus, and procedureThe motion sequence was identical to that in Experi-

ment 1. In the test display, the test shapes always appearedin the locations consistent with the motion of the disks. Asin Experiment 2, the colors of the two disks differed, andcolor consistency was manipulated at test. In the test dis-play, the colors of the disks in which the test shapes ap-peared were either consistent or inconsistent with thepreview colors. In the latter case, the pairings of novelshape and color were swapped from preview to test.


Mean correct RT data are reported in Fig. 4A. Contrary tothe claim that object correspondence operations aredominated by spatiotemporal information when spatio-temporal information is unambiguous, a robust colorconsistency effect was observed. Mean RT was 52 ms faster

in the color-consistent condition than in the color-incon-sistent condition, t(19) = 4.42, p < .001. There was a com-plementary 3.5% effect on accuracy, t(19) = 2.94, p = .008.

To ensure that color consistency effects were not causedby differences in the contrast between the shape and thecolor on which the shape appeared (a possibility, becausecolors differed in luminance), we conducted a controlexperiment (N = 20). Color disks were made larger (2.57�diameter), and the shapes always appeared within a darkgray disk (1.59�) centered within the larger color disk, con-trolling shape-background contrast (see Fig. 5). Otherwise,the method was identical to that in Experiment 3. A robustcolor consistency effect was again observed. Mean RT was42 ms faster in the color-consistent condition (885 ms)than in the color-inconsistent condition (927 ms),t(19) = 2.82, p = .01, with a complementary 3.6% effect onaccuracy, t(19) = 4.26, p < .001.

In Experiment 3, spatiotemporal history distinguishedthe locations of the objects after occlusion and thus wasinformative of correspondence. In addition, the test shapesalways appeared in the appropriate locations given themotion of the disks. Spatiotemporal information perfectlypredicted test shape, whereas color did not predict testshape (because on half the trials, color-shape bindingwas consistent, and on the other half, it was inconsistent).If spatiotemporal properties ‘‘trump” surface featureproperties when spatiotemporal properties are informa-

Fig. 5. Sample stimuli used in the Experiment 3 control.


tive, correspondence should have been established on thebasis of spatiotemporal information in Experiment 3, andcolor should not have influenced correspondence. Yet, a ro-bust effect of color consistency was observed. This resultcannot be accommodated by a theory holding that, in gen-eral, spatiotemporal information trumps surface featureinformation in correspondence operations (although it isstill possible that spatiotemporal information might trumpsurface feature information in other forms of correspon-dence, a topic addressed in the General Discussion).

6. Experiment 4

In Experiment 4, we expanded the design of Experiment3 to manipulate both spatiotemporal and color consistencyindependently (see Fig. 4B), allowing us to assess the rela-tive contributions of surface feature and spatiotemporalinformation to object correspondence in this paradigm. Inthe preview display, the novel shapes appeared withintwo disks that differed in color. In the test display, colorand position were either both consistent, one consistentand the other inconsistent, or both inconsistent.

The design of Experiment 4 provides two means to as-sess the contributions of spatiotemporal and surface fea-ture information to object correspondence. Similarly toExperiments 1–3, each of the conditions in which onlyone of the two dimensions is inconsistent (two middle barsof Fig. 4B) can be compared against the condition in whichboth are consistent (left-most bar of Fig. 4B). These con-trasts assess the performance decrement associated withintroducing inconsistency on one of the two dimensions.We expected to replicate the results of Experiments 1–3.The second method is to compare each of the conditionsin which only one of the two dimensions is consistent(two middle bars of Fig. 4B) against the condition in whichboth are inconsistent (right-most bar of Fig. 4B). These con-trasts assess the performance benefit associated with intro-ducing consistency on one of the two dimensions.

The latter type of comparison (against the both-incon-sistent baseline) allowed us to test the effect of consistencyon one dimension when the other dimension was inconsis-tent with the interpretation of correspondence. This pro-vided a particularly strong test of the role surface featuresin object correspondence. If an effect of color consistencywere to be observed in this comparison, it would indicatea contribution of surface feature information to correspon-dence operations even when spatiotemporal informationwas inconsistent with the interpretation of correspon-dence. Likewise, an effect of spatiotemporal consistency

would indicate a contribution of spatiotemporal informa-tion to correspondence operations even when surface fea-ture information was inconsistent with correspondence.

6.1. Method

6.1.1. ParticipantsForty new participants completed Experiment 4.

6.1.2. Stimuli, apparatus, and procedureThe method was identical to Experiment 3, except that

the test shapes could appear either at the consistent orinconsistent positions given the spatiotemporal history ofthe objects, and the color of the disks in which the testshapes appeared was either consistent or inconsistent withthe shape-color pairing in the preview display. Experiment3 was a 2 (spatiotemporal consistency) � 2 (color consis-tency) � 2 (same, new) design. Participants completed apractice block of eight trials followed by an experimentblock of 336 trials (42 in each condition). The entire sessionlasted approximately 55 min.


Mean correct RT data are reported in Fig. 4B.We first compared each of the conditions in which one

of the two dimensions was inconsistent against the both-consistent condition. Replicating Experiment 1, mean RTwas 69 ms faster in the both-consistent condition than inthe position-inconsistent/color-consistent condition,t(39) = 3.75, p < .001, with a complementary, but nonsig-nificant, 1.4% effect on accuracy, t(39) = 1.29, p = .20. Repli-cating Experiments 2 and 3, mean RT was 46 ms faster inthe both-consistent condition than in the position-consis-tent/color-inconsistent condition, t(39) = 4.47, p < .001,with a complementary, but nonsignificant, 1.3% effect onaccuracy, t(39) = 1.23, p = .22. The magnitude of the posi-tion effect (69 ms) and color effect (46 ms) did not differ,t(39) = 1.38, p = .18.

Next, we examined whether consistency on one dimen-sion would influence correspondence despite evidenceinconsistent with correspondence on the other dimension.Each of the conditions in which one of the two dimensionswas consistent was compared against the both-inconsis-tent condition. There was a reliable effect of spatiotempo-ral consistency. Mean RT was 61 ms faster in the position-consistent/color-inconsistent condition than in the both-inconsistent condition, t(39) = 3.47, p = .001, with a com-plementary 3.0% effect on accuracy, t(39) = 2.78, p = .008.In addition, there was a reliable effect of color consistency.Mean RT was 37 ms faster in the position-inconsistent/col-or-consistent condition than in the both-inconsistent con-dition, t(39) = 2.88, p = .006, with a complementary 2.9%effect on accuracy, t(39) = 2.55, p = .01.

The results of Experiment 4 demonstrate that the effectof consistency on one dimension is not strongly con-strained by consistency on the other dimension. An effectof color consistency was observed when spatiotemporalinformation was consistent with correspondence, and aneffect of color consistency was observed when spatiotem-poral information was inconsistent with correspondence.

2 We thank an anonymous reviewer for suggesting this possibility.


Similarly, an effect of spatiotemporal consistency was ob-served regardless of color consistency. An alternative wayto frame this point is to note that the combined effect ofcolor and spatiotemporal consistency was larger than theeffect of either type of consistency alone.

In addition to this main finding, Experiment 4 elimi-nates a potential concern with the design of Experiment3. In Experiment 3, performance may have been influencedby the consistency between the pre-occlusion and post-occlusion positions of the colors themselves. In the color-consistent condition of Experiment 3, the colors appearedin the appropriate locations given the spatiotemporal his-tory of the objects. However, in the color-inconsistent con-dition, the colors appeared in locations that wereinconsistent with the spatiotemporal history of the objects.Consider Fig. 4A. In the color-consistent condition, the redand purple disks (light grey and dark grey in the print ver-sion of the figure) are in the appropriate locations given themotion of the objects, but in the color-inconsistent condi-tion, they are not. (Of course, the test shapes themselveswere always in the appropriate locations.) It is thereforepossible that the effect in Experiment 3 was driven notby color-shape consistency but rather by color-positionconsistency. Note that this alternative account would stillbe problematic for the spatiotemporal dominance hypoth-esis, because, under that view, color should not determinecorrespondence at all.

Nevertheless, the results of Experiment 4 eliminate thisalternative explanation in terms of color-position consis-tency. Two conditions of Experiment 4 are relevant, the po-sition-inconsistent/color-consistent condition and theboth-inconsistent condition (two right-hand bars ofFig. 4B). These two conditions differ in the consistency ofthe shape-color pairings; shape-position pairing is incon-sistent in both. Comparison of these two conditions there-fore probes the effect of color consistency against aposition-inconsistent baseline. As reported above, therewas a reliable 37 ms effect of color-shape consistency onRT, and a reliable 2.9% effect on accuracy. This effect cannotbe explained by differences in the position of the colorsthemselves. In the both-inconsistent condition, the colorsappeared in the appropriate locations given motion beforeocclusion. In the position-inconsistent/color-consistentcondition, the colors appeared in the inappropriate loca-tions. Yet, a reliable advantage was found for the lattercondition over the former, eliminating the possibility thatit was the positions of the colors that was driving the ef-fect. We can therefore be confident that the color effectsin Experiments 3 and 4 were caused by the consistency be-tween color and test shape.

Finally, we have interpreted the results of Experiment 4as indicating that both spatiotemporal and surface featureconsistency contributed to the computation of object cor-respondence. It is possible, however, that correspondencewas established initially on the basis of the spatiotemporalfeatures of the objects, and that the effects of color consis-tency were generated later, only after spatiotemporal cor-respondence had already been computed. Similarly, it ispossible that correspondence was established initially onthe basis of surface feature cues, and that the effects of spa-tiotemporal consistency were generated only after corre-

spondence had already been established. Although thesealternatives are possible within the context of Experiments3 and 4 (in which both types of cues were potentially infor-mative of correspondence), the effects of spatiotemporaland color consistency in Experiment 1 and 2 could nothave been generated after correspondence had alreadybeen computed, because in these experiments, spatiotem-poral cues (Experiment 1) and color cues (Experiment 2)were the only cues available upon which to establish cor-respondence in the first place. Thus, the full data set indi-cates that both spatiotemporal and surface feature cuescan contribute to the computation of object correspon-dence and that effects of consistency on the two dimen-sions are largely independent.

7. Experiment 5

In the experiments so far, the objects always moved be-hind the occluder, and thus there were clear perceptualcues to indicate that the objects should be found behindthe occluder in the test display. It is possible that the effectsof color consistency were observed in the present experi-ments because motion provided a plausible spatiotemporalaccount of how the objects came to be found behind the oc-cluder. That is, surface features may contribute to objectcorrespondence only when there is coherent spatiotempo-ral information to account for their general locations.2

To address this possibility, in Experiment 5 we elimi-nated the motion information from the paradigm, elimi-nating any plausible spatiotemporal account of thedifference in location between the preview and test ob-jects. Experiment 5 therefore constituted a particularlystringent test of surface feature correspondence: Will sur-face feature consistency be sufficient to override spatio-temporal information that is directly at odds with aninterpretation of object continuity?

The Experiment 5 method (Fig. 6A) was essentially thesame as in previous experiments, except there was no ob-ject motion. The color disks disappeared from their posi-tions flanking the occluder, and 500 ms later, theoccluder was removed to reveal the color disks at the cen-ter of the screen. There was no plausible spatiotemporalaccount for the change in location, because there was nomotion to link the preview and test positions of the colordisks. Color consistency was again manipulated. If surfacefeatures are consulted only when spatiotemporal informa-tion provides a plausible link between the original and newlocations of the objects, no effect of color consistencyshould be observed.

The elimination of motion generated a paradigm verysimilar to the paradigm of Mitroff and Alvarez (2007);see Fig. 1. Just as in Mitroff and Alvarez, the two objectswere moved to new locations without any linking motion.That is, the Experiment 5 paradigm introduced a salientspatiotemporal discontinuity, allowing us to test whetherthe presence/absence of spatiotemporal discontinuity isthe primary cause of the difference between our findingsand those of Mitroff and Alvarez.

700

750

800

850

900

950

1000

Color Disks, 500 ms

Color Disks, 500 ms

Preview Shapes, 1000 ms

Disks Removed, 500 ms

Test Shapes

Res

pons

e Ti

me

(ms)

74 ms**

** p < .001

Test Shape Consistency

ConsistentNA

Color:Position:

InconsistentNA

A B

Fig. 6. (A) Sequence of events in a trial of Experiment 5. (In the print version of this figure, the two disk colors are represented as light grey and dark grey.)(B) Mean correct RT in each of the consistency conditions of Experiment 5. The stimuli inset in the bars of the graph show the test configuration in eachcondition for the illustrated sample trial.


7.1. Participants

Twenty new participants completed Experiment 5.

7.2. Stimuli, apparatus, and procedure

The method was the same as that in Experiment 2, withthe following exceptions. The initial positions of the two col-or disks were centered 4.3� to the left and right of the screencenter. After the preview shapes had been removed, the colordisks remained visible for 500 ms. They were then removedfor 500 ms (occluder only), followed by the test display.


Mean correct RT data are reported in Fig. 6B. Mean RT was74 ms faster in the color-consistent condition than in the col-

or-inconsistent condition, t(19) = 5.99, p < .001. There was acomplementary 5.3% effect on accuracy, t(19) = 3.30, p = .003.

In Experiment 5, a highly robust color consistency effectwas observed despite the fact that there was no spatiotem-poral link between the preview and test locations of theobjects, and thus no plausible spatiotemporal account ofhow the objects came to be behind the occluder. It appearsthat surface features can establish object correspondenceeven when there is direct spatiotemporal information thatis inconsistent with object continuity. This in turn suggeststhat the critical difference between the present study andthat of Mitroff and Alvarez is not necessarily the presenceof a salient spatiotemporal discontinuity, because Experi-ment 5 revealed a color consistency effect despite salientspatiotemporal discontinuity.

It is difficult to determine precisely why we found sur-face feature consistency effects, whereas Mitroff and Alva-


rez did not. We think the most plausible explanation is thatthe present paradigm was simply more sensitive to consis-tency manipulations (both of spatiotemporal and surfacefeature information) than was the method of Mitroff andAlvarez. First, novel shapes and articulatory suppressionin the present study minimized verbal encoding of the teststimuli and maximized the possibility that the features ofthe objects and the test stimuli were integrated into acoherent object representation in VWM (Luck & Vogel,1997), increasing sensitivity to manipulations of the bind-ing between object features and test shapes. Second, thepresentation of two test stimuli, instead of one, potentiallymultiplied the effects of consistency compared with previ-ous experiments using only a single test stimulus. This in-creased sensitivity was observed not only in the ability todetect effects of surface feature consistency but also inthe magnitude of the spatiotemporal consistency effects,which were approximately three times as large in the pres-ent study as in Mitroff and Alvarez (and other experimentsusing letters and a single test stimulus).

8. General discussion

In the present experiments, we examined the contribu-tions of spatiotemporal and surface feature information tothe computation of object correspondence across a briefperiod of occlusion. The object-file framework of Kahnemanet al. (1992) holds that object representations are addressedby their spatiotemporal properties and that spatiotemporalinformation is primary in determining object correspon-dence, a spatiotemporal dominance hypothesis. In five exper-iments modeled after the Kahneman et al. object-reviewingparadigm, we observed that object correspondence opera-tions were not necessarily restricted to spatiotemporal cues.A salient surface feature cue, color, was used to establish ob-ject correspondence across brief occlusion. Robust effects ofcolor consistency on object correspondence were observedacross a series of experiments that instituted progressivelymore stringent demands on surface feature correspondence.In Experiment 2, object correspondence was established onthe basis of surface feature cues when spatiotemporal infor-mation was ambiguous and color was the only feature avail-able to distinguish the objects. In Experiment 3, color wasconsulted in object correspondence operations despiteunambiguous spatiotemporal information that could havebeen used to compute correspondence. In Experiment 4,the contributions of spatiotemporal and surface featureinformation to object correspondence were found to be lar-gely independent. In addition, surface features were con-sulted even on trials when spatiotemporal informationwas inconsistent with the interpretation of correspondenceon the basis of color. Finally, in Experiment 5, effects of sur-face feature consistency were observed despite the absenceof motion to link the pre- and post-occlusion objects. Thus,color can be used to establish object correspondence evenin the presence of spatiotemporal information that directlycontradicts an interpretation of object continuity.

Together, these results provide clear evidence that corre-spondence operations across brief occlusion consult surfacefeature information and are not limited to spatiotemporalinformation. The results cannot be accommodated by theo-

ries claiming that, as a general rule, correspondence opera-tions consult only spatiotemporal information (Kahnemanet al., 1992), and they cannot be accommodated by theoriesclaiming that, as a general rule, spatiotemporal features‘‘trump” surface features when the former are informativeof objecthood (Flombaum et al., 2009; Scholl, 2007).

8.1. Converging evidence from other perceptual domains

The conclusions from the present study apply to the do-main of brief occlusion events. Converging evidence of sur-face feature correspondence has been found recently inother domains. Richard, Luck, and Hollingworth (2008;see also Hollingworth et al. (2008)) examined correspon-dence operations across saccadic eye movements. Partici-pants generated a saccade to one of two objects. Duringthe saccade, the objects were shifted so that the eyes landedbetween them, which simulated the common circumstancethat the eyes fail to land on a target object. Participants wereinstructed to correct their gaze to the object that was eitherin the appropriate location (e.g., ‘‘top”) or had the appropri-ate surface feature property (e.g., ‘‘red”). On a subset of tri-als, the two objects switched properties so that thecorrection had to be made to (1) an object that was in thecorrect location but had the wrong color or (2) to an objectthat had the correct color but was in the wrong location. Inaccordance with the results of the present study, both typesof inconsistency impaired participants’ ability to correctgaze to the appropriate object, suggesting that both spatialand surface feature properties are used to map objectsacross saccades. The Richard et al. data show that surfacefeatures are functional in object correspondence for themost frequent and the briefest form of disruption intro-duced by dynamic vision (saccades occur, on average, threetimes per second and last only 20–60 ms). In addition, Rich-ard et al. found that the weighting of surface feature and po-sition information was governed by the demands of saccadetarget selection. If the saccade target was selected on the ba-sis of its position, position cues were weighted more heavilythan surface feature cues in correspondence operationsafter the saccade. However, if the saccade target was se-lected on the basis of its surface features, surface featurecues were weighted more heavily than spatial cues in corre-spondence operations after the saccade.

It is still possible that there are other domains in whichspatiotemporal information is dominant. As reviewedabove, the most likely candidate domain is motion corre-spondence (Flombaum et al., 2009). In apparent motionexperiments using displays composed of multiple objects,spatiotemporal factors, such as inter-object distance, tendto drive object mapping. Surface feature information caninfluence the perception of apparent motion, but this typ-ically occurs only when spatial properties are non-infor-mative (Dawson, 1991). In addition, motion is perceivedbetween two objects even when those objects differ onsalient surface feature attributes (Kolers, 1972). Yet, recentevidence suggests a role for surface features in the compu-tation of apparent motion, even in the presence of unam-biguous spatial information. Moore et al. (2007)presented a dynamic object display in which an object diskwas shifted in a series of steps around a virtual circle. Par-


ticipants typically perceived a single object moving in a cir-cular path. In one of the frames, the size of the object waschanged. The position of the changed object was still con-sistent with the interpretation of a single object moving ona circular path. However, instead of perceiving a single ob-ject, participants perceived two objects simultaneously: atrace of the original object at the previous location and asecond (larger) object at the current location. Similar ef-fects have been found for color change (Moore & Enns,2004). Thus, when color and size were inconsistent withthe interpretation of continuity, the perception of a single,continuous object was blocked. Although this findingindicates that surface feature differences can controlapparent motion under some circumstances, it is still thecase that, in general, spatiotemporal factors appear to beweighted more heavily than surface features in apparentmotion.

Another possibility is that spatiotemporal information isdominant in the conscious perception of object persistence(Scholl, 2007). In this view, operations that lead to the con-scious experience of a single, persisting object consult spa-tiotemporal information preferentially. The present datacannot speak to this issue, because in the object-reviewingparadigm, correspondence is assessed indirectly (by meansof association with a test stimulus) and conscious experi-ence of object continuity is not probed (Mitroff et al.,2005). However, the Moore et al. (2007) data provide ademonstration proof that conscious perception of objectpersistence can be controlled by surface feature consis-tency. In addition, humans consciously perceive object per-sistence across eye movements (an object viewed across asaccade is perceived as continuously present), and corre-spondence operations across saccades can be controlledby surface feature attributes (Hollingworth et al., 2008;Richard et al., 2008). Thus, we can safely conclude thatalthough there may be some circumstances under whichspatiotemporal information is weighted more heavily thansurface feature information, there does not appear to be anyperceptual domain in which spatiotemporal information isthe sole input to object correspondence operations.

8.2. Object representations as addressable both byspatiotemporal features and by surface features

The original claim of spatiotemporal dominance was in-formed by a consideration of the computational benefits ofaddressing objects by their locations (Kahneman et al.,1992). These benefits are considerable. There may be mul-tiple cars in view, but there will typically only be one ob-ject occupying any given location. Addressing objects bytheir locations therefore allows the visual system to spec-ify a particular individual object (a token) and distinguishit from other objects with similar visual or conceptual fea-tures (Kahneman et al., 1992). In addition, by addressingan object by its unique spatial location, the visual systemcan ensure that features of the object at that location,and only the features of that object, are bound togetherthrough shared position (Treisman, 1988). In this manner,position can establish ‘‘objecthood”, enabling a coherent,object-based representation to be formed. Further,addressing objects by their locations could simplify object

reference in a variety of cognitive operations (Ballard, Hay-hoe, Pook, & Rao, 1997; Pylyshyn, 2000). Instead of refer-ring to a complex set of surface feature properties whenimplementing an action, such as ‘‘follow [white, 4-door,boxy car]”, one need only refer to the index marking thatobject, ‘‘follow [index]”. As long as the spatial index canbe maintained on the appropriate object, that object canbe followed without any reference at all to its surface fea-ture properties (as is likely to be the case in multiple-ob-ject tracking studies, Pylyshyn, 2004; Pylyshyn & Storm,1988). Finally, addressing objects by their locations couldalleviate limitations on memory for the surface featureproperties of objects. The phenomenon of change blind-ness has led some researchers to conclude that very littleperceptual detail is remembered from objects. Marking ob-jects by their locations would allow one to efficiently directattention back to the appropriate object location wheninformation about that object is required by the task (Pyly-shyn, 2000; Spivey, Richardson, & Fitneva, 2004).

There is validity to all of these arguments. And it islikely that there do exist forms of object representationthat are addressed by spatial location, particularly in dorsalstream systems that have been implicated in inherentlyspatial tasks such as multiple-object tracking (Culhamet al., 1998) and that have limited access to the surface fea-ture properties of objects. Our argument is that object rep-resentations also can be addressed by their surface featureproperties, which is likely to occur in ventral stream sys-tems with precise coding of object surface features butwith limited access to the spatiotemporal properties of ob-jects. In addition, we argue that the visual system makesuse of such object representations to individuate object to-kens and to establish object correspondence across briefperceptual disruption.

In order to do so, surface feature representations of ob-jects would need to retain sufficient object detail to distin-guish individual tokens of a particular object type. Contraryto claims that surface feature memory is impoverished(Pylyshyn, 2000; Spivey et al., 2004), token-specific detailsof complex objects are reliably preserved in visual workingmemory (VWM) (Gordon et al., 2008; Henderson & Holling-worth, 2003; Henderson & Siefert, 2001; Tarr, Bulthoff,Zabinski, & Blanz, 1997), even after attention has been di-rected elsewhere (Gajewski & Brockmole, 2006; Holling-worth, 2004; Johnson, Hollingworth, & Luck, 2008). AndVWM representations can retain quite precise surface fea-ture information, supporting subtle, within-category colordiscriminations (Zhang & Luck, 2008). Thus, brief forms ofvisual memory are certainly capable of retaining surfacefeature information of sufficient precision to individuateobject tokens and to establish object correspondence.

Because VWM can maintain token-specific surface fea-ture information, a spatial index is not necessarily re-quired to individuate tokens of a particular conceptualtype, undermining one of the key motivations of the ob-ject-file claim of spatiotemporal dominance (Kahnemanet al., 1992). For example, a representation of the surfacefeature properties of the white, 4-door, boxy car would beperfectly sufficient to distinguish that token of a car frommost other cars. One would only require spatial indexes toindividuate object tokens when there were multiple,


highly similar objects or a set of identical objects visible.Although laboratory tasks probing dynamic vision oftencreate precisely this circumstance (Kahneman et al.,1992; Pylyshyn & Storm, 1988; Treisman & Gelade,1980), a scene composed of multiple, identical objects iscertainly uncommon in the real world. Under normal cir-cumstances, a representation of the surface features of anobject will be sufficient to distinguish it from other visibleobjects.

We think it is unlikely that the visual system wouldignore such highly informative surface feature differencesbetween objects in favor of establishing correspondenceon the basis of spatiotemporal information alone. If, afterbeing distracted by the pedestrian, a red, 2-door, convertiblewere to be present in the expected location of the white se-dan, we doubt that many people would make an error in ob-ject correspondence and follow the red car to the wrongdestination. Instead, surface feature memory would contrib-ute to the correspondence operation, and the white carwould be successfully followed, despite some degree of per-ceived inconsistency in spatial information. The presentdata demonstrate that, indeed, memory for surface featuresis used to establish object correspondence across brief per-ceptual disruption, even in the presence of spatiotemporalinformation that is inconsistent with the interpretation ofcorrespondence. In general, we propose that since spatio-

Position consistent

Same

Experiment 1Correct RT (ms) 855Accuracy (%) 91.0

Color consistent

Same



Experiment 3 controlCorrect RT (ms) 885Accuracy (%) 89.6

Both consistent Color inconsiste

Same New Same N

Experiment 4Correct RT (ms) 942 1007 988 1Accuracy (%) 93.1 82.9 91.8 8

Color consistent

Same


temporal and surface features can both contribute to themapping of objects across brief disruption, both will be uti-lized by the visual system, and the contribution of the twosources of information will likely vary as a function of utilitywithin the current task (see also Richard et al., 2008). If thereare multiple similar objects visible, then spatial positioncould be the primary means to individuate objects andmap them across disruptions. However, if objects have clearsurface feature differences, surface feature information canprovide an efficient means to individuate them and mapthem across disruption.

Neither spatiotemporal nor surface feature informa-tion will always provide a reliable solution to the corre-spondence problem, however. Both are potentially proneto error. For example, if an object changes motion direc-tion during a brief perceptual disruption, correspon-dence on the basis of spatiotemporal properties mightcertainly fail. Similarly, if an object were to rotate dur-ing a perceptual disruption, revealing a different set ofsurface features, correspondence on the basis of surfacefeatures might fail. The use of both sources ofinformation, as observed in the present study, maxi-mizes the likelihood that the visual system will gener-ate a successful solution to the problem of objectcorrespondence.

Position inconsistent

New Same New

886 928 90783.9 88.8 85.9

Color inconsistent

New Same New

1044 1048 103980.0 89.4 81.4

969 1019 97888.1 90.4 90.7

957 927 95376.8 86.0 78.9

nt Position inconsistent Both inconsistent

ew Same New Same New

040 1012 1030 1049 10194.2 91.7 81.9 88.8 83.8

Color inconsistent

New Same New

940 931 96273.3 87.1 75.4


Acknowledgements

The research was supported by NIH Grant R01EY017356.We would like to thank Cathleen Moore, Steve Luck, andShaun Vecera for helpful discussions of the research.

Appendix

Mean percentage correct and RT data from Experiments1–5 and the Experiment 3 control.

References

Ballard, D. H., Hayhoe, M. M., Pook, P. K., & Rao, R. P. (1997). Deictic codesfor the embodiment of cognition. Behavioral and Brain Sciences, 20,723–767.

Culham, J. C., Brandt, S. A., Cavanagh, P., Kanwisher, N. G., Dale, A. M., &Tootell, R. B. H. (1998). Cortical fMRI activation produced by attentivetracking of moving targets. Journal of Neurophysiology, 80, 2657–2670.

Currie, C. B., McConkie, G. W., Carlson-Radvansky, L. A., & Irwin, D. E.(2000). The role of the saccade target object in the perception of avisually stable world. Perception and Psychophysics, 62, 673–683.

Dawson, M. R. W. (1991). The how and why of what went where inapparent motion: Modeling solutions to the motion correspondenceproblem. Psychological Review, 98, 569–603.

Feigenson, L., & Carey, S. (2005). On the limits of infants’ quantification ofsmall object arrays. Cognition, 97, 295–313.

Flombaum, J. I, Scholl, B. J, & Santos, L. R (2009). Spatiotemporal priority asa fundamental principle of object persistence. In B. Hood & L. Santos(Eds.), The origins of object knowledge (pp. 135–164). OxfordUniversity Press.

Gajewski, D. A., & Brockmole, J. R. (2006). Feature bindings endurewithout attention: Evidence from an explicit recall task. PsychonomicBulletin and Review, 13, 581–587.

Gordon, R. D., Vollmer, S. D., & Frankl, M. L. (2008). Object continuity andthe transsaccadic representation of form. Perception andPsychophysics, 70, 667–679.

Henderson, J. M. (1994). Two representational systems in dynamic visualidentification. Journal of Experimental Psychology: General, 123,410–426.

Henderson, J. M., & Hollingworth, A. (2003). Eye movements and visualmemory: Detecting changes to saccade targets in scenes. Perceptionand Psychophysics, 65, 58–71.

Henderson, J. M., & Siefert, A. B. C. (2001). Types and tokens intranssaccadic object identification: Effects of spatial position andleft–right orientation. Psychonomic Bulletin and Review, 8, 753–760.

Hollingworth, A. (2004). Constructing visual representations of naturalscenes: The roles of short- and long-term visual memory. Journal ofExperimental Psychology: Human Perception and Performance, 30,519–537.

Hollingworth, A., Richard, A. M., & Luck, S. J. (2008). Understanding thefunction of visual short-term memory: Transsaccadic memory, objectcorrespondence, and gaze correction. Journal of ExperimentalPsychology: General, 137, 163–181.

Horowitz, T. S., Klieger, S. B., Fencsik, D. E., Yang, K. K., Alvarez, G. A., &Wolfe, J. M. (2007). Tracking unique objects. Perception andPsychophysics, 69, 172–184.

Irwin, D. E. (1992). Memory for position and identity across eyemovements. Journal of Experimental Psychology: Learning, Memory,and Cognition, 18, 307–317.

Irwin, D. E., & Andrews, R. V. (1996). Integration and accumulation ofinformation across saccadic eye movements. In T. Inui & J. L.McClelland (Eds.), Attention and performance XVI: Informationintegration in perception and communication (pp. 125–155).Cambridge, MA: MIT Press.

Irwin, D. E., & Gordon, R. D. (1998). Eye movements, attention, andtranssaccadic memory. Visual Cognition, 5, 127–155.

Johnson, J. S., Hollingworth, A., & Luck, S. J. (2008). The role of attention inthe maintenance of feature bindings in visual short-term memory.Journal of Experimental Psychology: Human Perception and Performance,41–55.

Kahneman, D., Treisman, A., & Gibbs, B. J. (1992). The reviewing of objectfiles: Object-specific integration of information. Cognitive Psychology,24, 175–219.

Kolers, P. A. (1972). Aspects of motion perception. Oxford: Pergamon Press.Kruschke, J. K., & Fragassi, M. M. (1996). The perception of causality:

Feature binding in interacting objects. In Proceedings of the eighteenthannual conference of the cognitive science society (pp. 441–446).Hillsdale, NJ: Erlbaum.

Leslie, A., Xu, F., Tremoulet, P., & Scholl, B. J. (1998). Indexing and theobject concept: ‘‘What” and ‘‘where” in infancy. Trends in CognitiveSciences, 2, 10–18.

Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory forfeatures and conjunctions. Nature, 390, 279–281.

Mitroff, S. R., & Alvarez, G. A. (2007). Space and time, not surface features,underlie object persistence. Psychonomic Bulletin and Review, 14,1199–1204.

Mitroff, S. R., Scholl, B. J., & Wynn, K. (2004). Divide and conquer: Howobject files adapt when a persisting object splits into two.Psychological Science, 15, 420–425.

Mitroff, S. R., Scholl, B. J., & Wynn, K. (2005). The relationship betweenobject files and conscious perception. Cognition, 96, 67–92.

Moore, C. M., & Enns, J. T. (2004). Object updating and the flash-lag effect.Psychological Science, 15, 866–871.

Moore, C. M., Mordkoff, J. T., & Enns, J. T. (2007). The path of leastpersistence: Evidence of object-mediated visual updating. VisionResearch, 47, 1624–1630.

Moore, M. K., Borton, R., & Darby, B. L. (1978). Visual tracking in younginfants: Evidence for object identity or object permanence. Journal ofExperimental Child Psychology, 25, 183–198.

Pylyshyn, Z. W. (2000). Situating vision in the world. Trends in CognitiveSciences, 4, 197–207.

Pylyshyn, Z. W. (2004). Some puzzling findings in multiple objecttracking: I. Tracking without keeping track of object identities.Visual Cognition, 11, 801–822.

Pylyshyn, Z. W., & Storm, R. W. (1988). Tracking multiple independenttargets: Evidence for a parallel tracking mechanism. Spatial Vision, 3,179–197.

Richard, A. M., Luck, S. J., & Hollingworth, A. (2008). Establishing objectcorrespondence across eye movements: Flexible use ofspatiotemporal and surface feature information. Cognition, 109,66–88.

Scholl, B. J. (2007). Object persistence in philosophy and psychology. Mindand Language, 22, 563–591.

Scholl, B. J., & Pylyshyn, Z. W. (1999). Tracking multiple items throughocclusion: Clues to visual objecthood. Cognitive Psychology, 38,259–290.

Scholl, B. J., Pylyshyn, Z. W., & Franconeri, S. L. (1999). When arespatiotemporal and featural properties encoded as a result ofattentional allocation? Paper presented at the conference of theAssociation for Research in Vision and Ophthalmology.

Spivey, M., Richardson, D., & Fitneva, S. (2004). Thinking outside thebrain: Spatial indices to linguistic and visual information. In J. M.Henderson & F. Ferreira (Eds.), The interface of language, vision, andaction: Eye movements and the visual world (pp. 161–189). New York:Psychology Press.

Tarr, M. J., Bulthoff, H. H., Zabinski, M., & Blanz, V. (1997). To what extentdo unique parts influence recognition across changes in viewpoint?Psychological Science, 8, 282–289.

Treisman, A. (1988). Features and objects: The fourteenth Bartlettmemorial lecture. Quarterly Journal of Experimental Psychology, 40A,201–237.

Treisman, A., & Gelade, G. (1980). A feature-integration theory ofattention. Cognitive Psychology, 12, 97–136.

Xu, F., & Carey, S. (1996). Infants’ metaphysics: The case of numericalidentity. Cognitive Psychology, 30, 111–153.

Xu, F., Carey, S., & Welch, J. (1999). Infants’ ability to use object kindinformation for object individuation. Cognition, 70, 137–166.

Zelinsky, G. J., & Loschky, L. C. (2005). Eye movements serialize memoryfor objects in scenes. Perception and Psychophysics, 67, 676–690.

Zhang, W. W., & Luck, S. J. (2008). Discrete fixed-resolutionrepresentations in visual working memory. Nature, 453, 233–235.

Documents

Object correspondence across brief occlusion is established on the basis of both spatiotemporal and surface feature cues