Submitted February 2008

Short-term memory for facial identity and emotion

Murray Galster†, Michael J. Kahana?

Hugh R. Wilson‡ & Robert Sekuler†

†Volen Center for Complex Systems, Brandeis University, Waltham MA 02454?Department of Psychology, University of Pennsylvania, Philadelphia PA 19104

‡Centre for Vision Research, York University, Toronto ON M3J 1P3

For some time the relationship between processing of facial expression and facialidentity has been in dispute. We re-examined this relationship and evaluated whetherthe same relationship characterized both perception and short-term memory. In threeexperiments we examined perception and memory for realistic, synthetic human faces.Experiment 1 used subjects’ judgments to identify, from a large library of syntheticfaces, a set of 60 faces that were best recognized as expressing particular categoriesof emotion. Experiment 2 used the selected faces in a short-term recognition task.Subjects had to identify whether the emotional expression on a probe stimulus facematched the emotional expression on either of two study faces. Results showedthat identity strongly influenced recognition of emotional expression. Multidimen-sional scaling of similarity/dissimilarity judgments in Experiment 3 produced a 2-dimensional description of the faces’ perceptual representation. Distances amongstimuli in the MDS representation, which showed a strong linkage of emotional expres-sion and facial identity, proved to be good predictors of correct and false recognitionsin Experiment 2. The convergence of results from Experiments 2 and 3 suggests thatthe overall structure and configuration of the faces’ perceptual representation paralleltheir representation in short-term memory.

The international community recently instructed itsmember nations to frown upon smiles. By conven-tion, the photo in any new passport must show thepassport’s bearer unsmiling, with lips pressed togetherand an expression whose neutrality is beyond question.These rules are meant to aid automated facial recog-nition technology that is deployed against fraud andterrorism. Smiles or any other emotional expressionare frowned upon because they alter the geometry ofkey facial features, and thereby undermine the capac-ity of an automated biometric system to compare thepassport photo to the traveler’s own face. So, as faras the automated biometric system is concerned, facialexpression and identity are inextricably linked.

Although biometric recognition systems link facialidentity and facial expression, human observers do notalways follow suit. In fact, independence of facial iden-tification from other aspects of face processing was a

Supported by National Institutes of Health grantsMH068404, MH55687, MH61975 and EY002158, and by Cana-dian Institute for Health Research grant 172103. Many thanksto Shivakumar Viswanathan and Deborah Goren for contri-butions to the research reported here. This report is basedon data collected at Brandeis for M. G.’s Master’s thesis inNeuroscience. mail:sekuler@brandeis.edu

keystone of Bruce and Young (1986)’s influential ac-count of face recognition. That account describes theface recognition system as a set of modules, each of“which plays a distinct functional role, and whose op-eration can be eliminated, isolated or independentlymanipulated through experiment or as a consequenceof brain damage (p. 311).” In particular, Bruce andYoung’s model separates the modules specialized forrecognizing identity (the individual to whom the facebelongs) from the module specialized for analyzing theface’s expression (what emotion that face seems to ex-pressing).

A number of studies support the idea that thesetwo processes are distinct from one another percep-tually. For example, prosopagnosia, or face blind-ness, sometimes impairs a patient’s ability to determinesomebody’s identity from their face alone, while leav-ing recognition of facial emotion unaffected (Duchaine,Parker, & Nakayama, 2003; Sacks, 1985). In othercases the converse has been observed: processing offacial emotion is affected, but the processing of iden-tity is spared (Humphreys, Donnelly, & Riddoch, 1993).These dissociations are consistent with the proposi-tion that processing of facial expression and process-ing of facial identity activate pathways in the brainthat are at least partially separable (Hasselmo, Rolls,& Baylis, 1989; Sergent, Ohta, MacDonald, & Zuck,

1

2 GALSTER & SEKULER

1994; Gelder, Vroomen, Pourtois, & Weiskrantz, 1999;Gelder, Pourtois, & Weiskrantz, 2002; Posamentier &Abdi, 2003). Additionally, with neuro-normal subjects,Goren and Wilson (2006) found that manipulation ofa face stimulus’ spatial frequency content differentiallyaffects recognition of facial expression and facial iden-tity, with recognition of expression being affected to thegreater extent.

It is noteworthy that divergence between stimulusattributes at one level, such as for perception, does notguarantee that their independence is preserved at anyor all subsequent levels of processing, as Ashby andMaddox (1990) have pointed out. Although their ar-gument focused on decision making, we believe thatthe same general point holds for memory storage andrecall as well. As a result, demonstrations that facialidentity and facial emotion are or are not perceptuallyindependent say little about whether that independenceis preserved when it comes to memory. Therefore, weassessed the relationships between facial identity andfacial emotion at both the perceptual level and at thelevel of short-term memory.1

Despite evidence to the contrary, there have beenchallenges to the claim that identity and emotion areentirely independent processes (for a review, see Posa-mentier & Abdi, 2003). For example, D’Argembeauand Linden (2007) showed that memory of facial iden-tity is modulated by the face’s emotional expression,with the identity of a happy face being rememberedbetter than the identity of a face previously seen withan angry expression. Moreover, this modulation byemotion was effective even when little or no attentionis paid to the face’s emotion. Additionally, researchershave attempted to characterize the separability of iden-tity and emotional expression using a framework in-troduced by Garner (1974). In one such study, Ganeland Goshen-Gottstein (2004) asked subjects to catego-rize photographs either according to the identity theyshowed, or according to the emotion shown. The stim-uli were photographs of two different people (iden-tities) shown in two different emotional expressionsand in two different views. In either case, the dimen-sion that subjects attempted to ignore did influence thespeed with which items were classified on the other,attended dimension. Baudouin, Martin, Tiberghien,Verlut, and Franck (2002) used a similar experimentaldesign, but with schizophrenic as well as normal, non-schizophrenic subjects. Both groups of subjects showeda striking asymmetry of influence between the two di-mensions, with differences in identity exerting far, farless influence on judgements of emotion than vice versa.Moreover, the degree of inter-attribute interference var-ied strongly with the severity of negative symptoms inthe schizophrenic subjects.

Unfortunately, these two studies as well as other,previous studies of the relationship between process-ing of identity and emotion have based their conclu-sions with items that were a scant, unsystematic sam-

ple from the entire, task-relevant stimulus space. Forexample, Ganel and Goshen-Gottstein (2004) and byBaudouin et al. (2002) used only two different facesand two different emotions in each experiment, whichmakes it difficult to quantify the degree of cross-talk(non-separability) between dimensions. These studiesmade no real effort to test the boundary conditions forperceptual independence, which would have requiredstimulus materials whose similarity relations were care-fully chosen and controlled (Sekuler & Kahana, 2007).So in the experiments reported here, similarity relationsamong faces and emotions were manipulated so as tochallenge subjects’ ability to perceive and rememberfacial emotion independently of facial identity.

Using tasks that require subtle distinctions amongstimulus items, we asked whether perception of andshort-term memory for facial emotions were indepen-dent of the faces on which those emotions had beenexpressed. An analogy should clarify this question’stheoretical significance. Perceptual constancy is thetendency of an object’s appearance to remain constantdespite changes in the stimulus falling on the receptorsurface. To generate this stable percept, a sensory sys-tem takes a variable input and produces an invariantoutput (Hurlbert, 2007), that is, constancy is very muchthe product of neural mechanisms. As a result, carefulstudy of perceptual constancy, including conditions un-der which it fails, can illuminate its neural mechanismsand the computations that they perform. Our focushere can be described as “expression constancy,” thatis, examining the constancy of an emotional expressiondespite variation in the identity of the face on whichthat emotion happens to be expressed. And as is thecase for perceptual constancy, we believe that a carefulexploration of expression constancy can shed light onimportant brain mechanisms of face processing.

Our experiments make use of realistic synthetic faces(Wilson, Loffler, & Wilkinson, 2002) that afford goodcontrol over key stimulus properties. These key prop-erties include inter-item similarity, which strongly in-fluences short-term recognition memory (Sekuler & Ka-hana, 2007). Working with a large library of faces andemotional expressions, Experiment 1 identified an ap-propriate set of synthetic faces with which to explore thelinkage between memory for facial emotion and facialidentity. The goal was to identify individual faces thatbest expressed various emotions and degrees of emo-tion. The culled faces were then used in Experiment 2 asstimulus materials in a short-term recognition memorytask (Sternberg, 1966). From the face of each of six dif-ferent individuals, three female, we generated ten vari-ants, each expressing a different combination of emo-

1 Ashby and Maddox (1990) cast their argument in termsof the distinction between separable and integral dimensions(Garner, 1974). To avoid an unnecessary theoretical commit-ment, we use the more common term, “independent”, as asubstitute for “separable”.

FACIAL IDENTITY AND EMOTION 3

tion and intensity. Subjects’ memory for a particularemotional expression was degraded when that expres-sion was seen later on the face of a different individual.This result suggested a linkage between memory for fa-cial expression and facial identity, a hypothesis that wastested further in the two parts of Experiment 3. The firstpart of Experiment 3 used similarity/dissimilarity judg-ments and multidimensional scaling (MDS) to charac-terize the perceptual space within which combinationsof identities and emotions were represented. The sec-ond part of the experiment used the pairwise distancesin the MDS face space to account for the recognitionmemory results from Experiment 2.

Experiment 1

As mentioned before, exploring the relationship be-tween processing facial expression and processing fa-cial identity requires a set of face stimuli with appropri-ate qualities. Wilson et al. (2002) developed a class ofsynthetic faces that offer such qualities, including goodrealism and recognizability, achieved without sacrific-ing control of stimulus characteristics. These realisticsynthetic faces are known as Wilson faces, and full de-tails of their construction can be found elsewhere (Wil-son et al., 2002). In brief, these faces are constructedfrom several dozen independent measurements madeon photographs of a person’s face seen in two views,full face and slightly turned. Filtering of the faces re-moves textures and hair, leaving the hairline. The noseand mouth of a synthetic face reflect measurements ofan individual’s mouth width, lip and thickness, andalso the individual’s nose width and length. ExampleWilson faces are shown in Figure 1.

With these faces one can construct stimulus sets withgraded and distinct emotional expressions (Goren &Wilson, 2006), and with varying similarity relationsamong faces belonging to different individuals (facialidentity). Wilson faces are good approximations to ac-tual faces, producing levels of neural activation in thebrain’s face regions that are comparable to the activa-tions evoked by high-fidelity photographs of individualfaces (Loffler, Yourganov, Wilkinson, & Wilson, 2005),and exhibiting substantial face-inversion effects (Goren& Wilson, 2006), which is taken as a hallmark of faceprocessing (but see, Russell, Biederman, Nederhouser,& Sinha, 2007). The emotional expressions on Wilsonfaces can be extracted quickly, from presentations last-ing only 110 msec. This rapidity of processing wasimportant to our choice of stimuli, as conspecifics of-ten make sense of an emotional expression glimpsedin a single brief fixation (Ekman, 1993). Wilson facespreserve facial identity, which allows a face to be ac-curately matched to the photograph of the individualfrom which it were derived (Wilson et al., 2002; Wilson& Wilkinson, 2002; Goren & Wilson, 2006; Loffler, Gor-don, Wilkinson, Goren, & Wilson, 2005). Finally, thevalue of Wilson faces has been demonstrated in studies

of short-term visual memory (Yotsumoto, Kahana, Wil-son, & Sekuler, 2007; Sekuler & Kahana, 2007), and havebeen useful in studying the impact of facial emotions(Isaacowitz, Wadlinger, Goren, & Wilson, 2006).

A B C D

E F G

Figure 1. Examples of Wilson faces used as stimuli here.Panels A-C, and E-G are variants of a single female face. A-Cand G show this female with a expressions that are neutral(A), maximally sad (B), maximally fearful (C) and maximallyangry (G). Panels E-F show this same female exhibiting dif-ferent degrees of anger, from mild (E) to maximal (G). Forcomparison to C, Panel D shows maximal fear exhibited bya different female face. Other examples are available in thesupplementary material to this article.

Using Ekman and Friesen (1975)’s facial action cod-ing system as a guide, Goren and Wilson (2006) con-firmed that the emotional expression on a syntheticface could be controlled by modifying the relative posi-tions and/or orientations of ten facial structures thatcontribute to facial expression of emotion. With anominally-neutral face as a base, a desired emotion isproduced by adjusting variables such as the amountof visible sclera (Whalen et al., 2004), the positions ofupper and lower eyelids, the shape and position of theeyebrows (Sadr, Jarudi, & Sinha, 2003), the width ofnose, and the curvature and size of mouth. Moreover,the degree of each emotion can be systematicaly manip-ulated by varying the amount by which these structuralfeatures were altered. Additional details about the con-struction of synthetic faces that convey emotions aregiven elsewhere (Goren & Wilson, 2006).

While being photographed to provide the raw ma-terial for his or her Wilson face, each individual wasinstructed to assume and hold a neutral expression.However, even perfect compliance with this instruc-tion would not guarantee that the resulting facial ex-pression will consistently be perceived as intended. Infact, this result has been known since Darwin’s pio-neering explorations of facial expression (1872). Somepeople’s faces when physically at rest – that is, withrelaxed facial muscles – are perceived as expressingsome emotion. In fact, casual examination of all the

4 GALSTER & SEKULER

nominally-neutral Wilson faces suggested that some ofthese faces portrayed modest but distinct amounts ofone or other emotion. Consequently, Experiment 1 setout to identify Wilson faces whose neutral expressions,and other expressions as well, would be perceived mostaccurately. The selected faces would then be used asstimulus materials in the following experiments.

Stimuli

The initial stimulus pool comprised 79 realistic syn-thetic faces that were based on photographs of 40 Cau-casian females and 39 Caucasian males. Each of thesesynthetic faces was meant to display a neutral expres-sion. Using an algorithm introduced by Goren andWilson (2006), we generated nine variants of each neu-tral face. These variants expressed the emotions sad,angry and fearful, and did so with three different levelsof intensity. Earlier, Goren and Wilson (2006) had de-termined the stimulus conditions that produced eachface’s strongest perceived emotion. This was basedon subjects’ judgments of faces whose arrangement ofcharacteristic features were deemed to be most effec-tive in expressing a particular emotion, using criteriaincluding believability of the expression. Starting withthis information about the maximum intensity (100%)of each emotion that could be expressed by each face,we generated faces with emotional expressions at either33%, 67% or 100% of the metric distance from neutralto maximum.

Subjects

Seven undergraduate students naive to the purposeof the experiment participated. Four served in a singlesession each; three served in two sessions. All subjectswere paid for their participation.

Procedure

Stimuli were presented on a computer display lo-cated 114 cm from the subject. Each trial was begun bya key press that caused a fixation cross to appear at thecenter of the display. After 900 msec, the fixation crosswas replaced by a single Wilson face, which remainedvisible for 110 ms. The subject then responded usingthe keyboard, indicating whether the face appeared tobe angry, fearful, happy, neutral, or sad. The relativelybrief stimulus presentation was chosen for consistencywith conditions that had been used in most previousstudies with Wilson faces, and also in order to mimic thelimited information that a person might take in duringa single brief glance, minimizing reliance on detailedscrutiny.

Stimuli were displayed using Matlab with functionsfrom the Psychtoolbox (Brainard, 1997). On the displayscreen, each Wilson face measured approximately 5.5◦high and 3.8◦ wide. The spatial position of the face wasrandomized by the addition of horizontal and vertical

displacements drawn from a uniform random distri-bution with mean 12.6 minarc, and a range of 2.1-23.1minarc. No feedback was provided after a response.

In the 632 trials that made up a session, subjects saweach nominally neutral face four times, and each facethat was supposed to express some emotion once. Faceswere presented in a new random order in each session.

Results and Discussion

To select the faces that would be used later in ourmemory experiment, we identified the faces whosenominally neutral versions were most consistently cat-egorized as neutral. This screening process was stimu-lated by earlier observations that individual differences,including differences in facial anatomy, influence thesuccess with which individuals’ emotional expressionsare categorized (Gosselin & Kirouac, 1995). In our ex-periment, 13 faces were correctly classified as neutralat least 85% of the time. From these 13 faces, weidentified the six faces, three male and three female,whose emotions were most accurately identified, whenthe emotion was presented at its weakest intensity, thatis, 33% of its maximum strength. For the six faces, themean accuracy (and range) with which each emotionwas correctly categorized was 0.63 (.50-.90) for anger;0.45 (.20-.80) for sadness; and 0.60 (.50-.70) for fear. Asexpected, strongly expressed emotions were identifiedmore accurately than weaker ones were. In particular,faces meant to express emotions more strongly, that iseither 67% or 100% of maximum, were correctly catego-rized on virtually every presentation, despite the briefviewing time. As explained below, various forms ofthese six individuals’ faces were used as stimuli in thesucceeding experiments.

Experiment 2

With the stimuli identified in Experiment 1 as mostappropriate for our purposes, we used a short-termrecognition procedure to examine possible connectionsbetween memory for facial identity and memory for fa-cial expression. On each trial, subjects saw two succes-sively presented faces–mixtures of identities and emo-tions (including neutral faces). Then they had to judgewhether the category and degree of emotion on a sub-sequently presented probe face did or did not matchthe category and degree of emotion on one of the studyfaces. Stimuli for each trial were chosen from a set of60 possible faces, with 10 variants based on each of thesix faces identified in Experiment 1. Each set of 10 com-prised a neutral version of the face, and three differentlevels of intensity (33, 67, and 100% of maximum) foreach of three emotion categories (sadness, anger, andfear).

FACIAL IDENTITY AND EMOTION 5

SubjectsSeventeen subjects, nine females, participated in this

and the following experiment. The subjects, aged 18-29years, had normal or corrected-to-normal vision, andwere paid for their participation. None served in theprevious experiment. One female subject’s data had tobe discarded prior to analysis because of chance-levelperformance.

ProcedureOn each trial, subjects first saw two study faces pre-

sented one after the other. These two faces were fol-lowed by a test or probe face. The subject’s task wasto determine if the expression on the probe face exactlymatched the expression that had been seen on either ofthe two list faces, ignoring the identities. Subjects wereinstructed that a match required that both the expres-sion category (neutral, fearful, angry, or sad) and theintensity of that emotion (33, 67, or 100% of maximum)had to match. This instruction was reinforced on eachtrial by post-response feedback, distinctive tones thatsignaled if the subject’s response had been correct.

At the start of a trial, a fixation cross appeared at thecenter of the screen for 500 ms. The probe disappearedand the first study item face, s1, appeared for 110 ms,after which it disappeared. This was followed by a200 ms inter-stimulus interval, and then presentationof the second study item face, s2, for 110 ms. A 1200ms delay followed the two study items, during whicha tone sounded, indicating that the next stimulus facewould be the probe, p, which also displayed for 110ms. A subject responded by pressing one of two keys,indicating whether or not the expression (category andintensity) on the probe’s face matched that of eitherface in the study list. To minimize usefulness of verniercues, the horizontal and vertical location of each was jit-tered by a random displacement drawn from a uniformdistribution with mean ±12.6 minarc (range = 2.1-23.1minarc). The viewing distance was 114 cm.

The faces presented on a trial produced two majortypes of trials. A Target trial was one on which theprobe’s expression (category and intensity) matchedthe expression did previously appear (correct answeris “yes”); a Lure trial was one on which the probe’sexpression matched that of neither study item (correctanswer is “no”). Note that Lure trials included oneson which the emotional category, but not the intensityof the probe’s expression matched that seen on a studyitem. Lure and Target trials occurred in random order.With equal probability, the 60 different stimulus faces(combinations of emotion category and intensity) couldbe the probe on any trial.

Each subject served in four sessions of 320 trials each.In each session, Target and Lure trials occurred equallyoften. The first eight trials of each session (four Lure andfour Target trials) comprised practice and their resultswere discarded before data analysis.

In addition to equalizing the frequencies of Targetand Lure trials, several other constraints controlled themakeup of stimuli on particular subclasses of Targetand Lure trials. In describing these constraints, theterm ”identity” signifies the person whose face is rep-resented in a stimulus. More specifically, a face’s ”iden-tity” refers to the particular face of the six different peo-ple, three males and three females, whose faces passedthe sieve of Experiment 1. Target and Lure trials wereconstructed so that the probe’s identity matched that ofone item, both study items, or no study item. Addition-ally, any combination of emotion category and intensitycould appear only once in the study set. When twostudy items happened to share the same identity, ourexperimental design forced those two faces to displayemotions from different categories. Finally, on someLure trials, the probe’s emotion matched the emotionalcategory, but not the intensity of one of the study items.

Results and Discussion

Across subjects and sessions, a total of 17 trials hadto be discarded because of technical errors in recordingtheir responses. All the remaining 21,643 trials wereused in the analyses described below. Recognition per-formance level was quite stable over the four sessionsof testing, F(3,45)=.04, p= .989. In addition, a t-test fortwo independent samples showed that male and femalesubjects’ performance did not differ from one another,t(14) = -0.311, p = .76). Therefore, data for subsequentanalyses were averaged over both sessions and subjects’gender.

To determine whether memory for emotional ex-pression was influenced by the identity of the express-ing face, we first compared recognition of emotion (i)when the probe face’s emotional category and intensitymatched those of a study face, but their identities didnot match, and (ii) when the probe face’s emotionalcategory and intensity as well as the face’s identitymatched those of a study face. For every one of the 16subjects correct recognitions were higher in the secondcase –when emotion and identity both matched– thanthe first. The values of P(Yes) for the two cases differedsignificantly, t(15) = 10.383, p <.0001. So, even thoughthe judgments were supposed to be based on emotionalexpression alone, the identity of the probe face clearlyinfluenced memory for emotional expression.

Recall that a correct recognition required that theprobe’s emotion match not only the category of emo-tion of a study face, but also its intensity (33, 67, or100% of maximum). Subjects were to respond ”No”if the probe shared an emotional category with one ofthe study items, but expressed that emotion in a dif-ferent degree. Figure 2 shows the effect of varying thedegree of the probe’s emotional intensity of the probeand the emotional intensity of a study item. The twocurves represent trials on which the probe’s emotionalcategory and facial identity both matched a study item

6 GALSTER & SEKULER

(�), and trials on which the match was on emotionalcategory alone (�). The leftmost, open symbols on eachcurve comprise correct recognitions, Yes responses ontrials that were consistent with instructions given tosubjects. The curves’ remaining, filled symbols com-prise false recognitions, that is, Yes responses when thecorrect response was No. Note that the proportion ofYes responses declines as the emotional intensity of theprobe deviates from that of the probe, F(2,30)=52.588,p <.001. Although differences between the two setsof data points may seem modest, P(Yes) for a matchon both identity and emotional category is consistentlyhigher than P(Yes) for a match on emotional categoryalone, F(1,15)=16.554, p <.001. The interaction betweenthe two variables, degree and kind of match, was notsignificant, F(2,30)=1.072, p >.355. The difference be-tween corresponding points on the two curves confirmsthat judgments of emotional expression are influencedby the presence of match on identity –despite the factthat subjects were explicitly instructed to ignore thataspect of the stimulus when making recognition judg-ments. The lone point shown as an open triangle (4)represents trials on which the probe’s emotional cate-gory and intensity matched one of the study items, andthe probe’s identity matched that of both study items.Because such trials were relatively rare compared tothe other trial types represented in the figure, we optedagainst applying a genuine statistical analysis. How-ever, the figure does suggest an incremental effect whenthe probe matches the identity of both study faces (4)as opposed to matching the identity of only one studyface (�).

The horizontal gray bar in Figure 2 represents themean (±1 SeM) proportion of P(Yes) responses whenwhen the probe’s identity but not its emotional cate-gory matched the identity of a study item. This valueprovides a baseline that isolates the influence of sharedfacial identity. Note that when probe and study facesshare the same emotional category, but the intensityof that expression differs by 67% (rightmost �), P(Yes)falls to a level that would be expected from a match inidentity alone. So, a face’ whose emotion category isthe same as the probe’s but whose emotional intensitydiffers by 67% is no more likely to produce a false recog-nition than is a face that shares the probe’s identity butexpresses an entirely different category of emotion.

Experiment 3

Work using various physiological techniques has ledto the view “...that storage and rehearsal in workingmemory is accomplished by mechanisms that haveevolved to work on problems of perception and ac-tion (Jonides, Lacey, & Nee, 2005).” More specifically,the idea that perception and short-term memory shareessential resources has been supported by a number ofstudies(e.g., Kosslyn, Thompson, Kim, & Alpert, 1995;Postle, 2006; Agam & Sekuler, 2007). With behavioral

0

0.2

0.4

0.6

0.8

1.0

0 33% 67%

ProportionYesResponses

Mismatch in Emotional Intensity

Matches Identity Only

Matches Emotion

Matches Emotion and Identity

Matches Emotion and Both Identities

Figure 2. Effect of varying the definition of a match betweenthe probe face and the study faces. � and � represent P(Yeswhen the probe’s category of emotion matched that of a studyface, but their identities did not match; � and � representP(Yes) when the probe’s category of emotion and the probe’sfacial identity both matched those of a study item. The pointshown as 4 represents trials on which probe emotional cat-egory and intensity matched that of a study item, but theprobe’s facial identity matched that of both study items. Thegray horizontal bar represents mean P(Yes) ±1 SeM for trialson which a probe’s identity matched that of a study item, butthe probe expressed a different category of emotion than thatof either study item.

measures alone we could not directly test this sharedresources hypothesis, but we could probe one of its im-plications, namely structural parallels between (i) thememory representation of facial emotion and identity,as revealed in Experiment 2, and (ii) the perceptual rep-resentation of those same faces.

As a first step in exploring possible parallels be-tween perceptual and memory representations, sub-jects’ made perceptual judgments of the faces’ similar-ity or dissimilarity to one another. These judgmentswere then transformed into a representation of the per-ceptual space within which the faces were embedded.Finally, we determined whether the details of the per-ceptual representation could account for the correct andfalse recognitions that subjects made in Experiment 2’stests of short-term memory. To generate the perceptualmetric required for our stimuli we turned to multidi-mensional scaling (MDS). Using only faces with neu-tral expressions, Yotsumoto et al. (2007) used this samemethod to characterize the perceptual similarity spacefor a set of Wilson faces comprising just four differentidentities, all female. Our goal was first to character-ize the similarity space for faces used in Experiment2, which included not only several different male and

FACIAL IDENTITY AND EMOTION 7

female identities, but also expressions of various emo-tions and intensities of emotions. We would then tryto use the pairwise distances in this perceptual spacein order to predict Experiment 2’s results on short-termrecognition memory.

Stimuli

Stimuli were the same 60 combinations of facial iden-tity and facial expression of emotion that were usedearlier, in Experiment 2.

Subjects

All subjects who served in Experiment 2, plus oneadditional female subject, served here. The subjects,aged 18-29 years, had normal or corrected-to-normalvision, and were paid for their participation2.

Procedure

To quantify the similarity among faces of variousidentity-emotion combinations, we used the method oftriads (Torgerson, 1958), a procedure that worked wellin a study using non-emotional versions of Wilson faces(Yotsumoto et al., 2007). At trial onset, a fixation crossappeared in the center of the screen for 100 ms. Afterthe cross disappeared, three faces appeared, equidistantfrom the center of the screen and from each other. Thesubject then made two responses, first selecting the twofaces that were perceptually most similar, and then thetwo most different. In order to remind the subject of thecurrent criterion, “similar” or “different” appeared inthe center of the screen. Choices were made by mouseclicks on the desired faces. Once a particular face hadbeen selected, a thin blue frame surrounded it in orderto confirm the choice. If the subject mis-clicked on someface, the selection could be erased by clicking again onthe same face. Subjects were not instructed to base theirjudgments on any particular characteristic (i.e. eyes,mouth, expression, hair), but were instructed to go at arelatively rapid pace so as to reduce in-depth scrutinyof each face.

With 60 different faces, presenting each face with ev-ery other pair of faces would have required 205,320trials (60×59×58) for just a single presentation of eachpossible trio. To reduce that number to a more manage-able value, we resorted to a balanced-incomplete blockdesign (BIBD) (Weller & Romney, 1988). BIBD balancesthe number of times each pair of stimuli is seen, butdoes not balance the frequency with which each triad isseen. When a given pair was presented, the third facevaried so that repetitions were of pairs and not triads.Each subject served in one session of 930 trials, whichallowed each pair of faces to be presented six times.Each subject saw the 930 trials in a different order. Inorder to satisfy the stringent requirements for BIBD, weadded two additional neutral faces to our set of 60 faces;

results from those neutral faces were discarded withoutanalysis.3 Trials were self-paced.

Subjects’ responses were entered into a dissimilar-ity matrix, where the pair judged most similar was as-signed a value of three, the pair judged most differentwas assigned a value of one, and the undesignated pairreceived an intermediate value, two. Individual sub-ject’s matrices were summed and then normalized bythe frequency with each pair of faces occurred duringthe experiment. The resulting matrix was processedby SPSS’s ALSCAL routine using a Euclidean distancemodel to relate dissimilarities in the data to correspond-ing distances in the perceptual representation. We eval-uated the MDS solutions that were achieved in onethrough four dimensions.

Results and DiscussionTo determine the appropriate number of dimensions

to use from the series of MDS solutions, we comparedthe goodness of fit achieved in varying numbers of di-mensions. The results, expressed as values of STRESS(Borg & Groenen, 2005), can be seen in Figure 3. Thefact that STRESS declines from a 1-dimensional solu-tion to a 2-dimensional solution, and remains relativelyconstant thereafter, supports the idea that 2-dimensionscomprise an appropriate space within which to analyzethe MDS results. Using Kruskal’s rules of thumb as aguide (Borg & Groenen, 2005), the MDS fit to the datacan be characterized as between “good” and “excel-lent”. The quality of the 2-dimensional MDS solutionis confirmed by in Figure 4. The figure shows a scatter-plot, known as a Shepard plot, where all 1,770 interpointdistances ( 60×59

2 ) from the MDS solution have been plot-ted against their corresponding dissimilarities from theoriginal dataset. The pattern of scatter around valuesfor which x = y (redline in graph) gives no evidence ofan unaccounted for non-linear component.

Figure 5 is a graphical presentation of the MDS-baseddistribution of the 60 stimulus faces. As an aid to inter-preting the distribution of points, we regionalized theplot, identifying the obvious facets within the plot (Borg& Groenen, 2005). The facets, which were determinedby eye, identify compact regions of interest within theplot. Note first that the points corresponding to eachcategory of emotion form clusters that occupy distinctlocations in the 2-dimensional space. These clusters ofemotional categories range from sadness (at extremeleft), through neutral, fear, and anger (at extreme right).Note also that the clusters vary in their internal dis-persions, with the points corresponding for neutral

2 Experiment 3 was run before Experiment 2. As a result,no subjects in Experiment 3 had been previously exposed tothe synthetic faces.

3 It is not possible to construct BIBDs for any arbitrary num-ber of stimulus triplets. Adding the two extra neutral facesallowed us to find a BIBD algorithm that could be used withour stimuli (Fisher & Yates, 1963).

8 GALSTER & SEKULER

00..0022

00..0033

00..0044

00..0055

00..0066

11 22 33 44

Stress

Dimensions

Figure 3. Stress associated with MDS solutions in varyingnumbers of dimensions. Stress, which is inversely related togoodness of fit, declines substantially from a 1-dimensional toa 2-dimensional solution, and is roughly constant thereafter.

faces being more tightly clustered than the points corre-sponding to any of the other categories. Probably, thisreflects the fact that with the neutral category there wasonly one exemplar per face, but with all other categorieseach face had three exemplars, one for each intensity ofemotion.

At a finer level of analysis, within each category ofemotion, the points that represent the three exemplarsof the same face (e.g., a particular male face express-ing three levels of anger) are adjacent to one another.This confirms that both emotional intensity and facialidentity entered into subjects’ perceptual judgments ofsimilarity/dissimilarity. Finally, excepting the relativelytight cluster for neutral faces, within each category ofemotion, male faces are separated from female faces,as shown by the dashed lines separating each categorycluster into gender subsets. If one takes the locationof any identity’s neutral exemplar as the starting point,the distance of that same identity’s faces in any of theemotion categories provides a rough index of that face’sability to express that emotion. Interestingly, on this in-dex, we see no consistent difference between exemplarscorresponding to male and female faces. For example,although female faces seem to express fear and angersomewhat more vividly than do our male faces, malefaces more vividly express sadness. We return to thispoint later, in the General Discussion.

Previous studies demonstrated that short term recog-nition for various stimulus types is strongly depen-dent upon the similarity or dissimilarity among items.This result has been shown for patches of differing hue(Nosofsky & Kantner, 2006), complex sounds (Visscher,Kaplan, Kahana, & Sekuler, 2007), Gabor patches (Ka-hana, Zhou, Geller, & Sekuler, 2007), and emotionless

2D

Distances

Dissimilarities

Figure 4. Shepard plot based on the 2-dimensional MDSsolution. Points represent the interpoint distances from theMDS solution space versus the original dissimilarities amongitems. Note the relatively narrow scatter of points around thered 1:1 line, and the absence of obvious systematic trends tothe scatter.

synthetic faces (Sekuler, McLaughlin, Kahana, Wing-field, & Yotsumoto, 2006). This relationship is strongestwhen similarity/dissimilarity is expressed as a scalarvalue derived by summing the similarity or dissimilar-ity of a p item to each of the items in memory (Clark &Gronlund, 1996; Sekuler & Kahana, 2007). To assess thisvariable’s influence on short-term recognition memoryfor facial emotions, we regressed P(Yes) responses fromExperiment 2 against the summed separations in MDSspace of the p and each study face (Figure 5). Althoughboth correct and false recognitions should be relatedto summed separation, that relationship ought to differfor the two types of responses. In particular, one ex-pects that the proportion of correct recognitions will ex-ceed the proportion of false recognitions. Therefore weperformed separate linear regressions for the P(Yes) onTarget trials, where a Yes response constituted a correctrecognition, and on Lure trials, where a Yes responseconstituted a false recognition.

To carry out the regressions, we first ordered trialsof each type according to the trial’s summed distancebetween p and study items. Then, trials were separatedinto a number of equally populous bins, each represent-ing a different mean level of summed distances. Themean proportions of hits and false recognitions werethen calculated for the trials in each bin. Figure 6 showsthe result of the linear regressions on Target trials andon Lure trials. Letting D represent the mean value ofsummed distances, the best fitting lines were

P(Yes) = −.0398D + 0.812P(Yes) = −.0556D + 0.633

for Targets and Lures, respectively. The difference inslopes was not reliable, as the 95% confidence intervalsaround the two values overlapped. For fits to Targetand Lure trials the root mean squared deviations were0.032 and 0.039; the values of r2 associated with the

FACIAL IDENTITY AND EMOTION 9

Figure 5. The 2D MDS solution derived from subjects’ triadic similarity judgements. Labels are of the form E.M/F#.1/2/3,where E is the first letter of the emotion’s designation (Angry, Fearful, Neutral, Sad), M/F# is the gender and identity numberof the Wilson face, and 1/2/3 is the intensity of emotion (33, 67, and 100%). Red, black, green and blue lines enclose points thatrepresent to Sad, Neutral, Fearful and Angry faces, respectively. Within each region (except that for Neutral faces) a dashedline separates male (M) faces from female (F) faces. Note that faces representing different intensities of the same emotion andfacial identity are near each other, and that the order of facial identity relations are maintained across expression types.

two best fitting lines were 0.71 and 0.88. Together, thispair of values indicate that D, the independent vari-able derived from simple, Euclidean measurements inthe 2-dimensional similarity space, does a reasonablejob of predicting the proportion of recognitions, correctand false, produced by different combinations of studyfaces and probes.

To gauge the importance of including not only facialemotion but also facial identity when computing theregressor variable, we carried out additional regressionanalyses, on each of the one-dimensional projectionsfrom the two-dimensional space shown in Figure 5.The two sets of unidimensional regressors were gen-erated by collapsing first over the dimension of facialidentity, and then over the dimension of emotion cat-egory and intensity. In both of the regressions the re-gressor was summed dissimilarity, but one regressiontook into account only dissimilarity in the identity di-mension, while the other took account of dissimilaritycorresponding to emotional categories and their inten-sities.

When either dimension of the similarity space wasmuted as a regressor, the resulting values of r2 de-creased. When facial identity was muted, leaving emo-tion alone to enter into the computations, regressions

for Target and Lure produced r2 values of just 0.48 and0.52. The substantial decline when facial identity isomitted from the perceptual space parallels the demon-stration in Experiment 2 that facial identity influencessubjects’ memory-based judgments of facial emotion(see Figure 2) . Finally, when facial emotion was omittedfrom the perceptual space, leaving only facial identityto enter into the computations, the quality of the modeldeclines further, to just r2=0.32 and 0.26, for Target andLure trials respectively. Although a precise numeri-cal comparison is impossible, this weak but non-zerovalue of r2 is reminiscent of the finding in Experiment2 that facial identity alone, with no match on emotionwhatever, is able to attract some small proportion ofmemory-based false recognitions (Figure 2).

That the two regression lines in Figure 6 are separatedvertically suggests a point of theoretical importance. Asexplained above, some models of short term memory(e. g., Clark & Gronlund, 1996; Sekuler & Kahana,2007) assume that recognition responses are based on aprobe item’s summed similarity to the items in memory.For that variable, summed similarity, the regressionsFigure 6 substituted summed Euclidean distances asmeasured in Figure 5. (This substitution was made toavoid the need to make assumptions about the function

10 GALSTER & SEKULER

0.0

0.2

0.4

0.6

0.8

1.0

0 1 2 3 4 5 6 7 8

ProportionYESResponses

Summed Distances (Probe-Study Items)

Target trials

Lure trials

Figure 6. Proportion of Yes responses on Target trials (correctrecognitions) and on Lure trials (false recognitions) as a func-tion of the summed distances of the probe to the study items.Data from individual trials were ordered and then put intoequally populous bins for the purposes of linear regression.

relating similarity to distance.) If no variable other thansummed distance governed recognition responses, thenwould expect that for any given value of summed dis-tance, P(Yes) on Target trials would equal P(Yes) on Luretrials. In terms of Figure 6, the mostly-shared valuesof summed distance should cause the regression linesfor Lure and Target trials to be superimposed. That thetwo are not superimposed suggests that some addi-tional variable is influencing the recognition responses.Whatever this additional variable might prove to be, itspotency can be gauged from the substantial y-interceptdifference between the two regression lines in Figure 6.This additional variable might take the form of contex-tual information, for example, information about theepisodic or temporal context in which the probe facemost recently appeared (Howard & Kahana, 2002). Acomplete characterization of this variable, though, willrequire additional experiments targeted at this hypoth-esis.

General Discussion

The memory-based recognition judgments from Ex-periment 2 showed that emotional expression and facialidentity are processed in combination with one another.Experiment 3 produced an analogous result, but for per-ceptual processing of the same set of faces. The factthat the metric 2-dimensional perceptual representa-tion served well in accounting for memory performance

suggests that there is at least a rough structural paral-lel between the psychological space in which percep-tual similarity/dissimilarity judgments are made andthe psychological space used for short term recognitionmemory.

This result extends a previous finding with emotion-neutral synthetic faces in which variation and blend-ing of facial identities produced stimuli that lay alongthe orthogonal axes and along the diagonal of a threedimensional space. In that study, Yotsumoto et al.(2007) showed that the MDS-derived perceptual spacefor these stimuli preserved much, but not all of thestructure that had been built into the stimuli, and couldbe used in a stimulus-oriented account of recognitionmemory.

Parallels between memory and perception

Experiments 2 and 3 point to structural parallels be-tween the space in which faces are represented per-ceptually and the space in which faces are representedin short term memory. Although far from dispositive,these parallels are consistent with an emergent pro-cesses account of short-term memory, which holds thatthe temporary storage working memory makes impor-tant use of brain regions shared with sensory processing(Postle, 2006; Jonides et al., 2008). There are some obvi-ous limitations to our ability to elucidate possible par-allels between perception and short-term memory forfacial emotions. One set of limitations arises from factthat the data presented here represent averages oversubjects, over trials, and, in some cases, over diversesets of study faces and probes. A second set of limita-tions reflects the relatively narrow sample of facial iden-tities and emotions represented in Experiments 2 and 3.For reasons explained below, those experiments had toexclude Wilson faces that displayed smiles. Addition-ally, the faces used in those experiments were derivedfrom just the six individuals whose emotional expres-sions were judged to match best the intended targetcategories. Clearly, the possibility of parallels betweenprocessing in memory and perception should be testedfurther, with faces representing an even wider range ofidentities and emotional expressions.

Perceptual processing

Experiment 1 identified a set of “neutral” Wilsonfaces that most reliably were judged as expressing noemotion. In that process, subjects attempted to catego-rize the emotion, if any, that was shown by each briefly-presented face. A response was considered correct if itwas consistent with the category of emotion that hadbeen imparted to the face. As expected, success in cat-egorizing an emotion varied with the intensity of thatemotion (Woodworth & Schlosberg, 1954). Our sub-jects made essentially no errors when an emotion waseither 67 or 100% of the maximum for that emotion.

FACIAL IDENTITY AND EMOTION 11

However, subjects did miscategorize displays of emo-tion when emotional intensity was weak, i. e., whenan emotion was just 33% of its maximum intensity. Webelieve that this result is not an artifact of our havingused synthetic faces or even of our brief presentations.In fact, this finding reproduces classic results using a va-riety of face types and display durations (Woodworth &Schlosberg, 1954). More recently, Pollick, Hill, Calder,and Paterson (2003) used point-light displays of mov-ing faces and demonstrated a small, but significant re-lationship between duration and perceived emotionalexpression, with longer durations generally producingemotions that were judged more intense. In an accountof the circuitry responsible for recognition of emotions,Adolphs (2002) described a possible neural basis forthe sharpening of emotion judgements with duration,namely time-varying feed-forward and feedback inter-actions among components in a distributed network ofbrain regions.

It is interesting to note that in classic studies whensubjects were asked to categorize weak displays of emo-tion not only were their judgments frequently wrong,but also those judgments were easily manipulated, e.g.,by a misleading label provided by the researcher Fern-berger (1928). These powerful emotion-priming effectsare not easily accommodated by face recognition mod-els that emphasize the modularity of participating com-ponents (for example, Bruce & Young, 1986; Burton,Bruce, & Johnston, 1990). However, an interactive,distributed network model such as the one offered byAdolphs (2002) explicitly acknowledges the contribu-tion to emotion recognition of various forms of non-visual information, such as verbal labels.

Our characterization of perceptual processing wasbased on similarity/dissimilarity judgments with triadsof faces (Experiment 3). The perceptual space producedfrom subjects’ judgments by multidimensional scaling(Figure 5) suggested interesting differences in the emo-tional displays by male and female faces. In particular,within each emotional category, male and female facesoccupied distinct, but neighboring locations in the per-ceptual space. As an illustration, consider faces’ loca-tions within the sadness category. There, male facesoccupy more extreme locations relative to their neu-tral positions than do female faces, but the orderingby gender is reversed within the fear and anger cate-gories. We cannot tell if these results reflect genuinegender-related differences, or are somehow a result ofour having initially culled faces from the large library offaces. Recall that we limited our stimulus faces to onesthat most convincingly appeared as neutral when theywere intended to be neutral. To determine whether thegender-related differences were genuine would requirean extensive, random collection of faces that had notbeen been selected, as Experiment 1 did, against somepotentially biasing characteristic. It may well be thatsome sexually dimorphic characteristics of human facesmake it differentially easy for male and female faces to

express diagnostic signs of particular emotions (Becker,Kenrick, Neuberg, Blackwell, & Smith, 2007).

Short-term memory

In Experiment 2, the accuracy with which an ex-pressed emotion was recognized increased when theprobe face’s identity matched the identity of one ormore items in memory. So even though facial iden-tity was notionally irrelevant to the task, it still exerteda strong influence on subjects’ recognition responses.This result, which is illustrated in Figure 2, was con-firmed by the regression analysis in which the recog-nition results were far better fit using a regressor thattook both identity and expression into account thanusing a regressor that treated the two as independentdimensions.

Our study examined faces that were expressing theemotions of anger, sadness, and fear, but intentionallyomitted faces with happy expressions. Such faces wereexcluded from our stimulus set because many nomi-nally “happy” Wilson faces struck pilot subjects as pe-culiar, that is, not genuinely happy. The smiles weredescribed as inauthentic or insincere, what has beendubbed “professional” smiles generated out of cour-tesy rather than as an expression of happiness (Wallace,1997, p. 289). We believe the perceived lack of sincerityarose from the fact that Wilson faces’ lips were not per-mitted to part. In order to control simple, luminance-based cues that might on their own differentiate oneemotion from another, Goren and Wilson (2006)’s al-gorithm for expression generation foreclosed a displayof teeth, even when the Zygomaticus major muscle, aprincipal participant in smiles, would have moved thecorresponding real face’s skin enough to have partedits lips. This design decision was consequential be-cause human judges attempt to identify a happy ex-pression they make important use of the parting of thelips (Smith, Cottrell, Gosselin, & Schyns, 2005).

In Experiment 2, subjects most accurately recognizeda probe’s emotion when that probe’s identity matched theidentity of both study items (see Figure 2). When twostudy items happened to share the same identity, ourexperimental design forced those two faces to displayemotions drawn from different categories. Essentially,this constraint meant that subjects were exposed to twodifferent views of the same person’s face when encodingwhat they had seen. In principle, these two views couldenable subjects to extract some consistent facial struc-tures from the two study faces, which could be helpfulin evaluating the probe face’s emotion. In everyday so-cial encounters, we often need to read and make senseof emotions that are expressed on the faces of strangers,that is, people with whom our prior experience is lim-ited or even non-existent. In such circumstances, weare deprived of access to the optimum neutral baseline–that person’s own face when it is at rest— and mustjudge and then react to the emotional expression more

12 GALSTER & SEKULER

or less on its own, in an absolute framework that verylikely is influenced by long-term memory of previousencounters, with other people. Previous experiencemaking emotion judgments might explain Elfenbeinand Ambady (2002)’s finding that facial emotions ex-pressed by members of one culture are more accuratelyperceived and interpreted by members of that sameculture than they are by people who are not membersof that culture. It may be that this difference arises fromexperience-based learning of the characteristics of thatculture’s “typical” face(s) when the faces are physicallyat rest.

As just mentioned, experience with particular facesor types of faces seems to boost perceptual interpreta-tion of the emotional expressions on those faces. It is im-portant to note, therefore, that both our perceptual andmemory measurements were made with faces that wereunfamiliar to the subjects, at least upon initial view-ing. Although the best-known neuropsychological testsof face recognition memory depend upon face stimulithat are not familiar to people being tested (Duchaine& Weidenfeld, 2003), many key results on face per-ception and memory come from studies in which atleast some of the faces were familiar to the subjects (e.g.Bruce & Young, 1986; Troje & Kersten, 1999; Laeng &Rouw, 2001; Ge, Luo, Nishimura, & Lee, 2003; Ganel &Goshen-Gottstein, 2004; Kaufmann & Schweinberger,2004; Bredart & Devue, 2006). The distinction betweenfamiliar and novel faces may bear on the parallels weobserved between perception and recognition memory.Arguably, increasing familiarity with a particular faceallows recognition memory to approach a limit set bya subject’s ability to discriminate visually some changein the face. Thus, with images of Mao Tse-Tung as stim-uli, Ge et al. (2003) demonstrated that Chinese subjects’ability to recognize small distortions of Mao’s face frommemory was a close match to their ability to discrim-inate such distortions perceptually. This striking con-vergence between recognition memory and perceptionwas confirmed with faces of people who, although notfamous, highly-familiar to particular subjects (Bredart& Devue, 2006).

Representing facial emotion

Figure 5 shows the 2-dimensional MDS descriptionof the perceived similarities among our stimuli, whichvaried in facial emotion, facial identity and gender.This representation of similarities among multidimen-sional stimuli can be compared to some well-knowncharacterizations of the psychological space represent-ing emotion. For example, based on categorical judg-ments of facial expressions shown in a series of pho-tographs Schlosberg (1952) proposed a 2-dimensionalrepresentation for emotion. That representation, in-spired by Newton’s circular color surface, was spannedby two principal, opponent axes (Pleasant-Unpleasant,and Attention-Rejection). In analogy to the location

of desaturated colors in Newton’s color system, neu-tral expressions occupied a position at the center ofthe surface. Even a casual comparison of Schlosberg(1952)’s space and our own suggest that the two are notwell correlated. For example, in Figure 5, judgmentsof neutral expressions do not occupy a special positionfar from other emotions, but are sandwiched betweenthe judgments for two other emotions. Russell (1980)derived an alternative representation of affective space,starting from categorical judgments of emotion-relatedwords. Although the resulting space shared the di-mensionality of Schlosberg’s space, emotions were notmapped onto a circular surface, but onto to a circum-plex, that is, different emotions were systematically ar-ranged around the perimeter of a circle (for a rigorousdescription of circumplex structure, see Revelle & Ac-ton, 2004). Note that in a circumplex model, a neutralemotion would occupy a location on the circumferenceof the circle, surrounding by other emotions (compareFigure 5). Russell’s basic result has been supported by anumber of subsequent studies, including ones in whichsubjects judged the similarity of photographs of facialexpressions (see Posner, Russell, & Peterson, 2005, forreview).

The path of points in Figure 5’s MDS solution sug-gest a roughly-circular arc. The absence of smiling facesfrom the stimulus set in Experiments 3 could account fora portion of the obvious gap in what otherwise mighthave been a path consistent with the circumplex model.Potential data from other emotions that were omittedfrom our stimulus set could account for another portionof the gap. However, some inter-emotion proximitiesin Figure 5 are not consistent with their counterparts inRussell’s original circumplex account of affective states,or in a more recent revision (Posner et al., 2005) of thataccount. Most notably, compared to the standard cir-cumplex model, in Figure 5 the locations occupied byangry faces and fearful faces are interchanged. Neitherthis discrepancy, nor the obvious gap in what might bea fragment of a circular path bear strongly on the va-lidity of a circumplex model of emotion. After all, thejudgments upon which our MDS solution was basedreflect variation not only in category of emotion, butalso in intensity of emotion, facial identity and gen-der. Unavoidably, the outcome of the scaling processreflects not only the underlying structure of psycholog-ical or affective processes, but also the way in whichthose underlying processes happen to be sampled bywhatever particular set of stimuli is used to probe thatstructure (e. g., Bosten, Robinson, Jordan, & Mollon,2005). That limitation points the way toward futurestudies in which an analysis like that presented hereis expanded to encompass a more complete range ofemotions and subjects from populations that would beexpected to exhibit atypical representational spaces (e.g., D’Argembeau & Linden, 2004; Habak, Wilkinson, &Wilson, 2008).

FACIAL IDENTITY AND EMOTION 13

References

Adolphs, R. (2002). Neural systems for recognizing emotion.Current Opinion in Neurobiology, 12(2), 169–177.

Agam, Y., & Sekuler, R. (2007). Interactions between work-ing memory and visual perception: an ERP/EEG study.Neuroimage, 36(3), 933–942.

Ashby, F. G., & Maddox, W. T. (1990). Integrating informationfrom separable psychological dimensions. Journal of Exper-iment Psychology: Human Perception & Performance, 16(3),598–612.

Baudouin, J.-Y., Martin, F., Tiberghien, G., Verlut, I., & Franck,N. (2002). Selective attention to facial emotion and identityin schizophrenia. Neuropsychologia, 40(5), 503–511.

Becker, D. V., Kenrick, D. T., Neuberg, S. L., Blackwell, K. C.,& Smith, D. M. (2007). The confounded nature of angrymen and happy women. Journal of Personality and SocialPsychology, 92, 179–190.

Borg, I., & Groenen, P. J. F. (2005). Modern multidimensionalscaling : theory and applications (2nd ed ed.). New York:Springer.

Bosten, J. M., Robinson, J. D., Jordan, G., & Mollon, J. D.(2005). Multidimensional scaling reveals a color dimen-sion unique to ’color-deficient’ observers. Current Biology,15(23), R950-2.

Brainard, D. H. (1997). The psychophysics toolbox. SpatialVision, 10, 443-446.

Bredart, S., & Devue, C. (2006). The accuracy of memory forfaces of personally known individuals. Perception, 35(1),101–106.

Bruce, V., & Young, A. (1986). Understanding face recognition.British Journal of Psychology, 77, 305-327.

Burton, A. M., Bruce, V., & Johnston, R. A. (1990). Understand-ing face recognition with an interactive activation model.British Journal of Psychology, 81, 361-380.

Clark, S. E., & Gronlund, S. D. (1996). Global matching modelsof recognition memory: How the models match the data.Psychonomic Bulletin & Review, 3, 37–60.

D’Argembeau, A., & Linden, M. van der. (2004). Identity butnot expression memory for unfamiliar faces is affected byageing. Memory, 12(5), 644–654.

D’Argembeau, A., & Linden, M. Van der. (2007). Facial ex-pressions of emotion influence memory for facial identityin an automatic way. Emotion, 7(3), 507–515.

Darwin, C. (1872). The expression of the emotions in man andanimals. London: J. Murray.

Duchaine, B. C., Parker, H., & Nakayama, K. (2003). Nor-mal recognition of emotion in a prosopagnosic. Perception,32(7), 827–838.

Duchaine, B. C., & Weidenfeld, A. (2003). An evaluation oftwo commonly used tests of unfamiliar face recognition.Neuropsychologia, 41(6), 713-720.

Ekman, P. (1993). Facial expression and emotion. AmericanPsychologist, 48, 384–392.

Ekman, P., & Friesen, W. V. (1975). Unmasking the Face. Oxford:Prentice-Hall.

Elfenbein, H. A., & Ambady, N. (2002). On the universalityand cultural specificity of emotion recognition: a meta-analysis. Psychological Bulletin, 128, 203–235.

Fernberger, S. W. (1928). False suggestion and the Pideritmodel. Americal Journal of Psychology, 40, 562–568.

Fisher, R. A., & Yates, F. (1963). Statistical tables for biological,agricultural, and medical research (6th ed., rev. and enl. ed.).New York: Hafner Publishing Co.

Ganel, T., & Goshen-Gottstein, Y. (2004). Effects of familiarityon the perceptual integrality of the identity and expressionof faces: the parallel-route hypothesis revisited. Journal ofExperimental Psychology: Human Perception and Performance,30(3), 583–597.

Garner, W. R. (1974). The processing of information and struc-ture. Potomac, Md.: L. Erlbaum Associates; distributed byHalsted Press, New York.

Ge, L., Luo, J., Nishimura, M., & Lee, K. (2003). The lastingimpression of chairman Mao: hyperfidelity of familiar-facememory. Perception, 32(5), 601–614.

Gelder, B. de, Pourtois, G., & Weiskrantz, L. (2002). Fearrecognition in the voice is modulated by unconsciouslyrecognized facial expressions but not by unconsciouslyrecognized affective pictures. Proceedings of the NationalAcademy of Sciences, USA, 99, 4121–4126.

Gelder, B. de, Vroomen, J., Pourtois, G., & Weiskrantz, L.(1999). Non-conscious recognition of affect in the absenceof striate cortex. Neuroreport, 10, 3759–3763.

Goren, D., & Wilson, H. R. (2006). Quantifying facial expres-sion recognition across viewing conditions. Vision Research,46(8-9), 1253–1262.

Gosselin, P., & Kirouac, G. (1995). Decoding facial emotionalprototypes. Canadian Journal of Experimental Psychology, 49,313–329.

Habak, C., Wilkinson, F., & Wilson, H. R. (2008). Aging dis-rupts the neural transformations that link facial identityacross views. Vision Research, 48(1), 9–15.

Hasselmo, M. E., Rolls, E. T., & Baylis, G. C. (1989). The roleof expression and identity in the face-selective responsesof neurons in the temporal visual cortex of the monkey.Behavioural Brain Research, 32, 203–218.

Howard, M. W., & Kahana, M. J. (2002). A distributed rep-resentation of temporal context. Journal of MathematicalPsychology, 46, 269-299.

Humphreys, G. W., Donnelly, N., & Riddoch, M. J. (1993). Ex-pression is computed separately from facial identity, andit is computed separately for moving and static faces: neu-ropsychological evidence. Neuropsychologia, 31, 173–181.

Hurlbert, A. (2007). Colour constancy. Current Biology, 17(21),R906-7.

Isaacowitz, D., Wadlinger, H., Goren, D., & Wilson, H. (2006).Selective preference in visual fixation away from negativeimages in old age? An eye tracking study. Psychology &Aging, 21, 40-48.

Jonides, J., Lacey, S., & Nee, D. (2005). Processes of workingmemory in mind and brain. Current Directions in Psycho-logical Science, 14, 2–5.

Jonides, J., Lewis, R. L., Nee, D. E., Lustig, C. A., Berman,M. G., & Moore, K. S. (2008). The mind and brain of short-term memory. Annual Review of Psychology, 59, 193–224.

Kahana, M. J., Zhou, F., Geller, A. S., & Sekuler, R. (2007).Lure-similarity affects episodic recognition: Detailed testsof a noisy exemplar model. Memory & Cognition, 35, 1222–1232.

14 GALSTER & SEKULER

Kaufmann, J. M., & Schweinberger, S. R. (2004). Expressioninfluences the recognition of familiar faces. Perception, 33,399-408.

Kosslyn, S. M., Thompson, W. L., Kim, I. J., & Alpert, N. M.(1995). Topographical representations of mental images inprimary visual cortex. Nature, 378(6556), 496–498.

Laeng, B., & Rouw, R. (2001). Canonical views of faces andthe cerebral hemispheres. Laterality, 6(3), 193–224.

Loffler, G., Gordon, G. E., Wilkinson, F., Goren, D., & Wilson,H. R. (2005). Configural masking of faces: evidence forhigh-level interactions in face perception. Vision Research,45(17), 2287–2297.

Loffler, G., Yourganov, G., Wilkinson, F., & Wilson, H. R.(2005). fMRI evidence for the neural representation offaces. Nature Neuroscience, 8, 1386–1391.

Nosofsky, R. M., & Kantner, J. (2006). Exemplar similarity,study list homogeneity, and short-term perceptual recog-nition. Memory & Cognition, 34(1), 112–124.

Pollick, F. E., Hill, H., Calder, A., & Paterson, H. (2003).Recognising facial expression from spatially and tempo-rally modified movements. Perception, 32(7), 813–826.

Posamentier, M. T., & Abdi, H. (2003). Processing faces andfacial expressions. Neuropsychology Review, 13(3), 113–143.

Posner, J., Russell, J. A., & Peterson, B. S. (2005). The circum-plex model of affect: an integrative approach to affectiveneuroscience, cognitive development, and psychopathol-ogy. Development and Psychopathology, 17(3), 715–734.

Postle, B. R. (2006). Working memory as an emergent propertyof the mind and brain. Neuroscience, 139(1), 23–38.

Revelle, W., & Acton, G. S. (2004). Evaluation of ten psycho-metric criteria for circumplex structure. Methods of Psycho-logical Research Online, 9, 1-27.

Russell, J. A. (1980). A circumplex model of affect. Journal ofPersonality and Social Psychology, 39, 1161–1178.

Russell, R., Biederman, I., Nederhouser, M., & Sinha, P. (2007).The utility of surface reflectance for the recognition of up-right and inverted faces. Vision Research, 47, 157–165.

Sacks, O. (1985). The man who mistook his wife for a hat. NewYork: Harper-Collins.

Sadr, J., Jarudi, I., & Sinha, P. (2003). The role of eyebrows inface recognition. Perception, 32, 285-293.

Schlosberg, H. (1952). The description of facial emotion interms of two dimensions. Journal of Experimenal Psychology,44, 229–237.

Sekuler, R., & Kahana, M. J. (2007). A stimulus-orientedapproach to memory. Current Directions in PsychologicalScience, 16.

Sekuler, R., McLaughlin, C., Kahana, M. J., Wingfield, A., &Yotsumoto, Y. (2006). Visual recognition and temporalorder information are both well-preserved in aging. Psy-chology & Aging, 21(3), 632–637.

Sergent, J., Ohta, S., MacDonald, B., & Zuck, E. (1994). Seg-regated processing of facial identity and emotion in thehuman brain: A PET study. Visual Cognition, 1, 349-369.

Smith, M. L., Cottrell, G. W., Gosselin, F., & Schyns, P. G.(2005). Transmitting and decoding facial expressions. Psy-chological Science, 16, 184–189.

Sternberg, S. (1966). High speed scanning in human memory.Science, 153, 652–654.

Torgerson, W. S. (1958). Theory and methods of scaling. NewYork: J Wiley & Sons.

Troje, N. F., & Kersten, D. (1999). Viewpoint-dependent recog-nition of familiar faces. Perception, 28(4), 483–487.

Visscher, K. M., Kaplan, E., Kahana, M. J., & Sekuler, R. (2007).Auditory short-term memory behaves like visual short-term memory. Public Library of Science – Biology, 5(3), e56.

Wallace, D. F. (1997). A supposedly fun thing I’ll never do again:Essays and arguments. Boston: Little, Brown and Co.

Weller, S. C., & Romney, A. K. (1988). Systematic Data Collection(Vol. 10). Newbury Park, CA: Sage Publications.

Whalen, P. J., Kagan, J., Cook, R. G., Davis, F. C., Kim, H.,Polis, S., McLaren, D. G., Somerville, L. H., McLean, A. A.,Maxwell, J. S., & Johnstone, T. (2004). Human amygdalaresponsivity to masked fearful eye whites. Science, 306,2061.

Wilson, H. R., Loffler, G., & Wilkinson, F. (2002). Syntheticfaces, face cubes, and the geometry of face space. VisionResearch, 42, 2909-2923.

Wilson, H. R., & Wilkinson, F. (2002). Symmetry perception:a novel approach for biological shapes. Vision Research, 42,589–597.

Woodworth, R. S., & Schlosberg, H. (1954). Experimentalpsychology. New York: Holt.

Yotsumoto, Y., Kahana, M. J., Wilson, H. R., & Sekuler, R.(2007). Recognition memory for realistic synthetic faces.Memory & Cognition, 35, 1233–1244.