AJR:202, June 2014 1267

In addition to the expressed concern that another person can recognize a patient from reconstructed images, the advent of increas-ingly accurate and more widely available fa-cial recognition technologies suggests that such reconstructions might be applied intru-sively or maliciously to identify specific pa-tients. This identification would be possible in many circumstances, such as publication of imaging data in federated databases and as figures in the peer-reviewed literature, in which conventional anonymization has been performed to eliminate all traditional patient and study identifiers [4]. The possibility of computer identification of anonymized faces is more than speculative. Facebook offers fa-cial recognition software services to its more than 1 billion active users, and a casual web search of U.S. patents related to face recog-nition technology returns more than 22,000

Implications of Surface-Rendered Facial CT Images in Patient Privacy

Joseph Jen-Sho Chen1 Krishna Juluru2 Tara Morgan3 Ryan Moffitt4 Khan M. Siddiqui5 Eliot L. Siegel1

Chen JJ, Juluru K, Morgan T, Moffitt R, Siddiqui KM, Siegel EL

1 Department of Diagnostic Radiology and Nuclear Medicine, University of Maryland School of Medicine, 22 S Greene St, Baltimore, MD 21201. Address correspondence to J. J. S. Chen (jchen1@umm.edu).

2 Department of Radiology, Weill Cornell Medical College, New York, NY.

3 Department of Radiology and Biomedical Imaging, UCSF, San Francisco, CA.

4 Lutherville, MD.

5 Hinsdale, IL.

Medica l Phys ics and Informat ics • Or ig ina l Research

AJR 2014; 202:1267–1271

0361–803X/14/2026–1267

© American Roentgen Ray Society

Workstations for 3D and multi-planar reconstruction of CT im-ages have become ubiquitous in the routine practice of diagnos-

tic imaging. These tools not only enhance our ability to diagnose diseases, such as fractures, aneurysms, and neoplasms, but also can assist in therapeutic planning [1–3]. Surface recon-structions of CT, MRI, and other imaging da-tasets can be useful from the clinical and teaching perspectives to show superficial fea-tures that may otherwise be evident at routine physical inspection. High-resolution recon-structions from isotropic or near-isotropic da-tasets can be used to create detailed images of a patient’s face. This capability raises privacy concerns because the images could be used to identify a patient despite de-identification or anonymization of the patient’s protected health information.

Keywords: CT, facial recognition, HIPAA, privacy

DOI:10.2214/AJR.13.10608

Received January 18, 2013; accepted after revision September 11, 2013.

OBJECTIVE. Three-dimensional and multiplanar reconstruction of CT images has be-come routine in diagnostic imaging. The technology also facilitates surface reconstruction, in which facial features and, as a result, patient identity may be recognized, leading to risk of violations of patient privacy rights. The purpose of this study was to assess whether volunteer viewers can recognize faces on 3D reconstructed images as specific patients.

SUBJECTS AND METHODS. A total of 328 participants were included: 29 patients underwent clinically indicated CT of the maxillofacial sinuses or cerebral vasculature and were also photographed (group A); 150 patients volunteered to have their faces photographed (group B); and 149 observers reviewed the images. Surface-reconstructed 3D images of group A were generated from CT data, and digital photographs of both groups A and B were acquired for a total of 179 facial photographs. Image reviewers were recruited with a web-based questionnaire that required observers to match surface-reconstructed images generated from CT data with randomized digital photographs from among the 179 photographs. Data analyses were performed to determine the ability of observers to successfully match surface-reconstructed images with facial photographs.

RESULTS. The overall accuracy among the image observers was approximately 61%. No significant differences were found with regard to sex, age, or ethnicity and accuracy of im-age observers.

CONCLUSION. Image reviewers were relatively poor at even side-by-side matching of patient photographs with 3D surface-reconstructed images. This finding suggests that suc-cessful identification of patients using surface-rendered faces may be a relatively difficult task for observers.

Chen et al.Facial CT and Patient Privacy

Medical Physics and InformaticsOriginal Research

Dow

nloa

ded

from

ww

w.a

jron

line.

org

by L

ibra

ry O

f M

edic

ine

on 0

6/05

/14

from

IP

addr

ess

128.

103.

149.

52. C

opyr

ight

AR

RS.

For

per

sona

l use

onl

y; a

ll ri

ghts

res

erve

d

1268 AJR:202, June 2014

Chen et al.

links [5–10]. The technology is inexpensive, easily accessible, and ubiquitous.

Public Law 104–191, also known as HIPAA, stipulates safeguards “to ensure the integrity and confidentiality” of a patient’s health information [11]. To maintain a pa-tient’s confidentiality, his or her identity must be kept private and secure. One of the HIPAA provisions covers what are referred to as iden-tifiers of the patient, which include “full face photographic images and any comparable im-ages” [11, 12]. Although 3D surface-recon-structed images from a CT study are not ex-plicitly mentioned in the HIPAA regulations, concern has been expressed that these might fall into the category of comparable images or be so construed in legal proceedings.

Only a few investigators have reported concerns about whether surface-reconstruct-ed CT and MR images are recognizable in a way that would allow identification of specif-ic patients [13, 14]. And we are not aware of any previous attempts to quantify or estab-lish a baseline for patient identification by a human being using these reconstructed im-ages. This question has major implications for cross-sectional imaging studies of the maxillofacial sinuses, temporal bones, ce-rebral vasculature, brain, face, and head. If facial recognition is readily made from the imaging data, then any dataset that includes the face could be deemed impossible to de-identify for teaching, research, and other ac-ademic purposes or to protect an individual’s legal right to privacy. This has already led to explorations of obscuration techniques de-signed to protect against inappropriate facial recognition [15] (Chen, JJ, et al., presented at the 2011 annual meeting of the Radiological Society of North America). The purpose of this study was to evaluate whether observers can readily associate 3D reconstructed im-

ages of a patient’s facial features with a pho-tograph of the same patient.

Subjects and MethodsStudy Overview

This prospective study involving retrospec-tive CT examinations was approved by our insti-tutional review board, and informed consent was obtained from all participants. Our institution-al review board considered as study participants all 149 observers, 150 patient volunteers for facial photographs, and 29 volunteers for facial photo-graphs who underwent CT that included the face on the same day. This represented a total of 328 observers and patients who met all criteria for en-rollment in the study. Each was categorized as fol-lows according to his or her role in the study.

Head and Neck CT Plus Facial PhotographTwenty-nine patients (group A) underwent clini-

cally indicated CT of the maxillofacial sinuses or cerebral vasculature and acquisition of a photo-graph of the face. After informed consent was ob-tained and brief demographic data (age, sex, eth-nicity) were collected, each patient’s head and neck area was digitally photographed in various views, including a frontal view. The photographs were ob-tained with a 6-megapixel, 4× optical zoom digital camera (Lumix DMC-FX3K, Panasonic) with pro-prietary Mega optical image stabilization. All dig-ital photographs were acquired with a consistent and uniform combination of image stabilization, flash, optical zoom, and portrait settings. The pa-tients stood 4 feet (1.2 m) from the digital camera in front of a solid, cream-colored background with a uniform and consistent light source (Fig. 1A). Al-though 30 patients had enrolled, one was excluded from the study because of an incomplete CT dataset.

CT image acquisition was performed with a 64-MDCT scanner (Sensation 64, Siemens Health-care) with the following parameters: section thick-ness, 0.75 mm; rotation time, 1 s/rotation; pitch,

0.9; tube potential, 120 kVp; exposure, 63 mA. The CT datasets of the patients were used to cre-ate 3D surface-reconstructed images of each sub-ject’s facial features and contours. A 3D worksta-tion (AquariusNET server, TeraRecon) was used for image reconstruction. Images were reconstruct-ed with a standard color template with minimal window leveling applied to create a pink skin color (Fig. 1B). The person performing the surface re-construction was unaware of (blinded to) the iden-tity and demographic information of the patients and had no access to the digital photographs when recreating the 3D images.

Facial Photograph OnlyThe second set of study participants (group B)

included 150 patients recruited while in the di-agnostic imaging area for miscellaneous, routine clinical imaging studies. After informed con-sent was obtained and demographic surveys were completed, digital photographs of their faces were acquired in the same manner, with same digi-tal camera, and under the same conditions as for group A. The resulting database consisted of fa-cial photographs of 29 patients in group A and 150 patients in group B and 29 3D surface reconstruc-tions from CT data obtained during CT examina-tions of the face for patients in group A.

Image ReviewersA total of 149 image observers-reviewers were

registered to participate in matching the 3D sur-face-rendered images of individuals in group A with the combined and randomized digital photo-graphs of individuals from groups A and B. After providing informed consent, the image reviewers completed brief demographic surveys.

Web-Based QuestionnaireA web-based questionnaire was created to al-

low image reviewers to perform and document the task of selecting and matching a 3D reconstructed

A

Fig. 1—69-year-old man who underwent CT of maxillofacial sinuses.A, Digital photograph.B, Three-dimensional reconstructed image.

B

Dow

nloa

ded

from

ww

w.a

jron

line.

org

by L

ibra

ry O

f M

edic

ine

on 0

6/05

/14

from

IP

addr

ess

128.

103.

149.

52. C

opyr

ight

AR

RS.

For

per

sona

l use

onl

y; a

ll ri

ghts

res

erve

d

AJR:202, June 2014 1269

Facial CT and Patient Privacy

image of a patient (group A) with a subset of im-ages selected from a superset that included photo-graphs from groups A and B. Each image reviewer was required to register and sign in to the secure website. After informed consent and demograph-ic information were obtained, the reviewers were randomly presented a set of 58 questions. For each of these 58 questions, the web-based program pre-sented a surface-reconstructed 3D image of a pa-tient in group A along with a simultaneous array of five frontal view digital photographs of patients selected from groups A and B and a choice of none of the above. Each of the 29 3D-rendered images was presented in two different questions: In one a correct match appeared as one of the choices in the array of photographs, and in the other it did not. The presentations of the digital photographs in the arrays and of the 3D-rendered images were randomized. After deciding on one of the six pos-sible choices (one of the five photographs of faces or none of the above), the reviewer was automat-ically advanced to the next question until all 58 questions were answered. All reviewers complet-ed all 58 questions.

Statistical AnalysisStatistical analysis was performed with SAS

software (SAS Institute, version 9.2) and Excel 2007 (Microsoft, version 12). Data analyses were performed with various statistical techniques, in-cluding the chi-square test, Student t test, ANO-VA, and Pearson product moment correlation.

ResultsDemographics

Most of the participants in group A (pa-tients with both photographs and CT scans of the face) were African American (n = 16 [55.2%]), followed by white (n = 13 [44.8%]). The average age was 56.5 ± 13.1 (SD) years, and the group included 24 men and five women. Similarly, and reflecting our pa-tient base, most of the participants in group B (patients with photographs but not CT scans of the face) were African American (n = 93 [62.0%]), followed by white (n = 55 [36.7%]). The average age of group B was 53.5 ± 12.7 years, and the group included 130 men and 20 women. No statistically signifi-cant differences were noted between groups A and B in age (p = 0.25), sex (p = 0.73), or ethnicity (p = 0.66).

Of the 149 image reviewers, 29 (19.5%) were radiology residents, eight (5.4%) were radiology fellows, 15 (10.1%) were radi-ology attending physicians, 53 (35.6%) were other health care professionals, and 44 (29.5%) were in professions other than

health care. Most of the reviewers were white (n = 96 [64.4%]), followed by Asian (n = 35 [23.5%]), African American (n = 14 [9.4%]), and other (n = 4 [2.7%]). The mean age of the image reviewers was 36.8 ± 11.7 years, and the group included 81 men and 68 women.

Facial RecognitionThe overall accuracy (defined as percent-

age of questions for which the reviewers ei-ther correctly matched a digital photograph with the 3D reconstructed facial image or correctly selected none of the above) was 61%. The sensitivity (when an image review-er correctly matched the photograph with the reconstructed image) was 88%, and the specificity (when an image reviewer correct-ly chose none of the above when the recon-structed image did not match any of the ran-domly displayed photographs) was 50%.

The accuracy of identifying the matches declined as the age of the image reviewers increased, although the association was not strong (r = –0.137). Men selected the cor-rect answer choice 61.4% of the time, where-as women were correct 60.5% of the time. Whites were correct 61.9% of the time; Af-rican Americans, 62.2%; Asians, 58.1%; and those who reported themselves as other rac-es, 50.8%. In addition, differences in correct responses by occupational status were small: attending radiologists, 59.9%; radiology fel-lows, 59.9%; radiology residents, 62.4%; other health care professionals, 59.7%; and professionals not in health care, 63.1%. Sta-tistical analysis of the image reviewers’ de-mographic data showed no significant per-formance differences between the sexes (p = 0.59), ethnicities (p = 0.66), or occupa-tions (p = 0.96). In cases in which an image reviewer’s reported ethnicity was the same as that of the group A member, the success-ful identification rate was 65.8% (p < 0.001).

DiscussionHIPAA was designed to protect individu-

al privacy and security rights in a fast-grow-ing digital health care environment. As more health care and government institutions and for-profit industries work to create electronic medical records for the general population—with goals of cutting ever-ballooning health care costs and reducing life-altering medical errors—accessibility of the records and se-curity are growing concerns.

The concerns are magnified in the field of radiology, one of the early adopters in trans-forming medical imaging (a data-intensive

and challenging part of the complete elec-tronic medical record) into the digital age. Since the first implementation of PACS tech-nology, digital transformation has acceler-ated, becoming the standard for storing and displaying medical images [16]. With im-provements in storage and display technolo-gy that allow instantaneous access to an indi-vidual’s medical images for not only clinical but also teaching purposes, protecting these datasets is a priority. These datasets allow radiologists and other clinicians to diagnose diseases, prepare for a range of therapies, and determine whether specific treatments are effective. The data-rich information col-lected at a routine CT examination can be visualized with 3D-rendering software. For example, a routine CT dataset of a patient’s head and neck can be used not only to diag-nose pathologic conditions but also to recon-struct facial features in 3D. A 3D-rendered volume of a patient’s face appearing, for ex-ample, as an otherwise anonymized figure in a journal publication, an educational pre-sentation or webcast, or an open teaching file could be construed as a HIPAA viola-tion if the patient can be recognized by an-other individual or by use of facial recogni-tion software. The purpose of our study was to evaluate whether a 3D-rendered image of a patient’s facial features can be used by an-other person to identify that patient, poten-tially violating his or her private health in-formation rights.

In our study, we found that the ability to match a 3D facial image reconstructed from the CT dataset of an individual to his or her digital photograph was relatively poor when the correct answer was none of the above (50% correct) and relatively good (88% cor-rect) when one of the choices was the correct answer. This resulted in a combined accu-racy of less than 66%, even when the eth-nicity of the reviewer was the same as that of the subject of the reconstructed facial CT image. The poor performance in the cases in which the answer was none of the above was unexpected given that the candidate photo-graphs of the group A and B patients were not matched by age, sex, ethnicity, or habi-tus, which intuitively should make the task of matching reconstructed images to the cor-rect photographs easier. The possible choic-es thus contained mixed ages, ethnicities, habituses, and even sexes, which we thought would have been major clues in the match-ing exercise as opposed to the more difficult task of selecting the correct match from sim-

Dow

nloa

ded

from

ww

w.a

jron

line.

org

by L

ibra

ry O

f M

edic

ine

on 0

6/05

/14

from

IP

addr

ess

128.

103.

149.

52. C

opyr

ight

AR

RS.

For

per

sona

l use

onl

y; a

ll ri

ghts

res

erve

d

1270 AJR:202, June 2014

Chen et al.

ilar patients. The image reviewers also had no time constraint for each matching ques-tion and were able to use all clues available to them. The reviewers also were able to com-pare the images side by side, in contrast to real-life situations, in which an individual would primarily rely on memory to identify a 3D-rendered image (e.g., to recognize an acquaintance or public figure).

Reviewers were correct on only one half of the questions for which the correct answer was none of the above, a scenario that more closely simulates real-life facial recognition. An individual may see hundreds of faces on a given day, and typically only a small per-centage of these are friends and acquain-tances. Our data suggest that 3D-rendered CT images may be difficult for most peo-ple, medically trained or not, to match with known faces without simultaneously seeing that person or his or her photograph.

The web-based questionnaire was de-signed to test the ability to identify a surface-reconstructed CT image and to counteract potential biases. All of the digital photo-graphs, answer choices to the questions, and order of presentation for the 3D reconstruct-ed facial images were randomized to reduce recall bias and bias resulting from the order of the digital photographs. Although bias re-sulting from photograph-order effects has not been proved, we thought it was important to minimize the effect by presenting the choices

randomly because the order of presentation may be similar to the well-described bias as-sociated with the answer order of a multiple choice question or name-order on an election ballot [17, 18]. Krosnick et al. [19] studied presidential and local elections and found sig-nificant name-order effects on election out-comes, especially when a presidential can-didate’s name was listed first compared with last. In the closely contested U.S. presidential election of 2000, name-order effects are sus-pected of playing a role in the outcome of the election [19]. Similarly, an image reviewer may be more inclined to choose the first or second photographs seen instead of giving all photographs the same consideration.

In our study, neither the age nor the sex of reviewers influenced the ability to match 3D reconstructed CT images and photographs of study subjects. However, the ethnicity of the image reviewers did play a role in accuracy, as would be expected. When the ethnicity of an image reviewer was concordant with that of the subject of the surface-reconstruct-ed image, a correct match was made slight-ly less than 2 of 3 times, which is slightly (≈ 5%) but significantly higher than the rate when ethnicity differed. This finding lends support to previous research findings on ra-cial bias in facial recognition and eyewitness identification that showed observers were more accurate in identifying individuals of their own races [20–25].

One limitation of this study was the use of a simultaneous array of digital photographs (Fig. 2) for matching a 3D reconstructed im-age. Although simultaneous photo arrays have been used extensively by law enforce-ment agencies, this method of identifying individuals has been found to have inherent flaws [26, 27]. In our research study, some of the recommendations made by Wells et al. [27], such as randomization of photograph presentation, were implemented to reduce biases associated with simultaneous arrays. However, some associated systematic flaws might not have been eliminated. In future work, sequential image arrays requiring re-call of faces rather than direct comparison may be informative. In law enforcement, use of such arrays has been considered to re-duce false-positive identifications compared with use of simultaneous arrays [28]. We be-lieve that use of the simultaneous arrays like-ly increased accuracy substantially in com-parison with a study design that would have required matching based on recall of a previ-ously displayed image.

Another limitation was sample size and lack of representative diversity in study sub-jects compared with the general population, given that patients were selected from a De-partment of Veterans Affairs population more heavily weighted with older African American men than are the general U.S. and world pop-ulations. A more representative age, sex, and ethnic sampling of the general population may be considered for a future study. It also might have been helpful to have used more than one digital photograph of group A patients to de-termine the relative ability of human observers to match photographs of the face taken at dif-ferent angles or slightly different perspectives, including variability in lighting.

It also might have been instructive to cre-ate a control group in which subjects matched identical photographs rather than matching a photograph to a 3D surface-reconstructed image to establish a baseline for human pho-tographic matching. We anticipated that ob-servers would have nearly 100% accuracy with that task using a photo array paradigm but did not formally test this supposition.

Future studies should also investigate whether sophisticated software designed to perform human facial recognition with more objective information, such as relative size and position of the eyes, nose, and mouth and shape of the face, could outperform hu-man observers. It would also be interesting to explore the implementation of technical ap-proaches to decreasing the ability of human or machine observers to match photographs of patients with their surface-rendered facial CT or MR images. For example, it may be pos-sible to introduce alterations in facial surface contours or to perform surface feature warp-ing in some way to obscure facial features while preserving purpose-relevant data.

ConclusionThe results of our study suggest that un-

der side-by-side conditions individuals can be identified on 3D reconstructed facial im-ages in the same manner that they are on digital photographs (assuming digital pho-tography as the reference standard) in ap-proximately 61% of cases when there is a none of the above option. Even when the re-viewer and the patient have the same ethnic-ity, the accuracy is less than 66%. Although the findings do not directly simulate real-life situations, the findings provide a baseline for accuracy of patient identification under op-timal circumstances in which the reviewer can compare the reconstructed image with

Fig. 2—Screen shot shows sample presentation from web-based computer questionnaire with one 3D reconstructed image, five digital photographs in simultaneous array, and none of above box. Image reviewer was asked to select one of five digital photographs or none of above by clicking left mouse key. In this sample, correct digital photograph that matches 3D reconstructed image is at bottom left.

Dow

nloa

ded

from

ww

w.a

jron

line.

org

by L

ibra

ry O

f M

edic

ine

on 0

6/05

/14

from

IP

addr

ess

128.

103.

149.

52. C

opyr

ight

AR

RS.

For

per

sona

l use

onl

y; a

ll ri

ghts

res

erve

d

AJR:202, June 2014 1271

Facial CT and Patient Privacy

candidate photographs and the patients are not similar in sex, ethnicity, age, or habitus. Our research findings could be interpreted as simultaneously supporting the suggestion that remarkably lifelike surface-reconstruct-ed images can be successfully matched with patient photographs but also suggesting that this task of identification can be quite diffi-cult without familiar cues such as hair, skin color and markings, and the differences in the patient’s face when in the supine position for CT. The findings also establish a baseline for recognition rates for comparison purpos-es for other studies, including those conduct-ed with computer applications for automated facial recognition and for attempts at obscur-ing patient identity without distorting ana-tomic features below the skin level.

References 1. Fishman EK, Ney DR, Heath DG, Corl FM, Hor-

ton KM, Johnson PT. Volume rendering versus

maximum intensity projection in CT angiogra-

phy: what works best, when, and why. Radio-

Graphics 2006; 26:905–922

2. Rhea JT, Rao PM, Novelline RA. Helical CT and

three-dimensional CT of facial and orbital injury.

Radiol Clin North Am 1999; 37:489–513

3. Tomandl BF, Kostner NC, Schempershofe M, et

al. CT angiography of intracranial aneurysms: a

focus on postprocessing. RadioGraphics 2004;

24:637–655

4. Morgan TA, Flanders AE, Olmsted WW, Steen-

burg SD, Siegel EL. Challenges encountered and

lessons learned in initial experience with the next

generation of interactive radiology literature in Ra-

dioGraphics. RadioGraphics 2012; 32:929–934

5. Huang H, He H. Super-resolution method for face

recognition using nonlinear mappings on coher-

ent features. IEEE Trans Neural Netw 2011;

22:121–130

6. Lynnerup N, Clausen MI, Kristofferson AM,

Steglich-Arnholm H. Facial recognition and laser

surface scan: a pilot study. Forensic Sci Med

Pathol 2009; 5:167–173

7. Biswas S, Bowyer KW, Flynn PJ. Multidimen-

sional scaling for matching low-resolution face

images. IEEE Trans Pattern Anal Mach Intell

2012; 34:2019–2020

8. Prabhu U, Heo J, Savvides M. Unconstrained pose

invariant face recognition using 3D generic elastic

models. IEEE Trans Patter Anal Mach Intell 2011

Jun 7 [Epub ahead of print]

9. Kamgar-Parsi B, Lawson W, Kamgar-Parsi B. To-

wards development of a face recognition system

for watch-list surveillance. IEEE Trans Pattern

Anal Mach Intell 2011 Mar 8 [Epub ahead of print]

10. Arashloo SR, Kittler J. Energy normalization for

pose-invariant face recognition based on MRF

model image matching. IEEE Trans Pattern Anal

Mach Intell 2011; 33:1274–1280

11. Centers for Medicare and Medicaid Services web-

site. Health Insurance Portability and Account-

ability Act of 1996. www.cms.gov. Accessed Oc-

tober 30, 2012

12. U.S. Department of Health and Human Services.

Other requirements relating to uses and disclo-

sures of protected health information. Washing-

ton, DC: U.S. Government Printing Office,

2005:772–777

13. Prior FW, Brunsden B, Hildebolt C, et al. Facial

recognition from volume-rendered magnetic reso-

nance imaging data. IEEE Trans Inf Technol

Biomed 2009; 13:5–9

14. Mazura JC, Juluru K, Chen JJ, Morgan TA, John

M, Siegel EL. Facial recognition software success

rates for the identification of 3D surface recon-

structed facial images: implications for patient

privacy and security. J Digit Imaging 2012;

25:347–351

15. Budin F, Zeng D, Ghosh A, Bullitt E. Preventing

facial recognition when rendering MR images of

the head in three dimensions. Med Image Anal

2008; 12:229–239

16. Reiner B, Siegel E, Protopapas Z, Hooper F, Ghe-

brekidan H, Scanlon M. Impact of filmless radiol-

ogy on frequency of clinician consultations with

radiologists. AJR 1999; 173:1169–1172

17. Cronbach LJ. Further evidence on response sets

and test design. Educ Psychol Meas 1950; 10:3–31

18. Miller JM, Krosnick JA. The impact of candidate

name order on election outcomes. Public Opin Q

1998; 62:291–330

19. Krosnick JA, Miller JA, Tichy MP. An unrecog-

nized need for ballot reform: effects of candidate

name order. New York, NY: Oxford University

Press, 2004

20. Bothwell RK, Brigham JC, Malpass RS. Cross-

racial identification. Pers Soc Psychol Bull 1989;

15:19–25

21. Deffenbacher KA, Vetter T, Johanson J, O’Toole

AJ. Facial aging, attractiveness, and distinctive-

ness. Perception 1998; 27:1233–1243

22. Furl N, Phillips PJ, O’Toole AJ. Face recognition

algorithms and the other-race effect: computa-

tional mechanisms for a developmental contact

hypothesis. Cogn Sci 2002; 26:797–815

23. O’Toole AJ, Deffenbacher KA, Valentin D,

McKee K, Huff D, Abdi H. The perception of face

gender: the role of stimulus structure in recognition

and classification. Mem Cognit 1998; 26:146–160

24. O’Toole AJ, Peterson J, Deffenbacher KA. An

“other-race effect” for categorizing faces by sex.

Perception 1996; 25:669–676

25. Brigham JC, Barkowitz P. Do “they all look

alike”? The effect of race, sex, experience, and at-

titudes on the ability to recognize faces. J Appl

Soc Psychol 1978; 8:306–318

26. Malpass RS, Lindsay RC. Measuring line-up fair-

ness. Appl Cogn Psychol 1999; 13:S1–S7

27. Wells GL, Small M, Penrod S, Malpass RS, Fule-

ro SM, Brimacombe CA. Eyewitness identifica-

tion procedures: recommendations for lineups

and photospreads. Law Hum Behav 1998;

22:603–647

28. Lindsay RC, Wells GL. Improving eyewitness

identifications from lineups: simultaneous versus

sequential lineup presentation. J Appl Psychol

1985; 70:556–564

Dow

nloa

ded

from

ww

w.a

jron

line.

org

by L

ibra

ry O

f M

edic

ine

on 0

6/05

/14

from

IP

addr

ess

128.

103.

149.

52. C

opyr

ight

AR

RS.

For

per

sona

l use

onl

y; a

ll ri

ghts

res

erve

d