Virtual Reality for Psychological Assessment in Clinical Practicepsychology.unt.edu/~tparsons/pdf/Parsons (Phillips 2016... · 2016-09-13 · Virtual Reality for Psychological Assessment

Virtual Reality for Psychological Assessment in Clinical Practice

Thomas D. Parsons and Amanda S. PhillipsUniversity of North Texas

Clinical psychologists are often asked to make recommendations regarding a client’sability to function in everyday activities (e.g., work, classroom, or shopping). Commonapproaches include a combination of paper-and-pencil (including some computerautomated) assessments, behavioral observations of the client’s performance duringtesting, and self-report measures about their perceived deficits. From this combinationof assessments, observations, and self-reports, the psychologist is expected to makepredictions about the client’s ability to return to the classroom, return to work, andsuccessfully complete other activities of daily living. While there are advantages to thisapproach, there are also some shortcomings—perhaps most notable is the lack ofecological validity. Recent advances in virtual reality technologies allow for enhancedcomputational capacities for administration efficiency, stimulus presentation, auto-mated logging of responses, and data analytic processing. These virtual environmentsallow for controlled presentations of emotionally engaging background narratives toenhance affective experience and social interactions. Within this context virtual realitycan allow psychologists to offer safe, repeatable, and diversifiable assessments of realword functioning. Although there are a number of purported advantages of virtualreality technologies, there is still a need for establishing the psychometric properties ofvirtual reality assessments and interventions. This review investigates the advantagesand challenges inherent in the application of virtual reality technologies to psycholog-ical assessments and interventions.

Keywords: virtual reality, neuropsychology, psychological assessment, ecologicalvalidity

Clinical psychologists are increasingly beingasked to make prescriptive statements aboutevery-day functioning. Unfortunately, resultsfrom many psychological tests are not easilygeneralizable to real-world functioning. Com-mon approaches include a combination of his-tory taking, self-reports, paper-and-pencil cog-nitive assessments, and the psychologist’sobservations of the client’s behavior. From thiscombination, the psychologist is expected tomake predictions about the client’s ability toreturn to the classroom, return to work, and

successfully complete other activities of dailyliving. The limitations inherent in this processhave led to increasing calls for assessmentmethods that provide more generalizable dataabout client functioning (Burgess et al., 2006;Jurado & Rosselli, 2007). Virtual reality (VR)assessments have been developed to provide anenhanced understanding of client functioning inactivities of daily living (Campbell et al., 2009;Matheis et al., 2007).

While VR assessments are often presented astools for neurocognitive assessment in researchsettings, this article aims to provide an introduc-tion to VR assessment for clinical practice.First, current assessment methods are comparedwith assessment in virtual environments. Next,examples of VR assessments with clinical caseexamples are provided. Finally, considerationsfor the adoption of VR technologies in clinicalpractice are explored. We propose that the ad-dition of VR to current psychological assess-ment batteries can improve the generalizabilityof test results and increase the utility and rele-

This article was published Online First August 18, 2016.Thomas D. Parsons and Amanda S. Phillips, Department

of Psychology, Computational Neuropsychology and Sim-ulation Lab, University of North Texas.

Correspondence concerning this article should be ad-dressed to Thomas D. Parsons, Computational Neuropsy-chology and Simulation Lab, National Academy of Neuro-psychology, Department of Psychology, University ofNorth Texas, 1155 Union Circle #311280, Denton, TX76203. E-mail: [email protected]

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Practice Innovations © 2016 American Psychological Association2016, Vol. 1, No. 3, 197–217 2377-889X/16/$12.00 http://dx.doi.org/10.1037/pri0000028

197

mailto:[email protected]

http://dx.doi.org/10.1037/pri0000028

vance of psychologists’ recommendations toclients.

Self-Reports and Behavioral Observations

Although traditional assessment methodsprovide valuable information, there are a num-ber of limitations in using this approach (seeTable 1). Determining a client’s functional ca-pabilities requires precise control over the en-vironment and the ability to adjust the potencyor frequency of stimuli (White et al., 2014).This control is difficult to ensure in the tradi-tional assessment environment. Given that psy-chological tests are typically administered one-on-one in a controlled environment; thepsychologist may not receive a clear picture ofthe client’s cognitive functions in everyday ac-tivities. Although psychologists often use infor-mation from parent and teacher reports to get anidea of the patient’s everyday functioning, stud-ies indicate that the agreement between parentsand teachers is modest. For example, the con-cordance between parents and teachers on diag-nosing attention-deficit-hyperactivity disorder(ADHD) varies from .30 to .50 depending onthe behavioral dimensions being rated (Bieder-

man, Faraone, Milberger, & Doyle, 1993; Bie-derman, Keenan, & Faraone, 1990; Mitsis, Mc-Kay, Schulz, Newcorn, & Halperin, 2000;Newcorn et al., 1994; de Nijs et al., 2004).Additionally, there is often not a strong overlapbetween these rating scales and standard tests ofcognitive functioning, suggesting that these as-sessments may be reflective of different aspectsof behavior (see Table 1 for a list of potentialconfounds).

Normative Comparison ofCognitive Performance

For many psychologists, the way to get be-yond these limitations is to emphasize a norma-tive comparison approach to psychological as-sessment. For example, a psychologist can givea continuous performance test (CPT) to assess acognitive construct (e.g., attentional process-ing). Results from the CPT reveal the client’sperformance in areas of inattentiveness, impul-sivity, sustained attention, and vigilance. Thepsychologist can compare the client’s CPT re-sults with reference groups to determinewhether a client is performing as would be

Table 1Comparison of the Advantages and Disadvantages of Traditional Assessment

Advantages Disadvantages

Self-reports Allows identification of areas of clientfunctioning that have been impacted orpreserved.

Controlled environment limits thegeneralizability of results.

Provides information about the client’sperceptions of their performance.

Third-party reports may have poor inter-raterreliability.

Third-party reports provide data about clientfunctioning in various environments.

Third-party reports often fail to stronglyoverlap with cognitive tests.

Potential influence of self-report format onresponses.

Unable to measure automatic processing.Post hoc appraisal of past behavior.

Behavioral observation Provides insights into how the client may reactwhen faced with difficult tasks.

Controlled environment limits ecologicalvalidity of observations.

Normative comparison Allows for measurement of expectedperformance.

Tests are often measure abstract constructsrather than functioning.

Quantifies discrepancies in expectedperformance.

Many cognitive tests were developed for usewith normal populations, not for functionalassessment.

Tests may not be representative of real-worldsituations.

Tests lack generalizability to real-worldsituations.

Test sensitivity and potency cannot be altered.Time-consuming and expensive

administration.

198 PARSONS AND PHILLIPS

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

expected given their achievements and educa-tional attainment.

An unfortunate limitation of this approach isthat the tests used are typically based uponconstruct-driven assessments that do little toreplicate the client’s everyday activities. Fur-thermore, many of the tests commonly usedtoday were never meant for clinical assess-ments. Burgess and colleagues (2006) point outthat cognitive construct measures like theTower of London and the Wisconsin Card Sort-ing Test (WCST) were not originally designedto be used as clinical measures. Instead, thesemeasures were found to be useful tools forcognitive assessment among normal popula-tions and then later found their way into theclinical realm to aide in assessing constructsthat are important to carrying out real-worldactivities. This approach forces the psychologistto rely on measures that were designed for pur-poses other than predictions of real-world func-tioning. Burgess and colleagues (2006) arguedthat we need assessments capable of broadeningour understanding about the ways in which thebrain enables persons to interact with their en-vironment and organize everyday activities. Al-though many cognitive tests do give us someinsight into the client’s everyday performance,they do not provide direct knowledge aboutshortcomings in the functional capabilities ofthe client, which limits the accuracy and utilityof the psychologist’s recommendations (Chay-tor & Schmitter-Edgecombe, 2003; Manchester,Priestley, & Jackson, 2004).

Virtual Environments

One potential answer to the issues raisedabove is the addition of virtual environments to

the psychologists’ cognitive assessment battery,which allow clients to become immersed withina computer-generated simulation (Campbell etal., 2009; Matheis et al., 2007). Potential virtualenvironment use in assessment and rehabilita-tion of human cognitive processes is becomingrecognized as technology advances, and has anumber of advantages (see Table 2). While acomplete listing of all the virtual environment-based psychological assessments is outside thescope of this article, we provide examples vir-tual environments with case examples.

Virtual Classroom

Various virtual classrooms are being vali-dated for assessment of supervisory attentionalprocessing. Virtual Classrooms are simulationenvironments that were designed to assess po-tential attention deficits by embedding cognitivetasks (e.g., CPT; Stroop) into a virtual class-room environment (Díaz-Orueta et al., 2013;Iriarte et al., 2016; Parsons et al., 2007; Rizzo etal., 2006; see Table 3). In these virtual class-rooms the participant is seated at one of thedesks and is surrounded by desks, children, ateacher, and a white board much like theywould be in a real-world classroom. Variouscognitive tasks can be presented on the white-board in the front of the room and the partici-pant performs a task (e.g., Stroop or CPT) withauditory (e.g., airplane passing overhead, avoice from the intercom, and the bell ringing)and visual (e.g., children passing notes, a childraising his hand, the teacher answering theclassroom door, and principal entering theroom) distractors in the background. The num-ber and frequency of these distractors can beadjusted based on age, grade level, or other

Table 2Comparison of the Advantages and Disadvantages of Assessment in a Virtual Environment

Advantages Disadvantages

Precise and consistent presentation of stimuli. Construct-driven assessments will still lack some ecological validity.Clinician control of dynamic stimuli. Lack of guidelines for development, administration, and interpretation.Greater ecological validity. Pediatric and geriatric clients may need additional guidance.Availability of function led assessments. Geriatric clients may have trouble adjusting to the visual environment.Programs administer, record, score, and analyze data. Clients with Autism Spectrum Disorder may become over stimulated.Environmental stimuli and test sensitivity can be

adjusted.Clients with psychiatric disorders or high suggestibility may respond

unfavorably.Environment can be adjusted to reflect clients’

everyday surroundings.

199VIRTUAL REALITY IN PSYCHOLOGICAL PRACTICE

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Tab

le3

Rec

ent

Con

stru

ct-D

rive

nSt

udie

s(W

ithi

nP

ast

10Y

ears

)U

sing

aV

irtu

alC

lass

room

Env

iron

men

t

Stud

ySa

mpl

eR

esea

rch

desi

gnT

radi

tiona

lT

ests

Res

ults

Ada

ms

etal

.(2

009)

N�

35bo

ysag

es8–

14ye

ars;

19pa

rtic

ipan

tsw

ithA

DH

Dw

ere

com

pare

dto

16ag

e-m

atch

edco

ntro

ls.

Com

pari

son

ofpa

rtic

ipan

tpe

rfor

man

ceon

the

Con

tinuo

usPe

rfor

man

ceT

est

with

and

with

out

the

Vir

tual

Cla

ssro

om.

•B

ehav

ior

Ass

essm

ent

Syst

emfo

rC

hild

ren

(BA

SC)

•V

IGIL

Con

tinuo

usPe

rfor

man

ceT

est

•G

reat

ersp

ecifi

city

was

foun

dfo

rV

irtu

alC

PT.

•A

lthou

ghdi

ffer

ence

sbe

twee

nth

etw

ogr

oups

wer

eno

tsi

gnifi

cant

,a

stro

ngtr

end

was

obse

rved

for

corr

ect

targ

etid

entifi

catio

nan

dco

mm

issi

oner

rors

.B

ioul

acet

al.

(201

2)N

�36

boys

ages

7–10

year

s;20

part

icip

ants

with

AD

HD

wer

eco

mpa

red

to16

age-

mat

ched

cont

rols

.

AD

HD

and

cont

rols

child

ren

wer

efir

stte

sted

with

the

trad

ition

alco

mpu

teri

zed

CPT

.A

fter

10m

inth

eyw

ere

test

edw

ithth

evi

rtua

lC

PT.

•C

ontin

uous

Perf

orm

ance

Tes

t(C

PTII

)•

Con

ners

’pa

rent

sra

ting

scal

e•

Chi

ldB

ehav

ior

Che

ckL

ist

•St

ate

Tra

itIn

vent

ory

Anx

iety

Inve

ntor

y

•A

DH

Dpa

rtic

ipan

tssh

owed

asi

gnifi

cant

perf

orm

ance

decr

emen

t;de

crea

seof

corr

ect

hits

;an

din

crea

sed

reac

tion

time.

•A

DH

Dch

ildre

npe

rfor

med

wor

seth

anco

ntro

lson

both

the

Vir

tual

CPT

and

the

trad

ition

alco

mpu

teri

zed

CPT

.D

íaz-

Oru

eta

etal

.(2

014)

N�

57pa

rtic

ipan

tsw

ithA

DH

Dbe

twee

nth

eag

esof

6–16

year

s(2

6.3%

fem

ale)

.

Con

verg

ent

valid

ityst

udy

betw

een

both

aV

irtu

alC

PTan

dtr

aditi

onal

com

pute

rize

dC

PT.

Com

pare

dch

ildre

nun

derg

oing

med

ical

trea

tmen

tw

itha

non-

med

icat

edgr

oup

•C

onne

rs’

Con

tinuo

usPe

rfor

man

ceT

est

•W

ISC

-III

(Sel

ecte

dsu

btes

ts)

•B

oth

Vir

tual

CPT

and

the

trad

ition

alco

mpu

teri

zed

CPT

show

edsi

gnifi

cant

corr

elat

ions

.•

The

Vir

tual

CPT

(but

not

Con

ners

’C

PT)

was

able

todi

ffer

entia

tebe

twee

nA

DH

Dch

ildre

nw

ithan

dw

ithou

tph

arm

acol

ogic

altr

eatm

ent

(ina

ttent

ion,

impu

lsiv

ity,

proc

essi

ngsp

eed,

mot

orac

tivity

,an

dat

tent

ion

focu

s).


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Tab

le3

(con

tinu

ed)

Stud

ySa

mpl

eR

esea

rch

desi

gnT

radi

tiona

lT

ests

Res

ults

Gilb

oaet

al.

(201

1)N

�54

.29

with

Neu

rofib

rom

atos

isty

pe1

(NF1

);69

%fe

mal

e;m

ean

age

12.2

).25

cont

rols

72%

fem

ale;

mea

nag

e�

12.2

).

Com

pari

son

ofV

irtu

alC

PTan

dth

etr

aditi

onal

test

s.C

ross

sect

iona

lde

sign

.

•C

onne

rs’

Pare

ntR

atin

gSc

ales

-Rev

ised

•Si

gnifi

cant

diff

eren

ces

betw

een

the

NF1

and

cont

rols

onom

issi

oner

rors

and

com

mis

sion

erro

rsin

the

Vir

tual

CPT

.•

Poor

erpe

rfor

man

ceby

NF1

child

ren.

•Si

gnifi

cant

corr

elat

ions

betw

een

num

ber

ofta

rget

sco

rrec

tlyid

entifi

ed,

the

num

ber

ofco

mm

issi

oner

rors

,an

dre

actio

ntim

e.G

ilboa

etal

.(2

015)

N�

76.

41ch

ildre

nag

es8–

16w

ithac

quir

edbr

ain

inju

ry;

35ag

e-an

dge

nder

-m

atch

edco

ntro

ls.

Cro

ss-s

ectio

nal

desi

gn.

•T

est

ofE

very

day

Atte

ntio

nfo

rC

hild

ren

•W

echs

ler

Abb

revi

ated

Scal

eof

Inte

llige

nce

(Mat

rix

Rea

soni

ngan

dV

ocab

ular

y)•

Con

ners

Pare

ntR

atin

gSc

ales

-Rev

ised

•Si

gnifi

cant

betw

een

grou

pdi

ffer

ence

sfo

rnu

mbe

rof

targ

ets

corr

ectly

iden

tified

inth

eV

irtu

alC

PT.

•45

%of

the

child

ren

with

AB

Isu

ffer

edm

arke

dde

ficits

insu

stai

ned

atte

ntio

non

the

Vir

tual

CPT

.•

Atte

ntio

nal

perf

orm

ance

was

foun

dto

bere

late

dto

age,

age

atin

jury

/dia

gnos

isan

dtr

eatm

ent.

Iria

rte

etal

.(2

016)

N�

1282

child

ren

ages

6to

16.

Ano

rmat

ive

stud

y.Si

ngle

appl

icat

ion

with

desc

ript

ive

desi

gn.

Non

e•

Res

ults

wer

ecl

uste

red

into

diff

eren

tca

tego

ries

for

post

erio

ran

alys

is.

•D

iffe

renc

esby

age

and

gend

erw

ere

anal

yzed

,re

sulti

ngin

14gr

oups

,7

per

sex

grou

p.•

Dif

fere

nces

betw

een

visu

alan

dau

dito

ryat

tent

ion

wer

eal

soob

tain

ed. (tab

leco

ntin

ues)


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Tab

le3

(con

tinu

ed)

Stud

ySa

mpl

eR

esea

rch

desi

gnT

radi

tiona

lT

ests

Res

ults

Lal

onde

etal

.(2

013)

N�

38ad

oles

cent

sag

es13

–17

year

s.D

escr

iptiv

e/co

rrel

atio

nal

stud

yof

aV

irtu

alC

lass

room

Stro

opta

sk.

Con

verg

ence

valid

ityst

udy.

•D

elis

-Kap

lan

Exe

cutiv

eFu

nctio

nSy

stem

(Tra

ilM

akin

g,T

ower

,T

wen

tyQ

uest

ions

,V

erba

lFl

uenc

y,C

olor

-Wor

dIn

terf

eren

ce)

•B

ehav

ior

Rat

ing

Inve

ntor

yof

Exe

cutiv

eFu

nctio

n•

Chi

ldB

ehav

ior

Che

cklis

t

•V

R-S

troo

pta

skco

rrel

ated

with

D-K

EFS

and

BR

IEF.

•Pe

rfor

man

ceon

the

VR

-St

roop

task

was

corr

elat

edw

ithpa

per–

penc

ilSt

roop

task

.•

VR

-Str

oop

mor

eac

cura

tely

refle

cted

ever

yday

beha

vior

alfu

nctio

ning

.N

olin

etal

.(2

009)

8ch

ildre

nw

ithac

quir

edbr

ain

inju

ry,

ages

8to

12ye

ars.

Rep

eate

dm

easu

res

com

pari

sons

.•

VIG

ILC

ontin

uous

Perf

orm

ance

Tes

t•

No

diff

eren

cebe

twee

nth

eV

irtu

alC

PTan

dth

etr

aditi

onal

com

pute

rize

dC

PTon

tota

lof

omis

sion

s.•

Sign

ifica

ntly

mor

eco

mm

issi

ons

and

long

erre

actio

ntim

esin

the

Vir

tual

CPT

.N

olin

etal

.(2

012)

N�

5025

spor

ts-c

oncu

ssed

and

25m

atch

edco

ntro

lad

oles

cent

s.

Com

pari

son

ofth

etr

aditi

onal

CPT

and

Vir

tual

CPT

was

coun

terb

alan

ced

acro

sspa

rtic

ipan

ts.

•V

IGIL

Con

tinuo

usPe

rfor

man

ceT

est

•V

irtu

alC

PTsh

owed

grea

ter

sens

itivi

tyto

the

subt

leef

fect

sof

spor

tsco

ncus

sion

.•

The

spor

tsco

ncus

sion

grou

pre

port

edm

ore

sym

ptom

sof

cybe

rsic

knes

sth

anth

eco

ntro

lgr

oup.

Pars

ons

etal

.(2

007)

N�

2010

boys

diag

nose

dw

ithA

DH

Dan

d10

mat

ched

cont

rols

.

Inte

rgro

upco

mpa

riso

nof

part

icip

ants

with

AD

HD

and

norm

alco

ntro

ls.

•SW

AN

Beh

avio

rC

heck

list

•C

onne

rs’

CPT

II•

Stro

op•

Tra

ilM

akin

gte

sts

•N

EPS

Y(V

isua

lat

tent

ion,

desi

gnflu

ency

,ve

rbal

fluen

cy)

•W

ISC

-III

(Dig

itSp

an,

Cod

ing,

Ari

thm

etic

,V

ocab

ular

y)•

Judg

emen

tof

Lin

eO

rien

tatio

n

•A

DH

Dgr

oup

exhi

bite

dm

ore

omis

sion

erro

rs,

com

mis

sion

erro

rs,

and

over

all

body

mov

emen

tin

the

Vir

tual

CPT

.•

AD

HD

grou

pw

asm

ore

impa

cted

bydi

stra

ctio

nn

the

Vir

tual

CPT

.•

Vir

tual

CPT

was

corr

elat

edw

ithtr

aditi

onal

AD

HD

asse

ssm

ent

tool

s,be

havi

orch

eckl

ist,

and

trad

ition

alco

mpu

teri

zed

CPT

.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Tab

le3

(con

tinu

ed)

Stud

ySa

mpl

eR

esea

rch

desi

gnT

radi

tiona

lT

ests

Res

ults

Pars

ons

and

Car

lew

(201

6)

Tw

ost

udie

sre

port

ed:

Stud

y1:

50un

derg

radu

ate

stud

ents

(mea

nag

e�

20.3

7;78

%fe

mal

e).

Stud

y1:

•W

echs

ler

Tes

tof

Adu

ltR

eadi

ng•

Vir

tual

Stro

opta

skw

asco

rrel

ated

with

trad

ition

alta

sks

and

elic

ited

anin

terf

eren

ceef

fect

sim

ilar

toth

ose

foun

din

clas

sic

Stro

opta

sks.

Stud

y2:

8st

uden

tsw

ithhi

ghfu

nctio

ning

autis

m(m

ean

age

�22

.88)

and

10m

atch

edco

ntro

ls.

Nor

mat

ive

stud

yco

mpa

ring

Vir

tual

Stro

opto

trad

ition

alta

sks.

•D

elis

-Kap

lan

Exe

cutiv

eFu

nctio

ning

Syst

em:

Col

orW

ord

Inte

rfer

ence

Tes

t•

Stro

opta

skfr

omA

utom

ated

Neu

rops

ycho

logi

cal

Ass

essm

ent

Met

rics

•D

urin

gth

edi

stra

ctio

nco

nditi

onof

the

Vir

tual

Stro

opA

SDgr

oup

perf

orm

ance

decl

ined

.

Stud

y2:

Cro

ssse

ctio

nal

desi

gn.

•W

echs

ler

Abb

revi

ated

Scal

eof

Inte

llige

nce-

Seco

ndE

ditio

nPo

llak

etal

.(2

009)

N�

37bo

ysag

es9–

17ye

ars,

with

(n�

20)

and

with

out

AD

HD

(n�

17).

Cro

ssov

erde

sign

com

pari

ngV

irtu

alC

lass

room

onre

gula

rco

mpu

ter

scre

en.

•T

est

ofV

aria

bles

ofA

ttent

ion

(TO

VA

)•

Shor

tFe

edba

ckQ

uest

ionn

aire

•A

DH

Dgr

oup

perf

orm

edle

ssw

ell

onal

lC

PTta

sks.

•V

irtu

alC

PTsh

owed

effe

ctsi

zes

sim

ilar

toth

eT

OV

A.

•Se

lf-r

epor

ted

pref

eren

cefo

rV

irtu

alC

PT.

Polla

ket

al.

(201

0)N

�27

16bo

ysan

d11

girl

s,w

ithcl

inic

aldi

agno

sis

ofA

DH

D.

Dou

ble-

blin

d,pl

aceb

o-co

ntro

lled,

cros

sove

rde

sign

.

Vir

tual

Cla

ssro

omon

regu

lar

com

pute

rsc

reen

Tes

tof

Var

iabl

esof

Atte

ntio

n(T

OV

A;

Gre

enbe

rg&

Wal

dman

,19

93)

•M

etilp

heni

date

(MPH

)re

duce

dom

issi

oner

rors

toa

grea

ter

exte

nton

the

VR

-C

PTco

mpa

red

toth

eno

VR

-CPT

and

the

TO

VA

,an

dde

crea

sed

othe

rC

PTm

easu

res

onal

lty

pes

ofC

PTto

asi

mila

rde

gree

.•

Chi

ldre

nra

ted

the

VR

-CPT

asm

ore

enjo

yabl

eco

mpa

red

toth

eot

her

type

sof

CPT

.

Not

e.A

DH

D�

Atte

ntio

n-de

ficit/

hype

ract

ivity

diso

rder

;ASD

�A

utis

mSp

ectr

umD

isor

der;

BA

SC�

Beh

avio

rA

sses

smen

tSys

tem

for

Chi

ldre

n;C

PT�

Con

tinuo

usPe

rfor

man

ceT

est;

DK

EFS

�D

elis

–Kap

lan

Exe

cutiv

eFu

nctio

nSy

stem

;B

RIE

F�

Beh

avio

rR

atin

gIn

vent

ory

ofE

xecu

tive

Func

tion;

NE

PSY

�ab

brev

iatio

nof

the

term

“neu

rops

ycho

logy

”;N

F1�

Neu

rofib

rom

atos

isty

pe1;

TO

VA

�T

est

ofV

aria

bles

ofA

ttent

ion;

VR

�V

irtu

alR

ealit

y;W

ISC

-III

�W

echs

ler

Inte

llige

nce

Scal

efo

rC

hild

ren.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

testing needs. In addition, there are a number ofauditory distractors that can be adjusted, such asthe sounds of vehicles passing by and ambientclassroom noise. Other aspects of the virtualenvironment can also be adjusted, includingseating position of the client, the number ofvirtual students, and the sex of the teacher(Rizzo et al., 2006).

The case of G. T. A psychologist receivesa referral for assessment of attentional prob-lems. The client is G. T., a 10-year-old biracialmale who is experiencing difficulty sustainingattention during class and listening to instruc-tions. The information from the parent andteacher versions of the behavior rating scalesare somewhat inconsistent. His teacher endorsesa number of attention problems for G. T. such asfrequent forgetting of instructions before com-pleting schoolwork; G. T. fidgets often; andG. T. inappropriately leaves his seat duringclass. According to his teacher, G. T. also ap-pears to be easily distracted during teacher pre-sentations of class material. However, the par-ent rating reveals only minor difficulties withattention.

As part of a larger psychological battery,G. T. was assented to complete Virtual Class-room CPT and Stroop tasks. The virtual class-room was modified to closely reflect his typ-ical classroom environment by including thesame number of students, assigned desk loca-tion, and same sex of the teacher. While im-mersed in the Virtual Classroom, G. T. sat ata desk in the middle of the virtual classroomthat corresponded with where he sits in hisreal classroom at school. During the Stroopand CPT tasks, head movement tracking sen-sors logged G. T.’s frequent head movementsas he looked away from the blackboard toobserve vehicles passing by outside and vir-tual students moving around in the room.Moreover, the movement sensors logged hisbody movement and revealed that he was veryphysically active during the tasks. He showedpoor performance (compared with norm-reference groups) on the cognitive tasks.

The results of testing indicated G. T. wasshowing several behavioral symptoms ofADHD. Review of the data showed that symp-toms of inattention, such as maintaining atten-tion, are likely because of being easily derailedby classroom and outdoor distractors. This alsoaffected his ability to follow through with in-

structions and listen to the virtual teacher. Ofparticular note are the symptoms of hyperactiv-ity that were directly recorded. He frequentlymoved his legs and arms and squirmed aroundin his seat. He had difficulty sitting still andremaining seated.

Several recommendations were made basedon these observations. G. T. performed betterwhen placed in the front of the virtual class-room, so it was recommended that this adjust-ment be made in his actual classroom. It wasrecommended that classroom windows remainclosed if possible to reduce noise from outsidesources. Additionally, it was recommended thatpeople entering and exiting the room be reducedas much as possible. Finally, the specific dis-tractors were provided to the parent to sharewith G. T.’s teacher, so when these distractorsoccur the teacher will know it is necessary toredirect G. T.’s attention.

In summary, the Virtual Classroom hasseveral advantages as opposed to traditionalpen-and-paper testing. It represents the typi-cal classroom in which students with ADHDand autism often struggle to maintain atten-tion and engagement, which increases ecolog-ical validity compared with traditional testingmethods. In addition, functional impairmentscan be directly observed, which improves thequality of recommendations. The VirtualClassroom provides a record of which distrac-tions caused the client to look away from theboard and how distracting they were (howoften the client looked away). These data caninform recommendations to parents andteachers to remove specific distractors fromthe learning environment. The impact of plac-ing the client in different locations in thevirtual classroom can also be assessed. Theresults of such testing allow for recommen-dations based on direct observation aboutwhere to seat the client in the classroom sothey can perform at their best. The virtualclassroom records the head and body move-ments of the child in real time. Thus, the levelof activity is accurately recorded without thepotential bias that may influence parent andteacher reports (Pas & Bradshaw, 2014; Sayal& Taylor, 2005). Collecting and recordingthese body movement data in an ecologicallyvalid environment rather than relying on par-ent and teacher reports may be a better way ofassessing the “H” in ADHD.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Virtual Shopping Tasks

Another virtual environment approach thatmay be of interest to clinicians is the use of asimulated shopping environment. A number ofthese virtual shopping environments have beendeveloped to assess execution of everyday be-haviors in a virtual shopping center (see Table4). These virtual shopping environments havebeen developed in a manner that allows thepsychologist to systematically vary the informa-tion load (that affects goal maintenance). Forexample, the Virtual Environment GroceryStore (VEGS) uses multiple adaptive trials inthe assessment procedure by creating a pool of“multiple task assignments,” empirically deter-mining their baseline difficulty, and then addingconditions in the environment that affect base-line task difficulty via the manipulation of thedensity of items on shelves, the similarity ofpackaging, and the intensity and types of real-istic irrelevant distractions (e.g., loudness/typeof music in the background and loudspeakerannouncements). The VEGS platform offers arange of difficulties that may be used to makethe tasks sufficiently complex to avoid floor andceiling effects. The use of a simulated environ-ment allows older adults who may be physicallyor behaviorally impaired to be safely assessed.This would not be possible with the traditionalMET, which requires these tasks to be com-pleted in a real-world shopping environment(Parsons, McPherson, & Interrante, 2013).

The case of L. S. A psychologist receives areferral for assessment of memory problems.The client is L. S., a 58-year-old married, His-panic female who sought assessment after mem-ory difficulties appear to have resulted in herfrequently misplacing items around the houseand forgetting to attend appointments. She alsocontinually asks family members to help herwith shopping. Furthermore, she reported that attimes she would get up and go to her kitchenonly to feel confused because she could notremember what she intended to do in the kitch-en. She had a diagnosis of osteoarthritis, whichdecreased movement in her right knee, and de-nied any previous history of head-injury, sub-stance abuse, or psychiatric illness. Data col-lected during the clinical interview suggestedearly onset Alzheimer’s disease, but resultsfrom the neuropsychological assessment wereinconsistent.

To assess her cognitive functioning, L. S. wasasked to complete the VEGS as part of herlarger battery of tests. While immersed in theVEGS, L.S. often had difficulty with sustainedconcentration, which was demonstrated by theindirect route she took to the pharmacy andwhen shopping for items on the shopping list.She also had difficulty remembering how muchmoney was budgeted for purchasing items andcontinually asked her psychologist what shewas supposed to do next. She also performedpoorly on various aspects of the prospectivememory tasks. For example, she had to be re-minded to go to the coupon machine after 5 minof shopping. She also had difficulty recallingthe shopping items and frequently clicked onthe shopping list for a reminder. These difficul-ties were notably amplified when distractorswere present (e.g., announcements over the loudspeaker; people shopping in the same aisle asher).

These results suggest that L.S. is demonstrat-ing behavioral symptoms of suggestive of earlyonset Alzheimer’s disease, including difficultywith concentration, problem solving, and mem-ory for recently learned information. With per-mission from L.S., her caregiver was shown avirtual replay of her activities in the VEGS. Itrevealed areas of difficulty and supported a rec-ommendation that L. S. be provided assistancewith tasks that require sustained attention suchas shopping or household chores. The VEGSreport also revealed that she needed assistancewith tasks that require problem-solving skillssuch as budgeting. It was recommended thatfamily members provide L. S. with frequentreminders of newly learned information such asdates and changes in routines on an easily ac-cessible calendar and in-person.

Virtual Reality Apartment

Virtual Apartments are simulation environ-ments that allow clinicians to see how the clientbehaves in their home environment. The pur-pose of the Virtual Apartment is to assess po-tential cognitive deficits using a simulatedapartment. While a number of virtual apartmentenvironments have emerged, there is a consis-tent emphasis upon (a) ecologically valid rep-resentations of the client’s everyday activities ina living situation; (b) presentation of either tra-ditional cognitive tasks (e.g., CPT; Stroop) or


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Tab

le4

Rec

ent

Fun

ctio

n-L

edSt

udie

s(W

ithi

nP

ast

10Y

ears

)U

sing

aV

irtu

alSt

ore

Stud

ySa

mpl

eR

esea

rch

desi

gnT

radi

tiona

lte

sts

Res

ults

Atk

ins

etal

.(2

015)

N�

44he

alth

yyo

ung

and

N�

39he

alth

yol

d.

The

two

grou

psw

ere

com

pare

don

virt

ual

and

trad

ition

alm

easu

res.

•M

AT

RIC

SC

onse

nsus

Cog

nitiv

eB

atte

ry•

UC

SDPe

rfor

man

ce-B

ased

Skill

sA

sses

smen

t-B

rief

•H

opki

nsV

erba

lL

earn

ing

Tes

t•

Bri

efV

isua

lM

emor

yT

est

•E

ach

VR

outc

ome

mea

sure

disp

laye

dsi

gnifi

cant

age-

rela

ted

diff

eren

ces.

•T

radi

tiona

lm

easu

res

ofco

gniti

vefu

nctio

ning

wer

esi

gnifi

cant

lyas

soci

ated

with

VR

perf

orm

ance

acro

ssag

egr

oups

.C

anty

etal

.(2

014)

N�

30se

vere

TB

Ivs

.N

�24

heal

thy

cont

rols

.

Seve

reT

BI

(n�

30)

vs.

heal

thy

cont

rol

(n�

24).

•L

exic

alD

ecis

ion

Pros

pect

ive

Mem

ory

Tas

k•

Hop

kins

Ver

bal

Lea

rnin

gT

est

•T

rail

Mak

ing

Tas

k•

Con

trol

led

oral

wor

das

soci

atio

n•

Hay

ling

Sent

ence

Com

plet

ion

•L

ette

rN

umbe

rSe

quen

cing

•Sy

dney

Psyc

hoso

cial

Rei

nteg

ratio

nSc

ale

•V

irtu

alsh

oppi

ngpe

rfor

man

cedi

ffer

entia

ted

betw

een

TB

Ipa

tient

san

dth

eco

ntro

lgr

oup.

•M

easu

res

ofpr

ospe

ctiv

em

emor

y,ne

uroc

ogni

tive

func

tioni

ng,

and

psyc

hoso

cial

func

tioni

ngw

ere

sign

ifica

ntly

asso

ciat

edw

ithV

irtu

alsh

oppi

ngpe

rfor

man

ceam

ong

TB

Ipa

tient

s.C

arel

li,M

orga

nti,

Wei

ss,

Kiz

ony,

&R

iva

(200

8)

N�

24he

alth

yad

ults

.D

escr

iptiv

est

udy.

n/a

•V

irtu

alsu

perm

arke

tm

aybe

aus

eful

tool

inex

ecut

ive

asse

ssm

ent,

part

icul

arly

due

toits

tem

pora

lan

dac

cura

cym

easu

res.

Ere

z,W

eiss

,K

izon

y,&

Ran

d(2

013)

N�

20ch

ildre

nw

ithT

BI

vs.

N�

20he

alth

yco

ntro

ls.

Com

pari

son

ofT

BI

(n�

20)

tohe

alth

yco

ntro

ls(n

�20

)in

avi

rtua

lm

all.

n/a

•O

utco

me

mea

sure

sof

the

VM

all

succ

essf

ully

diff

eren

tiate

dbe

twee

nch

ildre

nw

ithT

BI

and

heal

thy

cont

rols

.Jo

sman

etal

.(2

006)

N�

20st

roke

patie

nts.

Com

pari

son

ofV

irtu

alSu

perm

arke

tto

trad

ition

alte

sts.

Beh

avio

ral

Ass

essm

ent

ofD

ysex

ecut

ive

Synd

rom

e(B

AD

S)•

Vir

tual

shop

ping

outc

ome

mea

sure

s(#

item

spu

rcha

sed,

#co

rrec

tac

tions

,du

ratio

npa

uses

/sto

ps)

wer

esi

gnifi

cant

lyas

soci

ated

with

the

key

sear

chsu

btes

tof

the

BA

DS.

Josm

anet

al.

(201

4)N

�24

stro

kean

d24

mat

ched

cont

rols

.C

ompa

riso

nof

stro

ke(n

�24

)an

dco

ntro

l(n

�24

)pa

rtic

ipan

ts.

Beh

avio

ral

Ass

essm

ent

ofD

ysex

ecut

ive

Synd

rom

e•

Res

ults

reve

aled

sign

ifica

ntdi

ffer

ence

sin

exec

utiv

efu

nctio

ning

betw

een

post

-str

oke

patie

nts

and

the

cont

rol

grou

pon

virt

ual

shop

ping

outc

ome

mea

sure

s.•

Vir

tual

shop

ping

outc

ome

mea

sure

sw

ere

also

sign

ifica

ntly

asso

ciat

edw

ithth

eB

AD

S.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Tab

le4

(con

tinu

ed)

Stud

ySa

mpl

eR

esea

rch

desi

gnT

radi

tiona

lte

sts

Res

ults

Josm

an,

Sche

nird

erm

an,

Klin

ger,

and

Shev

il(2

009)

N�

30sc

hizo

phre

nia

and

N�

30he

alth

yco

ntro

ls.

Com

pari

son

ofsc

hizo

phre

nia

(n�

30)

tohe

alth

yco

ntro

l(n

�30

)pe

rfor

man

ce.

Beh

avio

ral

Ass

essm

ent

ofD

ysex

ecut

ive

Synd

rom

e•

Vir

tual

shop

ping

outc

ome

mea

sure

sw

ere

sens

itive

todi

ffer

ence

sin

exec

utiv

efu

nctio

ning

betw

een

schi

zoph

reni

apa

tient

san

dco

ntro

lsan

ddi

ffer

entia

ted

betw

een

patie

nts

with

diff

erin

gle

vels

ofex

ecut

ive

func

tion.

•V

irtu

alsh

oppi

ngou

tcom

em

easu

res

wer

ene

gativ

ely

rela

ted

tosy

mpt

oms

ofsc

hizo

phre

nia

and

the

BA

DS.

Kan

get

al.

(200

8)N

�20

stro

kean

dN

�20

mat

ched

cont

rols

.C

ompa

riso

nof

Stro

ke(n

�20

)an

dco

ntro

l(n

�20

)pe

rfor

man

ce.

n/a

•V

irtu

alsh

oppi

ngou

tcom

em

easu

res

cons

iste

ntly

diff

eren

tiate

dbe

twee

nst

roke

patie

nts

(uni

late

ral

brai

nle

sion

)an

dth

eco

ntro

lgr

oup.

Klin

ger,

Che

min

,L

ebre

ton,

and

Mar

ié(2

006)

N�

5Pa

rkin

son’

san

dN

�5

age-

mat

ched

heal

thy

cont

rols

.

Park

inso

n’s

(n�

5)to

age-

mat

ched

heal

thy

cont

rol

(n�

5)pe

rfor

man

ce.

n/a

•V

irtu

alsh

oppi

ngou

tcom

em

easu

res

reve

aled

that

patie

nts

wal

ked

asi

gnifi

cant

lylo

nger

dist

ance

.•

Thi

ssu

gges

tsth

atth

eV

irtu

alSu

perm

arke

tm

aybe

aus

eful

tool

for

asse

ssin

gco

gniti

vepl

anni

ngin

patie

nts

with

Park

inso

n’s.

Oka

hash

iet

al.

(201

3)N

�10

brai

nda

mag

ean

dN

�10

age-

mat

ched

cont

rols

and

N�

10he

alth

y-ol

der

adul

tsco

mpa

red

toN

�10

heal

thy-

youn

g.

Com

pari

son

ofbr

ain

dam

age

(n�

10)

perf

orm

ance

toag

e-m

atch

edco

ntro

l(n

�10

);H

ealth

y-ol

d(n

�10

)an

dhe

alth

y-yo

ung

(n�

10).

Beh

avio

ral

Ass

essm

ent

ofD

ysex

ecut

ive

Synd

rom

e•

Perf

orm

ance

onth

evi

rtua

lsh

oppi

ngta

skw

assi

gnifi

cant

lyas

soci

ated

with

conv

entio

nal

cogn

itive

asse

ssm

ents

.•

Old

erpa

rtic

ipan

tsan

dpa

tient

sw

ithbr

ain

dam

age

scor

edsi

gnifi

cant

lyw

orse

onth

evi

rtua

lsh

oppi

ngta

sks.

Ran

d,B

asha

-Abu

Ruk

an,

Wei

ss,

and

Kat

z(2

009)

N�

9po

st-s

trok

e(n

�9)

and

N�

20he

alth

y-yo

ung

and

N�

20he

alth

yag

ing

Com

pari

son

ofpo

st-s

trok

e(n

�9)

vs.

heal

thy-

youn

g(n

�20

)vs

.he

alth

y-ol

d(n

�20

).

Beh

avio

ral

Ass

essm

ent

ofD

ysex

ecut

ive

Synd

rom

e(Z

ooM

apsu

btes

t)•

Vir

tual

shop

ping

task

ssu

cces

sful

lydi

ffer

entia

ted

betw

een

all

thre

egr

oups

.•

Vir

tual

shop

ping

outc

ome

mea

sure

sw

ere

sign

ifica

ntly

asso

ciat

edw

ithZ

ooM

apsu

btes

t.R

and,

Kat

z,an

dW

eiss

(200

7)N

�14

stro

kean

dN

�23

heal

thy

cont

rol

child

ren

and

N�

44yo

ung

adul

ts;

and

N�

26ol

der

adul

ts.

Com

pari

son

ofst

roke

(n�

14)

vs.

heal

thy

cont

rol

(chi

ldre

nn

�23

;yo

ung

adul

tsn

�44

;ol

der

adul

tsn

�26

).

n/a

•V

irtu

alsh

oppi

ngtim

esu

cces

sful

lydi

ffer

entia

ted

betw

een

post

-str

oke

patie

nts

and

the

cont

rol

grou

psw

ithpo

st-s

trok

epa

tient

sta

king

sign

ifica

ntly

long

erto

com

plet

eth

esh

oppi

ngta

sk.

(tab

leco

ntin

ues)


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Tab

le4

(con

tinu

ed)

Stud

ySa

mpl

eR

esea

rch

desi

gnT

radi

tiona

lte

sts

Res

ults

Ras

pelli

etal

.(2

012)

N�

9st

roke

and

N�

10he

alth

y-yo

ung;

and

N-

10he

alth

yag

ing.

Com

pari

son

ofst

roke

(n�

9)vs

.he

alth

y-yo

ung

(n�

10)

vs.

heal

thy-

old.

•T

est

ofE

very

day

Atte

ntio

n•

Iow

aG

ambl

ing

Tas

k•

Stro

opT

est

•Si

gnifi

cant

diff

eren

ces

wer

efo

und

betw

een

all

thre

egr

oups

ontw

oou

tcom

em

easu

res

ofth

evi

rtua

lsh

oppi

ngta

sk(i

.e.,

time,

erro

rs).

•T

heT

EA

(but

not

the

IGT

orSt

roop

)w

assi

gnifi

cant

lyas

soci

ated

with

virt

ual

shop

ping

perf

orm

ance

inpo

st-s

trok

epa

rtic

ipan

ts.

Wer

ner,

Rab

inow

itz,

Klin

ger,

Kor

czyn

,an

dJo

sman

(200

9)

N�

10m

ildco

gniti

veim

pair

men

tan

dN

�30

mat

ched

cont

rols

.

Com

pari

son

ofM

CI

(n�

30)

vs.

cont

rol

(n�

40).

Beh

avio

ral

Ass

essm

ent

ofD

ysex

ecut

ive

Synd

rom

e•

4of

the

8vi

rtua

lsh

oppi

ngou

tcom

em

easu

res

wer

eas

soci

ated

with

perf

orm

ance

onth

eB

AD

S.•

Vir

tual

shop

ping

perf

orm

ance

succ

essf

ully

diff

eren

tiate

dbe

twee

nM

CI

patie

nts

and

the

cont

rol

grou

p.Z

ygou

ris

etal

.(2

015)

N�

34m

ildco

gniti

veim

pair

men

tan

dN

-20

mat

ched

cont

rols

.

Com

pari

son

ofM

CI

(n�

34)

vs.

cont

rol

(n�

21).

•M

ini

Men

tal

•R

ey-O

ster

riet

hC

ompl

exFi

gure

Tes

t•

Rey

Aud

itory

Ver

bal

Lea

rnin

gT

est

•R

iver

mea

dB

ehav

iour

alM

emor

yT

est

•T

est

ofE

very

day

atte

ntio

n•

Tra

ilM

akin

gte

st•

Func

tiona

lR

atin

gSc

ale

for

Sym

ptom

sof

Dem

entia

•Fu

nctio

nal

Cog

nitiv

eA

sses

smen

tSc

ale

•V

irtu

alsh

oppi

ngpe

rfor

man

cew

asm

oder

atel

yco

rrel

ated

with

trad

ition

alne

urop

sych

olog

ical

test

s.•

Vir

tual

shop

ping

perf

orm

ance

was

able

todi

ffer

entia

tebe

twee

nM

CI

patie

nts

and

the

cont

rol

grou

p;ho

wev

er,

itw

asun

able

todi

ffer

entia

teM

CI

subt

ypes

.

Not

e.B

AD

S�

Beh

avio

ralA

sses

smen

tof

Dys

exec

utiv

eSy

ndro

me;

MC

I�

Mild

Cog

nitiv

eIm

pair

men

t;T

EA

�T

esto

fE

very

day

Atte

ntio

n;IG

T�

Iow

aG

ambl

ing

Tas

k;T

BI

�T

raum

atic

Bra

inIn

jury

;U

CSD

�U

nive

rsity

ofC

alif

orni

a,Sa

nD

iego

;V

R�

Vir

tual

Rea

lity;

VM

all

�V

irtu

alM

all.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

everyday activities (without embedded cogni-tive stimuli) in the environment; (c) low andhigh distraction conditions; and (d) logging ofbehavioral metrics (e.g., ambient body move-ment, head/eye gaze).

The ClinicaVR team has extended the virtualclassroom paradigm to a virtual apartment thatsuperimposes construct-driven stimuli (e.g.,Stroop and CPT) onto a large TV. The VirtualApartment Bimodal Stroop (VABS) is a 9.6-mintask. Participants are seated in the living room, infront of a flat-screen TV. A kitchen is located tothe left of the TV and a window is located to theright of the TV. The task builds on the unimodal(visually mediated) Stroop and measures cogni-tive interference using reaction time (RT), com-mission errors and omission errors, and RT vari-ability. The VABS extends the traditional Stroopparadigm via the inclusion of bimodal (auditoryand visually mediated) stimuli.

During the task, distracters appear in differentfield of view locations in the environment. Somedistracters are audio—visual: School Bus passingon the street and SUV viewed through window onthe right; iPhone ringing and vibrating on the table(in front of participant); Toy Robot moving andmaking noise on the floor (center). Auditory dis-tractors included: Crumple Paper (left); Drop Pen-cil (left); Doorbell (left); Clock (left) VacuumCleaner (right); Jack Hammer (right); Sneeze(left) Jet Noise (center). Visual distractors in-cluded: Paper airplane (flying from left to right infront of the participant), Woman walking in thekitchen (center). That condition was designed toassess RTs (simple and complex), selective atten-tion (matching the auditory and visual stimuli),and external interference control (environmentaldistracters).

In a preliminary study, Henry and colleagues(2012) with 71 healthy adult participants foundthat the VR-Apartment Stroop is capable of elic-iting the Stroop effect with bimodal stimuli. Initialvalidation data also suggested that measures of theVR-Stroop significantly correlate with measuresof the Elevator counting with distracters (rangingfrom .38 to .62), the Continuous PerformanceTask (CPT-II; ranging from .32 to .42), and theStop-it task (ranging from .37 to .39). Resultsfrom regression indicated that commission errorsand variability of RTs at the VR-ApartmentStroop were significantly predicted by scores ofthe Elevator task and the CPT-II. These prelimi-nary results suggest that the VR-Apartment Stroop

is an interesting measure of cognitive and motorinhibition for adults.

Virtual reality apartment medicationmanagement assessment. A different Vir-tual Apartment has been developed by Kurtz,Baker, Pearlson, and Astur (2007) to assessesmedication management skills among thosewith schizophrenia. The Virtual RealityApartment Medication Management Assess-ment (VRAMMA) lasts a maximum of 23min. The environment consists of a four-bedroom apartment containing a living room,bedroom, kitchen, and bathroom. During thepractice phase, clients are asked to do a numberof tasks that help build familiarity with thevirtual environment: using a joystick to navi-gate to the living room and turn on the TV,checking the time on the interactive clock ontop of the TV, turning on a light in the bedroom,checking the medication reminder post-it notein the kitchen (with a different prescription thanthe actual trial) after turning on the light andturning off the stove, and opening the medicinecabinet in the bathroom and taking out pills.During the testing phase clients start in theliving room and a message is displayed thatgives them the medications to take and the timethey must be taken. They must then use all theitems from the practice phase to take the correctmedication at the correct time (in 15 min). Sev-eral auditory distractors that simulate real-lifedistractions were included in the virtual apart-ment: a phone ringing, a doorbell ringing, a dogbarking, and a police siren. These auditory dis-tractors are introduced every 3 min. Significantevents, such as turning on lights or checking theclock, and the location and movement of theclient are recorded. The variables recorded dur-ing testing include: quantitative errors, qualita-tive errors, time discrepancy, total distance trav-eled, clock checks, and reminder note checks.The VRAMMA is able to distinguish betweenhealthy controls and those with schizophrenia,who perform worse. Kurtz et al. (2007) con-ducted a validation study with 25 schizophreniapatients and 18 matched healthy controls. Theyfound patients with schizophrenia made morequantitative errors concerning the number ofpills taken (p � .001), were less likely to takepills at the correct time (p � .01), and checkedthe clock less often (p � .001).

The VRAMMA has several advantages com-pared with traditional testing. The VRAMMA


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

shows good convergent validity with the Medica-tion Management Ability Assessment (Pattersonet al., 2002), a role-play task involving medicationmanagement that typically would be administeredin an office setting. Clinicians are able to directlyobserve client behavior in a more ecologicallyvalid environment rather than relying of self-report measures, which may be biased by a lack ofinsight from clients with schizophrenia (seeMintz, Dobson, & Romney, 2003 for a meta-analysis) or cognitive and emotional disruptions(Anticevic, Schleifer, & Youngsun, 2015). TheVRAMMA approximates an environment inwhich many clients would likely need to managemedication and includes common tools needed tocarry out this activity, such as a reminder note anda clock. It also allows clinicians to assess when theprocess of performing this task is disrupted andmake specific recommendations from these obser-vations.

The case of J. A. A psychologist receives areferral for assessment of concentration, and aresurgence of visual and auditory hallucina-tions. The client is J. A., a 23-year-old, singleWhite male who was diagnosed with schizo-phrenia in 2013 after hospitalization for a singleepisode of psychosis. He was released from thepsychiatric hospital after 1 month and given aprescription for antipsychotic medication. J. A.continued meeting regularly with a psychiatrist.Several months after being released from thepsychiatric hospital, J. A. began experiencingdifficulty with concentration, and a resurgenceof visual and auditory hallucinations. To ascer-tain whether medication adherence was at issue,J. A. was asked to complete the VRAMMA. Hetook fewer than the required number of pills andtook the pills 3 min after the 15-min time lapsehad expired. While in the virtual apartment,J. A. traveled around the apartment longer thannecessary to complete the task and was dis-tracted by the sound of the barking dog.

Based on these results, it appears that J. A. ishaving difficulty taking the correct number ofpills and using cues in the environment to takepills at the correct time. Positive symptoms arerelated to distance traveled in VRAMMA(Kurtz et al., 2007). J. A. traveled a significantdistance in the virtual apartment, which showshis medication adherence is low. It was recom-mended that J. A. use an automated pill dis-penser that holds the correct number of pills foreach day of the week, reminds J. A. to take

medication if not taken at a scheduled time, andsends a notification to his psychiatrist if he stopstaking medication. J. A. was advised to reduceauditory distractions at home by turning on afan to reduce noise from outside sources.

Considerations in the Adoption of VirtualReality Technologies

As can be seen there are some specific casesin which virtual reality-based psychological as-sessments may offer the psychologist with eco-logically valid assessments of day-to-day activ-ities. That said, there are a number ofconsiderations that go into the decision to addnew technologies to one’s battery of tests (seeStandards for Educational and PsychologicalTesting (American Educational Research Asso-ciation, American Psychological Association, &National Council on Measurement in Educa-tion, 2014). While some of these issues areconcerns related to the current generation ofvirtual reality based assessments, others reflectoutdated concerns from an earlier generation ofplatforms., for example, it used to be the casethat the equipment needed to conduct such as-sessments was bulky and expensive. Recent ad-vances in virtual reality technology have madethe use of simulations in assessment more fea-sible and affordable. Smaller, easier-to-useequipment and reduced cost make virtual envi-ronment assessment a practical tool psycholo-gists can use to gather precise functional dataand to provide customized recommendations toclients (Bohil et al., 2011). Furthermore, vali-dated virtual environments with automatedstimulus presentation, data capture, and scoringare emerging that include sample characteristicsfor norm-referenced assessment. However,there are other concerns that continue to thisday. The dearth of established guidelines for thedevelopment, administration and interpretationof these assessments could lead to importantpsychometric pitfalls. At minimum, all virtualreality-based psychological assessments musthave (and many now do) standardized instruc-tions for administration and methods for scoringand interpreting test results provided in a testmanual. While some virtual environments arebeing designed for limit testing, more workneeds to be done in this area.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Need for VE-Based NeuropsychologicalAssessments to Be Sufficiently Standardized

While the use of virtual environments forassessment is an emerging area of application,adoption will require substantial research anddevelopment to establish acceptable psycho-metric properties and clinical utility. Although areview of VR therapies has revealed statisticallylarge effects on a number of affective domains(Parsons & Rizzo, 2008), findings must be in-terpreted with caution given that some VR stud-ies do not include control groups, and many arenot randomized clinical trials, limiting the con-fidence that the enhancements were caused bythe VR intervention. An important resolution toclinical heterogeneity of outcome measures invirtual environment research is the standardiza-tion of outcomes and the measures used to as-sess these outcomes. The selection of outcomemeasures for standardization need to be relevantto the client’s treatment and health status as wellas psychometrically sound. Another pressingneed among psychologists is the identificationof VE-based assessments that reflect relevantunderlying cognitive and behavioral capacitiesfor assessments of varying degrees of psycho-logical deficits. VE-based assessments mustdemonstrate relevance beyond that which isavailable through simpler means of assessment.As such, there is specific need for VE-basedassessments to be sufficiently standardizedwithin the range and nature of responses avail-able to participants within the virtual environ-ment to allow for reliable measurement.

VE-based assessment studies have oftensought to establish construct validity by dem-onstrating significant associations between vir-tual environments and paper-and-pencil assess-ments (e.g., virtual classroom assessments). Inthe area of function led assessment, multiplecognitive domains may be involved in the sim-ulation of real-world tasks, and associationswith traditional construct driven tests may benecessarily lower than is typically desired toestablish construct validity. In this context, thedegree to which a VE-based model using afunction led approach accurately predicts rele-vant real-world behavior may be more impor-tant than large-magnitude associations with tra-ditional construct driven paper-and-pencil tests(e.g., virtual shopping tasks).

Issues for Use of Virtual Environments inSpecific Patient Populations

In addition to psychometric and technical is-sues, clinicians, researchers, and policymakerswill need to scrutinize emerging VE-based as-sessments to ensure adherence to legal, ethical,and human safety guidelines. The matching ofspecific technologies to the needs and capacitiesof the client will also require careful consider-ation by psychologists. How will virtual envi-ronments be experienced by certain clinicalpopulations? In pediatrics and geriatrics, humanguidance is critical for safeguarding the client’sfull comprehension of assessment use and in-struction. Geriatric clients in particular may findadjusting to virtual platforms, on the whole,difficult (Miller et al., 2014).

Although virtual environments have beensuccessfully applied to the study of age differ-ences in spatial navigation among both healthyand demented elderly, virtual environment-based tasks may be complicated by visual, au-ditory, or motor impairment. In comparisonwith younger controls, aging patients may per-form more poorly on virtual environment-basedtasks simply because of the normative agingprocess or because of lack of experience withcomputers. Maximum effort should be exertedto ensure equitability in sensorimotor capacitiesbetween younger and older adult subjects. Asystematic review by Miller et al. (2014) intro-duced concern regarding the feasibility ofhome-use VE and gaming systems for physicalrehabilitation of older adults. Such systemscould be therapeutic to existing physical impair-ment or could be preventative. A main limita-tion is the low quality of studies investigatingthe effectiveness of these systems in older adultpopulations. Furthermore, some studies citedheightened fall risk, overexertion, and muscu-loskeletal irritation. There is need for more rig-orous research methods including more consis-tent and strenuous reporting of exercise dosagesand adherence. Moffatt (2009) suggests a num-ber of helpful methodological practices in as-sessing older adults in research studies of nav-igation skills, including: (a) allowing agingpatients to practice and ensure maximum famil-iarization with the computer platform, (b) in-cluding measures of computer experience, vi-sual ability, and motor function, and (c)including assessments requiring the same sen-


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

sorimotor capacities, but not physical naviga-tion.

A potential barrier to adoption of virtual re-ality technology among clinicians is concernsabout the ability of older adults to use thistechnology. Wandke, Sengpiel, and Sönksen(2012) have outlined several pervasive mythsabout older adults and human-computer inter-action. They state that the myth that older adultsare not interested in computers is not true. Dyckand Smither (1994) conducted a survey ofadults over age 55 and found these older adultswere less computer anxious and had more pos-itive attitudes about computers than adults un-der 30. Wandke et al. (2012) also address themyth that “you can’t teach an old dog newtricks.” While there are decreases in brain plas-ticity in older age, this does not mean thatlearning ceases. It may be the case that someolder adults have had frustrating experiencesthat lead to giving up on learning how to usenew technologies. However, this effect shouldnot be overgeneralized, as older adults are ofteninterested in using newer technologies (Sayago,Sloan, & Blat, 2011).

When assessing older adults for memory per-formance, it is important to avoid invoking stereo-type threat. Chasteen, Bhattacharyya, Horhota,Tam, and Hasher (2005) found that invoking ste-reotype threat about memory abilities in olderadults harms performance on memory tasks, par-ticularly when these adults are aware their mem-ory is being assessed. Subtle and unambiguousage-related stereotypes have also been found toinfluence older adults’ performance on a numberof cognitive and physical tasks (see Lamont,Swift, & Abrams, 2015 for a meta-analysis), suchas and map learning (Meneghetti, Muffato, Suit-ner, De Beni, & Borella, 2015) driving a car(Lambert et al., 2016), and hand grip strength(Swift, Lamont, & Abrams, 2012). It is possiblethat assessment in a virtual environment such asthe VEGS could reduce stereotype threat by ob-scuring the true purpose of the task.

Simulation technology may also be problematicfor individuals with autism spectrum disorder.Given pronounced sensory issues commonlyfound in this population, the head mounted dis-play or even the graphical interface may be expe-rienced as intolerable. Moreover, there is con-cerned that too intense a stimulus presentationmay aggravate sensory processing difficulties.This is an important concern though there is no

evidence from two different studies with studentsdiagnosed with autism that they experience nega-tive effects over and above those experienced bystudents without autism (Parsons & Carlew, 2016;Wallace et al., 2010). However, while these twostudies tend to suggest that negative effects wereself-reported as low, they involved screen-basedvirtual environments. As we adopt newer andmore immersive technologies (i.e., HMDs) it isimportant to consider the potential negative effects(i.e., dizziness, sickness, and displacement) to en-sure that wearable technologies (e.g., HMDs) canprovide an acceptable space for children to use;especially children with disabilities. With thissaid, there is some evidence that suggests childrendo not experience HMDs any more negativelythan screen-based media (Peli, 1998). Althoughmore work is needed in this area, these findingssupport the potential of VR technology for con-tinued greater approximations (Bohil et al., 2011)of cognitive processes in the real world.

Furthermore, individuals with severe psychi-atric conditions that cause limited self-aware-ness, high suggestibility, and/or an altered senseof reality (e.g., hallucinations, delusions) mayrespond undesirably to immersion in a virtualenvironment. There is also the potential of un-intended negative effects of exposure to virtualenvironments—stimulus intensity, if taken toofar, may exacerbate rather than ameliorate adeficit. High-fidelity virtual environments maybe confusing for these individuals and increasenegative behaviors after exposure to the envi-ronment. Flat-screen presentation of virtual en-vironments has proven to be an acceptable al-ternative to full immersion with theenvironment, and may be more appropriate forcertain clinical groups (Attree et al., 1996).

Summary and Conclusions

There are many different types of tests avail-able to psychologists for determining a client’slevel of functioning. The challenge for psychol-ogists is choosing tests that provide accurateinformation for making prescriptive statementsto clients, parents, and teachers based on thebest evidence available. Self-reports can behelpful in collecting data on specific areas offunctioning, but also suffer from lack of agree-ment among informants (Biederman et al.,1993, 1990; Mitsis et al., 2000; Newcorn et al.,1994; de Nijs et al., 2004) potential bias from


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

clients (Schwarz, 1999), and provides post hocappraisal of behavior. Normative comparison ofperformances on cognitive assessments allowsthe psychologist to determine if a client’s per-formance is similar or divergent from peers, butprovides limited information about daily func-tioning because they are construct-driven ratherthan function-led.

The addition of virtual reality to a psycho-logical battery provides an opportunity for psy-chologists to obtain more ecologically validdata about client functioning in simulations ofreal-world environments. Virtual environmentsallow the psychologist to have greater control ofdynamic perceptual stimuli and the sensitivityof the test, while also capturing data about clientperformance in activities of daily living (Bohilet al., 2011). The computerized nature of thesetests allows for the accurate capture of neurobe-havioral data, as well as precise recording andscoring of neuropsychological test results(Campbell et al., 2009). Several virtual environ-ments have already been developed to for psy-chologists to use in neuropsychological assess-ment, such as the virtual classroom, virtualgrocery store, and the virtual apartment. Al-though more validation studies need to be con-ducted with virtual reality assessments, the ben-efits of using this technology for understandingdaily functioning are clear. In addition, smallerand more affordable equipment makes virtualreality a viable option for use in psychologicalassessment.

Preferably, virtual assessments will be addedto flexible assessment batteries tailored to eachindividual within the context of the presentingquestion. Thus, traditional construct drivenmeasures should not be abandoned. In somecircumstances, construct driven assessmentsmay be more appropriate in terms of assessing aspecific construct that is generalizable acrossenvironments. For instance, working memorymay be more easily assessed by a simple spantask. The allure of the virtual assessment liesprimarily in enriching stimulus presentation,logging additional variables, and databasebuilding rather than the automation of the entirepsychological battery and the minimization ofhuman interaction.

Virtual environments may add to an existingpsychological battery when the psychologist isattempting to make accurate predictions about aperson’s behavior within the real world. In a

virtual environment, the psychologist can mea-sure functional output of constructs within thecomplexity of a real-world environment. Forexample, in a virtual classroom, selective atten-tion can be measured by conducting tests suchas the CPT in a real world environment. In avirtual environment grocery store, prospectivememory may be assessed using a real-worldtask like remembering to pick up a prescriptionat the pharmacy. Cognitive interference can beassessed in a virtual apartment that includescommon distractors found in an everyday envi-ronment.

Technological innovations, such as virtual re-ality, allow psychologists to expand our meth-ods for designing and implementing assess-ments capable of collecting information thatprovides an accurate picture of client limita-tions. These advances improve the prescriptivestatements psychologists dispense by providingthe opportunity to observe client functioning inreal-world environments—a practice that mightotherwise be infeasible because of clients’ be-havioral and physiological impairments. Byadopting virtual reality as a method for assess-ing clients, psychologists increase the potentialpositive impact of neuropsychological assess-ment for improving the daily functioning ofclients through accurate understanding of neu-ropsychological deficits and directly relevantrecommendations.

References

Adams, R., Finn, P., Moes, E., Flannery, K., & Rizzo,A. S. (2009). Distractibility in Attention/Deficit/Hyperactivity Disorder (ADHD): The virtual real-ity classroom. Child Neuropsychology, 15, 120–135. http://dx.doi.org/10.1080/09297040802169077

American Educational Research Association, Amer-ican Psychological Association, & National Coun-cil on Measurement in Education. (2014). Stan-dards for educational and psychological testing.Washington, DC: American Educational ResearchAssociation.

Anticevic, A., Schleifer, C., & Youngsun, T. C.(2015). Emotional and cognitive dysregulation inschizophrenia and depression: Understandingcommon and distinct behavioral and neural mech-anisms. Dialogues in Clinical Neuroscience, 17,421–434.

Atkins, A. S., Stroescu, I., Spagnola, N. B., Davis,V. G., Patterson, T. D., Narasimhan, M., . . . Keefe,


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

http://dx.doi.org/10.1080/09297040802169077

http://dx.doi.org/10.1080/09297040802169077

R. S. E. (2015). Assessment of age-related differ-ences in functional capacity using the Virtual RealityFunctional Capacity Assessment Tool (VRFCAT).The Journal of Prevention of Alzheimer’s Disease,2, 121.

Attree, E. A., Brooks, B. M., Rose, F. D., Andrews,T. K., Leadbetter, A. G., & Clifford, B. R. (1996,July). Memory processes and virtual environ-ments: I can’t remember what was there, but I canremember how I got there. Implications for peoplewith disabilities. In ECDVRAT: 1st EuropeanConference on Disability, Virtual Reality and As-sociated Technologies, Reading, UK (Vol. 118).

Biederman, J., Faraone, S. V., Milberger, S., &Doyle, A. (1993). Diagnoses of attention-deficithyperactivity disorder from parent reports predictdiagnoses based on teacher reports. Journal of theAmerican Academy of Child & Adolescent Psychi-atry, 32, 315–317. http://dx.doi.org/10.1097/00004583-199303000-00011

Biederman, J., Keenan, K., & Faraone, S. V. (1990).Parent-based diagnosis of attention deficit disorderpredicts a diagnosis based on teacher report. Jour-nal of the American Academy of Child & Adoles-cent Psychiatry, 29, 698–701. http://dx.doi.org/10.1097/00004583-199009000-00004

Bioulac, S., Lallemand, S., Rizzo, A., Philip, P.,Fabrigoule, C., & Bouvard, M. P. (2012). Impactof time on task on ADHD patient’s performancesin a virtual classroom. European Journal of Pae-diatric Neurology, 16, 514–521.

Bohil, C. J., Alicea, B., & Biocca, F. A. (2011).Virtual reality in neuroscience research and ther-apy. Nature Reviews Neuroscience, 12, 752–762.

Burgess, P. W., Alderman, N., Forbes, C., Costello,A., Coates, L. M. A., Dawson, D. R., . . . Channon,S. (2006). The case for the development and use of“ecologically valid” measures of executive func-tion in experimental and clinical neuropsychology.Journal of the International NeuropsychologicalSociety, 12, 194–209. http://dx.doi.org/10.1017/S1355617706060310

Campbell, Z., Zakzanis, K. K., Jovanovski, D.,Joordens, S., Mraz, R., & Graham, S. J. (2009).Utilizing virtual reality to improve the ecologicalvalidity of clinical neuropsychology: An FMRIcase study elucidating the neural basis of planningby comparing the Tower of London with a three-dimensional navigation task. Applied Neuropsy-chology, 16, 295–306. http://dx.doi.org/10.1080/09084280903297891

Canty, A. L., Fleming, J., Patterson, F., Green, H. J.,Man, D., & Shum, D. H. K. (2014). Evaluation ofa virtual reality prospective memory task for usewith individuals with severe traumatic brain in-jury. Neuropsychological Rehabilitation, 24, 238–265. http://dx.doi.org/10.1080/09602011.2014.881746

Carelli, L., Morganti, F., Weiss, P. L., Kizony, R., &Riva, G. (2008). A virtual reality paradigm for theassessment and rehabilitation of executive functiondeficits post stroke: Feasibility study. 2008 VirtualRehabilitation. http://dx.doi.org/10.1109/icvr.2008.4625144

Chasteen, A. L., Bhattacharyya, S., Horhota, M.,Tam, R., & Hasher, L. (2005). How feelings ofstereotype threat influence older adults’ memoryperformance. Experimental Aging Research, 31,235–260. http://dx.doi.org/10.1080/03610730590948177

Chaytor, N., & Schmitter-Edgecombe, M. (2003).The ecological validity of neuropsychologicaltests: A review of the literature on everyday cog-nitive skills. Neuropsychology Review, 13, 181–197. http://dx.doi.org/10.1023/B:NERV.0000009483.91468.fb

de Nijs, P. F., Ferdinand, R. F., de Bruin, E. I.,Dekker, M. C. J., van Duijn, C. M., & Verhulst,D. C. (2004). Attention-deficit/hyperactivity disor-der (ADHD): Parents’ judgment about school,teachers’ judgment about home. European Child& Adolescent Psychiatry, 13, 315–320. http://dx.doi.org/10.1007/s00787-004-0405-z

Díaz-Orueta, U., Garcia-López, C., Crespo-Eguílaz,N., Sánchez-Carpintero, R., Climent, G., & Nar-bona, J. (2014). AULA virtual reality test as anattention measure: Convergent validity with Con-ners’ Continuous Performance Test. Child Neuro-psychology: A Journal on Normal and AbnormalDevelopment in Childhood & Adolescence, 20,328–342.

Dyck, J. L., & Smither, J. A. (1994). Age differencesin computer anxiety: The role of computer expe-rience, gender and education. Journal of Educa-tional Computing Research, 10, 239–248. http://dx.doi.org/10.2190/E79U-VCRC-EL4E-HRYV

Erez, N., Weiss, P. L., Kizony, R., & Rand, D.(2013). Comparing performance within a virtualsupermarket of children with traumatic brain in-jury to typically developing children: A pilotstudy. OTJR: occupation, participation andhealth, 33, 218–227.

Gilboa, Y., Kerrouche, B., Longaud-Vales, A., Kief-fer, V., Tiberghien, A., Aligon, D., . . . PauleChevignard, M. (2015). Describing the attentionprofile of children and adolescents with acquiredbrain injury using the Virtual Classroom. [Ad-vance online publication]. Brain Injury, 29, 1691–1700. http://dx.doi.org/10.3109/02699052.2015.1075148

Gilboa, Y., Rosenblum, S., Fattal-Valevski, A., Tole-dano-Alhadef, H., Rizzo, A. S., & Josman, N.(2011). Using a Virtual Classroom environment todescribe the attention deficits profile of childrenwith Neurofibromatosis type 1. [Advance onlinepublication]. Research in Developmental Disabil-


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

http://dx.doi.org/10.1097/00004583-199303000-00011

http://dx.doi.org/10.1097/00004583-199303000-00011

http://dx.doi.org/10.1097/00004583-199009000-00004

http://dx.doi.org/10.1097/00004583-199009000-00004

http://dx.doi.org/10.1017/S1355617706060310

http://dx.doi.org/10.1017/S1355617706060310

http://dx.doi.org/10.1080/09084280903297891

http://dx.doi.org/10.1080/09084280903297891

http://dx.doi.org/10.1080/09602011.2014.881746

http://dx.doi.org/10.1080/09602011.2014.881746

http://dx.doi.org/10.1109/icvr.2008.4625144

http://dx.doi.org/10.1109/icvr.2008.4625144

http://dx.doi.org/10.1080/03610730590948177

http://dx.doi.org/10.1080/03610730590948177

http://dx.doi.org/10.1023/B:NERV.0000009483.91468.fb

http://dx.doi.org/10.1023/B:NERV.0000009483.91468.fb

http://dx.doi.org/10.1007/s00787-004-0405-z

http://dx.doi.org/10.1007/s00787-004-0405-z

http://dx.doi.org/10.2190/E79U-VCRC-EL4E-HRYV

http://dx.doi.org/10.2190/E79U-VCRC-EL4E-HRYV

http://dx.doi.org/10.3109/02699052.2015.1075148

http://dx.doi.org/10.3109/02699052.2015.1075148

ities, 32, 2608–2613. http://dx.doi.org/10.1016/j.ridd.2011.06.014

Henry, M., Joyal, C. C., & Nolin, P. (2012). Devel-opment and initial assessment of a new paradigmfor assessing cognitive and motor inhibition: Thebimodal virtual-reality Stroop. Journal of Neuro-science Methods, 210, 125–131. http://dx.doi.org/10.1016/j.jneumeth.2012.07.025

Iriarte, Y., Diaz-Orueta, U., Cueto, E., Irazustabar-rena, P., Banterla, F., & Climent, G. (2016). AU-LA-Advanced Virtual Reality Tool for the Assess-ment of Attention: Normative Study in Spain.Journal of Attention Disorders, 20, 542–568.http://dx.doi.org/10.1177/1087054712465335

Josman, N., Hof, E., Klinger, E., Marie, R. M., Gold-enberg, K., Weiss, P. L., & Kizony, R. (2006).Performance within a virtual supermarket and itsrelationship to executive functions in post-strokepatients. 2006 International Workshop on VirtualRehabilitation. http://dx.doi.org/10.1109/iwvr.2006.1707536

Josman, N., Kizony, R., Hof, E., Goldenberg, K.,Weiss, P. L., & Klinger, E. (2014). Using the virtualaction planning-supermarket for evaluating executivefunctions in people with stroke. Journal of Strokeand Cerebrovascular Diseases, 23, 879–887. http://dx.doi.org/10.1016/j.jstrokecerebrovasdis.2013.07.013

Josman, N., Schenirderman, A. E., Klinger, E., &Shevil, E. (2009). Using virtual reality to evaluateexecutive functioning among persons with schizo-phrenia: A validity study. Schizophrenia Research,115, 270–277. http://dx.doi.org/10.1016/j.schres.2009.09.015

Jurado, M. B., & Rosselli, M. (2007). The elusivenature of executive functions: A review of ourcurrent understanding. Neuropsychology Review,17, 213–233. http://dx.doi.org/10.1007/s11065-007-9040-z

Kang, Y. J., Ku, J., Han, K., Kim, S. I., Yu, T. W.,Lee, J. H., & Park, C. I. (2008). Development andclinical trial of virtual reality-based cognitive as-sessment in people with stroke: Preliminary study.CyberPsychology & Behavior, 11, 329 –339.http://dx.doi.org/10.1089/cpb.2007.0116

Klinger, E., Chemin, I., Lebreton, S., & Marié, R. M.(2006). Virtual action planning in Parkinson’s dis-ease: A control study. CyberPsychology & Behav-ior, 9, 342–347. http://dx.doi.org/10.1089/cpb.2006.9.342

Kurtz, M. M., Baker, E., Pearlson, G. D., & Astur,R. S. (2007). A virtual reality apartment as a mea-sure of medication management skills in patientswith schizophrenia: A pilot study. SchizophreniaBulletin, 33, 1162–1170. http://dx.doi.org/10.1093/schbul/sbl039

Lalonde, G., Henry, M., Drouin-Germain, A., Nolin,P., & Beauchamp, M. H. (2013). Assessment of

executive function in adolescence: A comparisonof traditional and virtual reality tools. Journal ofNeuroscience Methods, 219, 76–82. http://dx.doi.org/10.1016/j.jneumeth.2013.07.005

Lambert, A. E., Watson, J. M., Stefanucci, J. K.,Ward, N., Bakdash, J. Z., & Strayer, D. L. (2016).Stereotype threat impairs older adult driving. Ap-plied Cognitive Psychology, 30, 22–28. http://dx.doi.org/10.1002/acp.3162

Lamont, R. A., Swift, H. J., & Abrams, D. (2015). Areview and meta-analysis of age-based stereotypethreat: Negative stereotypes, not facts, do the dam-age. Psychology and Aging, 30, 180–193. http://dx.doi.org/10.1037/a0038586

Manchester, D., Priestley, N., & Jackson, H. (2004).The assessment of executive functions: Comingout of the office. Brain Injury, 18, 1067–1081.http://dx.doi.org/10.1080/02699050410001672387

Matheis, R. J., Schultheis, M. T., Tiersky, L. A.,DeLuca, J., Millis, S. R., & Rizzo, A. (2007). Islearning and memory different in a virtual envi-ronment? The Clinical Neuropsychologist, 21,146 –161. http://dx.doi.org/10.1080/13854040601100668

Meneghetti, C., Muffato, V., Suitner, C., De Beni, R.,& Borella, E. (2015). Map learning in young andolder adults: The influence of perceived stereotypethreat. Learning and Individual Differences, 42,77– 82. http://dx.doi.org/10.1016/j.lindif.2015.08.015

Miller, K. J., Adair, B. S., Pearce, A. J., Said, C. M.,Ozanne, E., & Morris, M. M. (2014). Effectivenessand feasibility of virtual reality and gaming systemuse at home by older adults for enabling physicalactivity to improve health-related domains: A sys-tematic review. Age and Ageing, 43, 188–195.

Mintz, A. R., Dobson, K. S., & Romney, D. M.(2003). Insight in schizophrenia: A meta-analysis.Schizophrenia Research, 61, 75–88. http://dx.doi.org/10.1016/S0920-9964(02)00316-X

Mitsis, E. M., McKay, K. E., Schulz, K. P., Newcorn,J. H., & Halperin, J. M. (2000). Parent-teacherconcordance for DSM–IV attention-deficit/hyper-activity disorder in a clinic-referred sample. Jour-nal of the American Academy of Child & Adoles-cent Psychiatry, 39, 308–313. http://dx.doi.org/10.1097/00004583-200003000-00012

Newcorn, J. H., Halperin, J. M., Schwartz, S., Pas-cualvaca, D., Wolf, L., Schmeidler, J., & Sharma,V. (1994). Parent and teacher ratings of attention-deficit hyperactivity disorder symptoms: Implica-tions for case identification. Journal of Develop-mental and Behavioral Pediatrics, 15, 86 –91.http://dx.doi.org/10.1097/00004703-199404000-00004

Nolin, P., Martin, C., & Bouchard, S. (2009). Assess-ment of inhibition deficits with the virtual class-room in children with traumatic brain injury: A


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

http://dx.doi.org/10.1016/j.ridd.2011.06.014

http://dx.doi.org/10.1016/j.ridd.2011.06.014

http://dx.doi.org/10.1016/j.jneumeth.2012.07.025


http://dx.doi.org/10.1177/1087054712465335

http://dx.doi.org/10.1109/iwvr.2006.1707536

http://dx.doi.org/10.1109/iwvr.2006.1707536

http://dx.doi.org/10.1016/j.jstrokecerebrovasdis.2013.07.013



http://dx.doi.org/10.1016/j.schres.2009.09.015

http://dx.doi.org/10.1016/j.schres.2009.09.015

http://dx.doi.org/10.1007/s11065-007-9040-z

http://dx.doi.org/10.1007/s11065-007-9040-z

http://dx.doi.org/10.1089/cpb.2007.0116

http://dx.doi.org/10.1089/cpb.2006.9.342

http://dx.doi.org/10.1089/cpb.2006.9.342

http://dx.doi.org/10.1093/schbul/sbl039

http://dx.doi.org/10.1093/schbul/sbl039



http://dx.doi.org/10.1002/acp.3162

http://dx.doi.org/10.1002/acp.3162

http://dx.doi.org/10.1037/a0038586

http://dx.doi.org/10.1037/a0038586

http://dx.doi.org/10.1080/02699050410001672387

http://dx.doi.org/10.1080/13854040601100668

http://dx.doi.org/10.1080/13854040601100668

http://dx.doi.org/10.1016/j.lindif.2015.08.015

http://dx.doi.org/10.1016/j.lindif.2015.08.015

http://dx.doi.org/10.1016/S0920-9964%2802%2900316-X

http://dx.doi.org/10.1016/S0920-9964%2802%2900316-X

http://dx.doi.org/10.1097/00004583-200003000-00012

http://dx.doi.org/10.1097/00004583-200003000-00012

http://dx.doi.org/10.1097/00004703-199404000-00004

http://dx.doi.org/10.1097/00004703-199404000-00004

pilot-study. Studies in Health Technology and In-formatics, 144, 240–242.

Nolin, P., Stipanicic, A., Henry, M., Joyal, C. C., &Allain, P. (2012). Virtual reality as a screening toolfor sports concussion in adolescents. Brain Injury,26, 1564 –1573. http://dx.doi.org/10.3109/02699052.2012.698359

Okahashi, S., Seki, K., Nagano, A., Luo, Z., Kojima,M., & Futaki, T. (2013). A virtual shopping test forrealistic assessment of cognitive function. Journalof Neuroengineering and Rehabilitation, 10, 59.http://dx.doi.org/10.1186/1743-0003-10-59

Parsons, T. D., Bowerly, T., Buckwalter, J. G., &Rizzo, A. A. (2007). A controlled clinical compar-ison of attention performance in children withADHD in a virtual reality classroom compared tostandard neuropsychological methods. Child Neu-ropsychology, 13, 363–381. http://dx.doi.org/10.1080/13825580600943473

Parsons, T. D., & Carlew, A. R. (2016). Bimodalvirtual reality Stroop for assessing distractor inhi-bition in autism spectrum disorders. Journal ofAutism and Developmental Disorders, 46, 1255–1267. http://dx.doi.org/10.1007/s10803-015-2663-7

Parsons, T. D., McPherson, S., & Interrante, V.(2013). Enhancing neurocognitive assessment us-ing immersive virtual reality. Paper presented atthe 17th IEEE Virtual Reality Conference, Green-ville, SC.

Parsons, T. D., & Rizzo, A. A. (2008). Affectiveoutcomes of virtual reality exposure therapy foranxiety and specific phobias: A meta-analysis.Journal of Behavior Therapy and ExperimentalPsychiatry, 39, 250 –261. http://dx.doi.org/10.1016/j.jbtep.2007.07.007

Pas, E. T., & Bradshaw, C. P. (2014). What affectsteacher ratings of student behaviors? The potentialinfluence of teachers’ perceptions of the schoolenvironment and experiences. Prevention Science,15, 940 –950. http://dx.doi.org/10.1007/s11121-013-0432-4

Patterson, T. L., Lacro, J., McKibbin, C. L.,Moscona, S., Hughs, T., & Jeste, D. V. (2002).Medication management ability assessment: Re-sults from a performance-based measure in olderoutpatients with schizophrenia. Journal of ClinicalPsychopharmacology, 22, 11–19. http://dx.doi.org/10.1097/00004714-200202000-00003

Peli, E. (1998). The visual effects of head-mounteddisplay (HMD) are not distinguishable from thoseof desk-top computer display. Vision Research, 38,2053–2066. http://dx.doi.org/10.1016/s0042-6989(97)00397-0

Pollak, Y., Shomaly, H. B., Weiss, P. L., Rizzo,A. A., & Gross-Tsur, V. (2010). Methylphenidateeffect in children with ADHD can be measured byan ecologically valid continuous performance test

embedded in virtual reality. CNS Spectrums, 15,125–130.

Pollak, Y., Weiss, P. L., Rizzo, A. A., Weizer, M.,Shriki, L., Shalev, R. S., & Gross-Tsur, V. (2009).The utility of a continuous performance test em-bedded in virtual reality in measuring ADHD-related deficits. Journal of Developmental and Be-havioral Pediatrics, 30, 2–6. http://dx.doi.org/10.1097/DBP.0b013e3181969b22

Rand, D., Basha-Abu Rukan, S., Weiss, P. L., &Katz, N. (2009). Validation of the Virtual MET asan assessment tool for executive functions. Neuro-psychological Rehabilitation, 19, 583–602. http://dx.doi.org/10.1080/09602010802469074

Rand, D., Katz, N., & Weiss, P. L. (2007). Evaluationof virtual shopping in the VMall: Comparison ofpost-stroke participants to healthy control groups.Disability and Rehabilitation, 29, 1710 –1719.http://dx.doi.org/10.1080/09638280601107450

Raspelli, S., Pallavicini, F., Carelli, L., Morganti, F.,Pedroli, E., Cipresso, P., . . . Riva, G. (2012).Validating the neuro VR-based virtual version ofthe Multiple Errands Test: Preliminary results.Presence, 21, 31– 42. http://dx.doi.org/10.1162/PRES_a_00077

Rizzo, A. A., Bowerly, T., Buckwalter, J. G., Klim-chuk, D., Mitura, R., & Parsons, T. D. (2006). Avirtual reality scenario for all seasons: The virtualclassroom. CNS Spectrums, 11, 35–44. http://dx.doi.org/10.1017/S1092852900024196

Sayago, S., Sloan, D., & Blat, J. (2011). Everydayuse of computer-mediated communication toolsand its evolution over time: An ethnographicalstudy with older people. Interacting with Comput-ers, 23, 543–554. http://dx.doi.org/10.1016/j.intcom.2011.06.001

Sayal, K., & Taylor, E. (2005). Parent ratings ofschool behaviour in children at risk of attentiondeficit/hyperactivity disorder. Acta PsychiatricaScandinavica, 111, 460–465. http://dx.doi.org/10.1111/j.1600-0447.2004.00487.x

Schwarz, N. (1999). Self-reports: How the questionsshape the answers. American Psychologist, 54, 93–105. http://dx.doi.org/10.1037/0003-066X.54.2.93

Swift, H. J., Lamont, R. A., & Abrams, D. (2012).Are they half as strong as they used to be? Anexperiment testing whether age-related social com-parisons impair older people’s hand grip strengthand persistence. British Medical Journal Open, 2,e001064. http://dx.doi.org/10.1136/bmjopen-2012-001064

Wallace, S., Parsons, S., Westbury, A., White, K.,White, K., & Bailey, A. (2010). Sense of presenceand atypical social judgments in immersive virtualenvironments: Responses of adolescents with Au-tism Spectrum Disorders. Autism, 14, 199–213.http://dx.doi.org/10.1177/1362361310363283


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

http://dx.doi.org/10.3109/02699052.2012.698359

http://dx.doi.org/10.3109/02699052.2012.698359

http://dx.doi.org/10.1186/1743-0003-10-59

http://dx.doi.org/10.1080/13825580600943473

http://dx.doi.org/10.1080/13825580600943473

http://dx.doi.org/10.1007/s10803-015-2663-7

http://dx.doi.org/10.1007/s10803-015-2663-7

http://dx.doi.org/10.1016/j.jbtep.2007.07.007

http://dx.doi.org/10.1016/j.jbtep.2007.07.007

http://dx.doi.org/10.1007/s11121-013-0432-4

http://dx.doi.org/10.1007/s11121-013-0432-4

http://dx.doi.org/10.1097/00004714-200202000-00003

http://dx.doi.org/10.1097/00004714-200202000-00003

http://dx.doi.org/10.1016/s0042-6989%2897%2900397-0

http://dx.doi.org/10.1016/s0042-6989%2897%2900397-0

http://dx.doi.org/10.1097/DBP.0b013e3181969b22

http://dx.doi.org/10.1097/DBP.0b013e3181969b22

http://dx.doi.org/10.1080/09602010802469074

http://dx.doi.org/10.1080/09602010802469074

http://dx.doi.org/10.1080/09638280601107450

http://dx.doi.org/10.1162/PRES_a_00077

http://dx.doi.org/10.1162/PRES_a_00077

http://dx.doi.org/10.1017/S1092852900024196

http://dx.doi.org/10.1017/S1092852900024196

http://dx.doi.org/10.1016/j.intcom.2011.06.001

http://dx.doi.org/10.1016/j.intcom.2011.06.001

http://dx.doi.org/10.1111/j.1600-0447.2004.00487.x

http://dx.doi.org/10.1111/j.1600-0447.2004.00487.x

http://dx.doi.org/10.1037/0003-066X.54.2.93

http://dx.doi.org/10.1136/bmjopen-2012-001064

http://dx.doi.org/10.1136/bmjopen-2012-001064

http://dx.doi.org/10.1177/1362361310363283

Wandke, H., Sengpiel, M., & Sönksen, M. (2012).Myths about older people’s use of information andcommunication technology. Gerontology, 58,564–570. http://dx.doi.org/10.1159/000339104

Werner, P., Rabinowitz, S., Klinger, E., Korczyn,A. D., & Josman, N. (2009). Use of the virtualaction planning supermarket for the diagnosis ofmild cognitive impairment: A preliminary study.Dementia and Geriatric Cognitive Disorders, 27,301–309. http://dx.doi.org/10.1159/000204915

White, S. W., Richey, J. A., Gracanin, D., Bell,M. A., LaConte, S., Coffman, M., & Kim, I.(2014). The promise of neurotechnology in clinicaltranslational science. Clinical Psychological Sci-

ence. Advance online publication. http://dx.doi.org/10.1177/2167702614549801

Zygouris, S., Giakoumis, D., Votis, K., Doumpoula-kis, S., Ntovas, K., Segkouli, S., . . . Tsolaki, M.(2015). Can a virtual reality cognitive trainingapplication fulfill a dual role? Using the virtualsupermarket cognitive training application as ascreening tool for mild cognitive impairment.Journal of Alzheimer’s Disease, 44, 1333–1347.

Received April 25, 2016Revision received July 23, 2016

Accepted July 25, 2016 �

Members of Underrepresented Groups:Reviewers for Journal Manuscripts Wanted

If you are interested in reviewing manuscripts for APA journals, the APA Publicationsand Communications Board would like to invite your participation. Manuscript reviewersare vital to the publications process. As a reviewer, you would gain valuable experiencein publishing. The P&C Board is particularly interested in encouraging members ofunderrepresented groups to participate more in this process.

If you are interested in reviewing manuscripts, please write APA Journals [email protected]. Please note the following important points:

• To be selected as a reviewer, you must have published articles in peer-reviewedjournals. The experience of publishing provides a reviewer with the basis for preparinga thorough, objective review.

• To be selected, it is critical to be a regular reader of the five to six empirical journalsthat are most central to the area or journal for which you would like to review. Currentknowledge of recently published research provides a reviewer with the knowledge baseto evaluate a new submission within the context of existing research.

• To select the appropriate reviewers for each manuscript, the editor needs detailedinformation. Please include with your letter your vita. In the letter, please identifywhich APA journal(s) you are interested in, and describe your area of expertise. Be asspecific as possible. For example, “social psychology” is not sufficient—you wouldneed to specify “social cognition” or “attitude change” as well.

• Reviewing a manuscript takes time (1–4 hours per manuscript reviewed). If you areselected to review a manuscript, be prepared to invest the necessary time to evaluatethe manuscript thoroughly.

APA now has an online video course that provides guidance in reviewing manuscripts. Tolearn more about the course and to access the video, visit http://www.apa.org/pubs/authors/review-manuscript-ce-video.aspx.


Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

http://dx.doi.org/10.1159/000339104

http://dx.doi.org/10.1159/000204915

http://dx.doi.org/10.1177/2167702614549801

http://dx.doi.org/10.1177/2167702614549801

Documents

Virtual Reality for Psychological Assessment in Clinical Practicepsychology.unt.edu/~tparsons/pdf/Parsons (Phillips 2016... · 2016-09-13 · Virtual Reality for Psychological Assessment