Activating Narcissus: Cognitive and Affective Systems Transformed Through "Serious" Game Play

8/2/2019 Activating Narcissus: Cognitive and Affective Systems Transformed Through "Serious" Game Play.

1/12

Ac#va#ngNarcissus:Cogni#veandAffec#veSystems

TransformedThrough"Serious"GamePlay

HeidiBoisvert MeiSiElectronicArtDepartment Cogni#veScienceDepartment

RensselaerPolytechnicIns#tute RensselaerPolytechnicIns#tute

[email protected] [email protected]

Gamesaspopularartformsoffertoall animmediatemeansofpar3cipa3oninthefull

lifeofasociety,suchasnosingleroleorjobcanoffertoanyman.MarshallMcLuhan1

With the rapid development of computer and game console hardware, graphics, ar7ficialintelligenceandnetworktechnologies inrecentyears, videogamesarebecomingmoreand

morecapableofprovidingavivid,fic7onalrealitywithinwhichplayerscanimmerse,andhave

a


2/12

GamesDevelopmentonference(GD),the worlds largest professional-only gameindustry

event, has addeda newG4 track, indica7ng that thegap between serious games andthe

mainstreamindustryis closing. GamescholarStewartWoodshasevenventuredthatserious

gamesarethegoal of thosewithinthegame industry for thefutureof games, no7ng that

manydevelopers wishtocreateseriouscontentorexperiences thataretypically represented

withintradi7onalnarra7veformssuchasbooksorfilm.2

Almostall booksongamedesigncitetheDutchanthropologist,JohanHuizinga,as thefirstto

interrogate, thoughnever quiteresolve,thedis7nc7onbetweenseriousness andplay.But a

lesser knownar7factis lark.Abts 190sbooken7tledSeriousGames,whichaddresses

theuseof analogboardgames in educa7on, science, government and industry. In hisfirst

chapter,7tled TheReunionofAc7onandThought,Abtforwardsansurprisingly an7cipatory

defini7onofserious games: Weare concernedwithseriousgames in thesensethat these

gameshaveanexplicitandcarefullythoughtouteduca3onalpurpose andarenotintendedto

beplayedprimarilyforamusement...this doesnotmeanthatseriousgamesarenot,orshouldnotbeentertaining.3

Asearly asthe1980s,scholars,too,begantoseethepoten7alofvideogamesforeduca7on

andenhancinglearningoutcomes.Gameassessmentresearch,therefore,primarilytargetsfive

mainareas:learningtolearn throughnovel environmentalengagement, skilldevelopment,

such as induc7ve discovery and problem-solving, and hand-eye coordina7on, mo7va7on,

memoryreten7on,andu7lityforspecial groupssuchasa


3/12

oflearningwould involvemanydifferentelements,includingananalysisofcausalmechanisms,

logical explana7ons, crea7on and defense of arguments, management of limited resources,

tradeoffsofprocessesin acomplexsystem,andawayto resolve conflicts...Moreshallow level

include perceptual learning, motor skills, defini7on of words, proper7es of objects, and

memoriza7onoffacts.4

Theyconcludethatasidefromthe depthofthe skills affordedbya seriousgame,thereisthe

persistentques7onoftheiru7lityandrelevancetotherealworld.AsGraesseretalbemoan:

[t]he scien7fic status of game features proposed by game designers is greatly in need of

computa7onal andempiricalinquiry.Aside from the computa7onal essenceof games,thereis

theques7onofwhat makesthemsuccessful psychologically.Atthispoint inthescience,there

are few firmanswers to such ques7ons about the essence of gamesand their psychological

impact.5

Even when the empirical data exists, the quality of research design possesses severe

shortcomings.InEvalua7ngthePoten7al ofSeriousGames:WhatanWeLearnfromPrevious

ResearchonMediaEffectsandEduca7onal Interven7on,MarcoEnnemosermakesa strongandwell-reasonedcri7queof current naive researchmethodologiesemployedin studying

seriousgames,andinpar7cular,the over-relianceonmedia effects. InvokingMcLuhan,he

counsels[t]hemedium isnot themessage.Dontstudy mediaeffects.6 Hecallsforbe


4/12

Enjoyment, perhaps, is one of the most taken for granted, and oCen under-inves7gated

assump7ons pervading the impact of serious games on learning, development and social

change. Theplayersreac7ontoserious gamesimplicitlysupportsanincreasein enjoyment,

which translates into a new found interest in a topic, what sikzentmihaly calls the flow

experience.Flowoccurswhenthelearnerhassuchintense concentra7onthat7meandfa7guedisappear.

Yet in awide evalua7on of seriousgames by uihuaShen,HuaWang, andUte Ri


5/12

that "there is no [mental] ac7vity that is not situated."9 Next, wewill illustrate how our

essen7almentalandpsychologicalfunc7onsareembodiedaccordingtothistheory.

lark, therefore,mapsoutamoregranulardepic7onofcogni7ondistributedbetweenmind,

bodyandworld,whichhelabels "principlesofecological assembly,"whereinheproposesthat

theembodiedagent(whathelabelsthe"cannycognizer")recruits,some7mes,exploits,oCen

onthespot, a mix ofproblem-solving resources/opportuni7esprovidedby dynamic loopsof

perceptual andmotor rou7nes,combinedwithneuralprocessingandstorage,ac7ve sensing,

and iterated bouts of environmental affordances, which produce self-s7mula7ng cycles of

material scaffolding toyieldan acceptable resultwitha minimumofeffort,andno central

meaner(lark,13).

2.1DecisionMaking

Embodiedcogni7ontheory pointsout thatour decisionmakingprocess is embodied, which

means itdoesnothappenpurelyinourmind,butleverageresourcesintheenvironmentandinpar7cular our body-environment rela7onship.Therefore,physical movementscanhelpus to

think, solveproblems,andmakedecisions. This has been evidencedin many scenarios. For

example,peopleoCenwalkarounda roomwhiletryingto decidewherethefurniture should

go,andwedonot rotateTetris piecesbecauseweknowwherethey cango,but ratheras a

methodtofindoutwheretheycango.10

2.2Emo7on

Both our own emo7onal experience and our ability to understand anothers emo7onal

experiencearedeeply rootedinourphysicalbody.This canbetracedbacktotheJames-Lange

theory of emo7on, which states that emo7on is the mind's percep7on of physiological

condi7onsthatresultfromsomes7mulus. 11 Thetheory (arrivedatbyWilliamJamesandarl

Lange independently in 1884) famously underscores this by way of example, when James

remarks:ommonsensesays,wemeeta bear,arefrightenedandrun...Thehypothesishereto

bedefendedsays thatthis orderofsequenceis incorrect....Withoutthe bodilystates following

ourpercep7on,the la


6/12

(evenanavataronascreen)inthesamewayaswhenwemoveourownbody. 13Thisprovides

aphysiologicalbasisforhowpeoplecanunderstandeachother,haveempathy,andtreatone's

owngestureasasocials7mulus.

2.3Percep7on

Evenourbasicperceptualfunc7onsarenotstraighorwardandimmediateprocesses.Instead,

percep7onrequiresinterac7onwiththeenvironment.As AlvaNoe,a proponentoftheenac7ve

view,observesthe"sensorimotormodel ofpercep7onsuggeststhatanimportantrole is played

byembodiedac7onintermsofinforma7onpickupandini7allytuningcircuitrywhichsupports

perceptualawareness."14ForNoe,sensorypercep7onisnotjustsomethingthatunfoldsinthe

brain,butinsteadamodeofac7veandmo7vatedexplora7onoftheenvironmentdrawingon

animplicitunderstandingofsensorimotorregulari7es.To model visioncorrectly,forexample,

he believes "wemustmodel it not as something that takes place inside the brain, but as

somethingthatdirectlyinvolves notonlythebrain,butalsotheanimatebodyandtheworld."15

Whendiscussinghumanpercep7onandac7on,J.J.Gibsonsimilarlyestablishesthatthewayourbody actsservestwofunc7onssimultaneously:one isapproachingtoaccomplishthecurrent

task, such as standing s7ll, catching a ball or reaching a des7na7on, and the other is

informa7oncollec7onforthenextcycleofac7onplanning.16

2.4Memory

Moreover,our memory is oCenoff loadedto theenvironment. This can best be illustrated

through Gibsons concept of affordance, which he describes as a person-environment

rela7onship, such as the height ra7o of the stairs to a persons leg length, which can be

immediatelypickedupbythepersonfordecidinghowhe/sheshouldact.Infact,Gibsons view

ofpercep7onandac7ondoes notinclude theconstructofmemory.Instead,pastexperienceis

encodedintovariousaffordancesoftheenvironment.

Memory canalso beoff loaded to body-environment rela7onships that arecreated by the

individual.Kinesthesiais suchanexample. IntheGoldin-Meadows2003 study,twomatched

groupsof childrenwere asked tomemorize a list, and then carry out somemathema7cal

problemsolvingbefore tryingtorecall the list. In thestudy,onegroupcould freely gesture

during an intervening math task, while the other group was asked not to move while

conduc7ngthesametask.Theresults showedthatthegroupthatwas notallowedtogesture

13 McNeil, D. Action via Mirror Neuron System. (Cambridge: Cambridge University Press, 2005), 250.

14 Noe, Alva. Action in Perception (Representation and Mind). (Cambridge, Ma: MIT Press, 2005), 87.

15 ibid, 30.

16 Gibson and Gibson, 77. need full citation.


7/12

performedsignificantly lower inthememoryrecall testthanthegroupthatwasabletomove

aroundduringthe interveningmathtask. As Goldin-Meadowconcluded,"thephysical actof

gesturing plays an ac7ve (not merely expressive) role in learning, reasoning, and cogni7ve

changebyprovidinganalterna7ve(analog,motoric,visuospa7al)representa7onalformat."1

2.5Embodiedogni7onDepartsfromEarlierTheoriesofogni7on

The theoriesof embodiedcogni7onare fundamentallydifferent frommany other prevailing

theories ofcogni7on.Forexample,RenDescartes describedadualis7crela7onshipbetween

ourmindandbody,whichclaimsthatthehumanmindandconsciousnessareanabstracten7ty

whichisseparatedfromthephysical body.Earlytheoriesofhumancogni7on, stemmingfrom

cyberne7cs, perceived themind as a collec7on ofmentalprocessesand func7ons. Herbert

Simon defines cogni7ve science as "the study of intelligence and intelligent systems, with

par7cular referenceto intelligentbehavior ascomputa7on."18 Whilethisdefini7ondoesnot

completely rule out thepossibility that our physicalbody plays a role in consciousnessand

cogni7on, it clearly places more emphasis on computa7on and informa7on processing, andinscribesthecomputer as a perfectmetaphor for how ourmindfunc7ons. HowardGardner

goesfurtherandargues thatitis necessary (andsufficient)to"positalevelofanalysiswholly

separatefromthebiologicalorneurological,ontheonehand,andthesociological orcultural,

ontheother.19 This viewsuggeststhathumanconsciousnessandcogni7oncanbesimulated

orcreatedfromnon-physiologicalformsusingrepresenta7onsunrelatedtoourphysicalbody.

Recent studies into cogni7ve extension, which reify thecri7cal roleboth thebody andthe

environmentplayinins7ga7ngandsustainingcogni7veprocesses,therefore,offerinsightinto

thepoten7alofusingembodiedtechnologiestoenhancedeeperlearningoutcomesinserious

games.Mostexis7nggamesrelyongeneric,mechanical gesturesimposedbygamecontrollers,

which have li


8/12

3.ogni7onExtensionEnactedinSeriousGamesthroughEmbodiedogni7on

Inthemuch cited,seminalTac7le-VisualSubs7tu7onSystemstudyconductedoverthecourse

ofseveralyearsby BachyRita,20

a blindsubject isrigged intoa head-mountedcamera withsensorsa


9/12

structuresthattransformproblemspacesinways thataid(orsome7mesimpede)thinkingand

reasoningaboutsometargetdomain(s).Suchorganism-inducedmodifica7onofvirtualspaces

in games, which oCenmimic real spaces, but are far more permeable, offer an alternate

mechanismforus toenhanceproblem-solvingandskilla


10/12

neutrally realized process of thought,but instead a "coupledneural-bodily unfolding that is

itselfusefullyseenasanorganismicallyextendedprocessofthought."25 Movement,inessence,

isaformofthinking.

Interes7ngly, emo7on and affect rely upon a similar dynamic coupling, or feedback loop.

Emo7ons,Damasioclaims,arecomplex,largelyautomatedprogramsofac7onsconcoctedby

evolu7on,26 and like other aspectsof our brain func7onandconsciousness,emerged, and

con7nuetoevolve,as aliferegulator. Theac7ons,hesuggests,arecomplementedbya

cogni7on program that includes certain ideas and modes of cogni7on, but the world of

emo7ons is largely one of ac7ons carried out in our bodies, from facial expressions and

posturesto changes in viscera and internalmileu.2 Thus, affec7ve cues in games,which

triggerdopamine,causesanemo7on-feelingcycle,whichservestosustainengagement,which

inturndeepenslearningoutcomes.

3.3RealizeEmbodiedogni7oninSeriousGamesusingKinectTheMicrosoCKinect,whichcametomarket inNovember2010,is anaddondevicefor the

XBox360,whichconsistsofsoCwareandanonrange cameratechnology,whichinterprets 3D

sceneinforma7onfromacon7nuouslyprojectedinfraredstructuredlight.Thesensorfeatures

a"RGBcamera,depthsensorandmul7-arraymicrophonerunningproprietarysoCware,which

provides full-body 3D mo7on capture, advanced gesture and facial recogni7on and voice

recogni7on capabili7es," per thecompany website. S7ll fairly primi7ve, theKinect canonly

capturetwentyjointsperplayer,andtrackuptwoac7veplayers,whomustac7vatethesensor

throughhandgesturesfor thescenetocalibrate. But once ac7vated, theKinect providesa

device-free interface for theplayer tousetheir body movementsandgestures tonaturally

interact,therebyembodyingcommunica7on.

Asdiscernedabove,theenvironment,virtual orphysical, inwhichperceptually guidedac7ons

takeplaceoffers individuals amaterialoutlet for cogni7vescaffolding(definedbyVgotsky as

theprocessbywhicha learner leverages theenvironmenttoaidin learningbyoffloadinglow

level brainac7vi7es).Therefore,ifthe Kinect's sensor-driveninterac7vescreen,whichresponds

togesture andvoice,servesas boththevirtualenvironmentandanotherlayeroffeedbackon

topoftheself-s7mula7ongeneratedbythegesturealone,thescreencanpoten7ally func7on

as aquasi,real 7mecogni7venicheconstruc7on. Indoingso,this malleableandresponsive

25 Clark, Andy. Supersizing the Mind: Embodiment, Action and Cognition. (New York: Oxford UniversityPress, 2008), 126.

26 Damasio, Antonio. Self Comes to Mind: Constructing the Conscious Brain. (New York: Pantheon,2010), 113.

27 ibid, 114.


11/12

subs7tutefora physicalenvironmentlendsitselftoprogramming--construc7ng--func7onality

toenhancelearningandmodifybehaviorchange.

4.EmbodiedLearning:AaseStudy

Whileembodiedlearningprac7cesfacilitatedbysensor-basedtechnologyiss7llrela7velynew,

afewresearchers,likeourselves,havebeguntodocumentitspromiseineduca7onalsengs.

Aseminal studyon"Technology-enabledEmbodiedLearningEnvironments"ini7atedbyMina

. Johnson-Glenberg,DavidBirchfield,Lisa Tolen7no,andTatyanaKoziupaas partofa STEM

research grant recently concluded at Arizona State University's Situated Mul7-Media Arts

LearningLab(SMALLab). onductedovertwosixdaysessions,withhighschoolstudents,the

same teacher, and curricular content, the researchers broke the students into two control

groups androtatedstudents betweentechnology-basedembodiment learningenvironmentat

theSMALLab, andregularinstruc7onina classroom. Inbothsengs, thestudy focusedon

teaching7tra7on(addingunknownsolu7onofacidorbasetoa knownreactantofunknownmolarity un7l the endpoint of reac7on occurs), which is part of any tradi7onal chemistry

curriculum.

TheSMALLabis aneduca7onalplaormthatengagesthemajor sensemodali7es(i.e. visual,

auditory,andkinesthe7c)thathumans usetolearn.Itusesinfraredmo7ontrackingcamerasto

send informa7on to computersabout wherea student is in a floor-projectedenvironment.

Studentsstepintotheac7ve, definedspaceandgrabawand(atrackableobject)thatallows

thephysical bodytonowfunc7onlike a3Dcursorintheinterac7vespace.Thespaceallowsfor

mul7ple students (upto4) tobe tracked simultaneously. Teachersusewirelessperipheral

devices(remotecontrols andgame remotes)tocontrol theflowofthe dynamicvisual,textual,

physicalandsonicmediathatstudentsinteractwith.

Notsurprisingly,thestudyfoundthatplacementintheembodiedenvironmentconsistentlyled

togreaterlearninggains (effectsizes from.53to1.93).Theresearchersbelievethefindingsare

duetothreeprimaryfeatures:embodied,kinesthe7clearning,thehighdegreeofcollabora7on

designedintothelearningplaorm,andthe"noveltyeffect"oftheenvironmentalexperience.

Thestudyprofessesthatinternalknowledgestructures that aregatheredinan ac7ve mul7-

modal andkinesthe7cmannerarelearnedfaster,andthatfirst degreeembodied learningisdriven by locomo7on (increased sensorimotor ac7va7on and its varying environmental

affordances),percep7onofimmersion,andtangibleobjectmanipula7onwithcontent-relevant

gestures.

MinaJohnson-Glenberg,DavidBirchfield,LisaToledinoandTatyanaKoziupaconcludethat:


12/12

Ifthecogni7veprimi7vesofunderstandingareembodied,thenlearningenvironmentsthathighlightthe

physicalgroundedknowledge,andthatinducementalsimula7onsof abstract constructs like chemical

reac7ons...maybemorepowerfullearningmethodsthan techniques emphasizing observa7on and

symbolmemoriza7on.28

Whilesuchfindings are bynomeansconclusive,andbarelyscratchthescien7ficsurface,thesestrategiesofferinspiringpossibili7esfortheintegra7onofdevice-freekinesthe7cengagement.

5.onclusionandFutureWork

Mul7pleresearchareas nowsupportthe tenetthatembodimentis apowerful underpinningof

cogni7on. To a lesser extent, some disciplines accept cogni7ve extension as an ac7ve,

unhierarchicalnego7a7on between thebody,mindandenvironment that equally op7mizes

cogni7on.Butwhencombinedwithserious gamemechanics,whichas D.W.Shaffercontends

havetremendouseduca7vepowertointegratethinking,social interac7onandtechnologyinto

thelearningexperience,

29

opportuni7esforincreasingimpactgrowexponen7ally.

Bothpersuasive,culturalstrategiesandemerging,embodiedtechnologypossesstheabilityto

deeplyencodeus withunseenmoral,socialandpoli7calstructures.Serious gamedesign,which

harnessescommercial mediamechanicsandnarra7vedevices, andappliesa deepknowledge

of cogni7ve neuroscience to natural user interfaces can enhance cogni7ve and affec7ve

systems. We argue that such game-based experiences carry the poten7al to shape and

powerfullytransformnotonlyindividualatudes andbehaviors,butthe underlyingvaluesthat

governoursociety.

Becauseofthe Kinectsmainstreampopularity,easeofuseandaffordability,webelieveitwill

notonlychange whocanplaygames,butwill alsofundamentallyalterthe wayweengagewith

games,howwewill make gamesin thefuture, and theabilityofgames toeffect long term

learning,behaviorandemo7onaloutcomes.

To further explore how the affordances of kinesthesia, vocal recogni7on and full body

immersion, providedby theKinect canenhanceseriousgames,wehave several projects in

produc7onthatweareintheprocess ofcarryingoutoverthenexttwoyears(ofwhichwemay

showworks inprogress),which focuson: 1) increasing conceptual reten7onwhen teaching

logic,2)ins7ga7ngbehavioralchangeinex-offendersinana

Documents

Activating Narcissus: Cognitive and Affective Systems Transformed Through "Serious" Game Play