Virtual Hamlet: Combining Motion Capture and Real Time Digital Puppetry AugustusWendellwendell@njit.eduNewJerseyInstituteofTechnologyUnitedStatesofAmerica

LouisWellswells@njit.eduNewJerseyInstituteofTechnologyUnitedStatesofAmerica As a theatrical production, Hamlet poses aparticulardirectorialchallenge:theGhost.Howshouldanapparitionbepresentedonstagealongsidemortalcharacters? Traditional Shakespearean theatrepresented apparitions using theatrical effects tovisually separate these characters from the mortalcharacters. Modern theatrical productions mustaddress this same issue with every new staging.Solving thisquestioncanhelp laya clear conceptualfooting for the rest of the production’s conceptualconsiderations.ArecentproductioninNewYorkCityin2015directedbyAustinPendletonoptedtomaketheGhostinvisibletoeveryonebutthoseguardswhofirst encounterhim, andonceHamlet seeshim,onlyHamlet. This decision rendered Hamlet mentallydisturbed in the eyes of the audience (Isherwood2017).KennethBranagh’s cinematicHamlet in1996createdaghostthathada“specialeffectsandahorror-filmlook.”(Maslin1996)Thisapproachfavorsbrutalrealism expected in modern cinema. These twoapproaches represent extremes in productionsolutionstotheghost;psychologicalillness,andpalefacesinarmor. WhenHamletwasfirstproduced,theElizabethanstage was full of special effects (Brockett, Franklin2008). Shakespeare was a practitioner of specialeffects technology such as pyrotechnics, rope andpulleyandtrapdoors.Hiscommitmenttoauthenticityandeffecteven ledto theaccidentaldestructionofatheaterbyfirein1613aftertheuseofacannon-basedspecial effect. As conceived by faculty in a theatreprogramandadigitaleffectsdesigndepartmentatthe

same university, the challenge of representing theghostandtheShakespeareanspiritoftheatricaleffectbecameanopportunityforcollaboration.Thiswasouropportunity to create a new type of ghost, one thatbridges the traditions of theater and cutting edgeinteractivemedia. Theater has a history of stage projections datingfromthe1700’suseofmagiclanterndevices(Figure1)tothe1920’swheninnovatorslikeErwinPiscatorbegantheirexperimentswiththemedium(Figure2).

Figure 1. A magic lantern projector device.

Figure 2. Erwin Piscator staging of Sturmflut, 1926 with pro-

jected imagery.

Throughout the twentieth century there was anextraordinary growth in the implementation ofprojectionsfromexperimentaltomainstreamtheater(TheatreCommunicationsGroup2011).Whilethesizeand complexity of projections has grown over timetheyhavebeenuntil recently staticorpre-recorded.WithemergingtechnologiessuchasMicrosoft’sKinectCamera and Derivative Software’s TouchDesigner

projectionscanbeaninteractive,responsiveadditionto the live theatrical experience. Our collaborationutilizedthesetechnologiesincreationofourghost.Theatrehasalongtraditionofextendingthebodyinperformance.Costumes,makeupandprostheticsarethe bread-and-butter of the art. Greek theatre usedlargescalemasksandcostumestoamplifytheatricalexpression,whileElizabethantheatreutilizedextrememakeupapplicationsandcostumingforsupernaturalcharacters. Physical puppetry provides a clearmechanismtoextendtheperformativeandexpressivecapabilities of the actor. There are, however, somepractical limitations to the physical puppetryapproach: the movement and visual effect of thepuppet is limited to the physical realities of theperformer.Pushingbeyondtheseboundaries,cinemahas used digital motion capture to create suchcharactersasGollumintheLordof theRings trilogyandtherecentPlanetoftheApesfilms.AndySerkis,anactor known for hismotion captured performances,states, “Performance capture is a tool that allowsactors to transform themselves into many differentcharacters.You’renotconfinedbyphysicality.Youcanplay anything” (Hart 2017). Advances in virtualstagingandperformancecapture,notablyusedintheproductionof the filmAvatar,haveextendedmotioncaptureforcharactersintotherealtimerealm. Although hybrid physical/digital stagings havebeenundertakenbefore(Meador,W.Scott,etal2009,Bermudezetal2002),thefieldhasnotkeptpacewithrapidly evolving technology. Our productioncombined recent advances in inexpensive real-timemotioncapturewiththetheoreticalunderpinningsoftheatrical puppetry to perform a spatio-digitalcharacterinalivetheatricalvenue.AKinectsensoronstage captured the movements of a physical actorperforming the ghost. This datawasmanipulated inDerivative Software TouchDesigner and opticallyprojectedasanabstractdigital apparitionbackontothestageconcurrenttotheperformance(Figures3,4).Thevirtualpuppetbecameatoncepuppetandavatar,bothextendingandreplacingthephysicalbodyoftheactor.Thedualspaceofthisperformance,halfbody-sizedspaceof theactorandhalfvirtualizedspaceofthe projected apparition, plays to the notion of theCybridspace(Anders1999).

Figure 3. Technical rehearsal still from the NJIT 2016 Ham-

let Performance.

Figure 4. Production still from the NJIT 2016 Hamlet

Performance.

Thispostercontextualizes,describesandpresentstheSpring2016theatricalproductionofHamletattheNew Jersey Institute of Technology featuring adigitallycreatedprojectedparametricGhostcharacterperformedinrealtimebyamotion-capturedonstageactor.Bibliography Anders,P.(1999)EnvisioningCyberspace.1sted.NewYork:

McGraw-Hill,1999.Print.Bermudez, J, Agutter, J, Syroid, N, Lilly, B, Sharir, Y,

Lopez,T,Westenskow,D,andForesti,S (2002). "In-terfacingVirtualandPhysical Spaces through theBody:thecyberPRINTProject."InThresholds-Design,Research,EducationandPracticeintheSpaceBetweenthePhysicalandtheVirtual:Proceedingsofthe2002AnnualConfer-

enceoftheAssociationforComputerAidedDesigninAr-chitecture, 395-400. ACADIA. Pomona, California: CalPoly,Pomona,2002.

Brockett,O.G,andHild,F.J.(2008).HistoryoftheThea-

tre.Boston:Pearson.Print.Delbridge,M.,andTompkins,J.(2012)."Reproduction,

mediation,andexperience:virtualreality,motioncap-tureandearlymoderntheatre."Space–Event-Agency–Experience.

Hart,H.(2017)."WhenWillAMotion-CaptureActorWinAn

Oscar?".WIRED.N.p.,Web.7Apr.2017.Isherwood,C.(2017)."Review:‘Hamlet’AsAnAfter-Party

ThatGotOutOfHand".Nytimes.com. N.p.,Web. 1Apr.2017.

Maslin, J. (1996) "Hamlet". Partners.nytimes.com. N.p.,

1996.Web.1Apr.2017.Meador,W.S.,etal.(2004)"Mixingdancerealities:collab-

orativedevelopmentoflive-motioncaptureinaper-formingartsenvironment."ComputersinEntertainment(CIE)2.2:12-12.

TheatreCommunicationsGroup(2011)UniversityLi-

braryHomePage,2016http://www.tcg.org/publica-tions/at/dec11/projection.cfm.[31March2016].