Embed Size (px)
Citation preview
Augmenting live performance dance throughmobile technology
Dr Paul GolzUniversity of Worcester
City Campus, Castle StreetWorcester, WR1 3AS
Alex ShawGlastonbridge Software Ltd
33 Balmwell GroveEdinburgh, EH16 6HG
We present a pilot study investigating the use of mobile technology to augment live performance dance.An augmented performance was created and viewed through an tablet device then analysed from a technicaland audience standpoint. Low complexity augmentations were found to be very effective, however the deviceplaced restrictions on higher complexity augmentations, the performance length and the stage/audiencespatial arrangements. A low-number audience test indicated that augmentation of live performance wasa credible concept, though there are some clear challenges to be overcome specifically around speed oftechnology and weight of device.
Augmented Reality, Art, Audience Engagement, Performance, Dance
1. INTRODUCTION
A recent interim report for the Arts Council (NestaDigital R&D 2014) found that around a quarter of artsorganisations use technology alongside/connectedwith their artwork, though the performative arts(theatre, dance) are only half as likely to usethese as their non-performative counterparts. A2011 study by Dance/USA (WolfBrown 2011)indicated that “younger audiences are categoricallymore interested in all forms of technology-basedengagement” and that this demographic is theleast interested in engaging with the traditionalperformance paradigm.
Augmented Reality (AR) is now well establishedin the visual arts (Cukrov 2011; Chayka 2011;Re+Public 2014) and artists are deliberately creatingworks to be viewed primarily through mobile devicessuch as phones and tablets. There is someevidence that use of these technologies are helpingengage the viewer (Gilroy et al. 2008). Withinthe performance arts, however, AR has not yetbeen widely adopted possibly due to the difficultiesassociated with a multiple-viewpoint paradigm andits dynamic nature leading to high computationaldemands. Where AR is used within the performativearts it tends to be provided through the addition ofpre-recorded video added to the scene when anestablish marker is detected, such as within the DansAR project (Affexity Research 2013). Those whoattempt live augmentation typically provide a uniform
viewpoint based on a single video stream capture,for example Deakin Motion.Lab and the AustralianNational Ballet (Vincs et al. 2011).
Live performance is characterised by an audiencewith slightly varying viewpoints. Additionally eachviewpoint shifts independently as the audiencechooses which point of stage to focus on. ForAR to support such a paradigm it would requirean individualised augmentation based on viewpoint.With the ubiquity of mobile devices, this becomesa matter of simply providing software that caninterpret the scene from the appropriate angle andproviding a correct augmentation. However viewinga performance through a mobile device introducesmany questions around the user experience, forexample:
• Are audience members willing to engage witha live three dimensional performance viewedthrough a two dimensional screen?
• Can we maintain a kinaesthetic connectionwith the audience when viewed through amobile device?
• Are all types of augmentation equally validwithin live performance?
With the addition of a mobile device the traditionalideas behind creating and viewing performanceneed to be reconsidered, the traditional two-wayinteraction between audience and performance
c© The Authors. Published by BISL.Proceedings of BCS HCI 2014 - Sand, Sea and Sky - Holiday HCI, Southport, UK
Augmenting live performance dance through mobile technology • Golz & Shaw
becomes a partially mediated experience: audience-computer-performance. Clearly both aspects need tobe considered from a technical and user experiencepoint of view.
In this paper we present the our initial findings intowhat is, to the best of our knowledge, the first attemptto create a personalised augmentation of a livedance performance.
2. METHODOLOGY
2.1. Overview
Augmented Reality, or AR, is the set of techniqueswhereby synthetic sense data is added congruentlyto a live, observed reality. It is the movement tocombine computed and real information to create aricher experience of both. This can include reactivemusical compositions, haptic (touch-experience)sculpture, and visual additions and effects.
Augmented reality is often characterised by the useof a digital camera and a computer to enhance avisual experience. Within the context of a danceperformance it makes sense to think of it as anextension of video dance, with certain caveats.In video dance, one takes source material andperforms editing actions on it, in augmented realityone does not modify the source video manually, butdescribes to the computer the time and manner inwhich the editing should take place. Further, theediting of the piece is beholden to maintain somesemblance of continuity with the real-world dance orthe augmentation would lose its connection to theobservers real-world experience.
Taking control of a visual experience offers severaladvantages. We can position objects on the stagewith the dancer, and not be restricted by constructioncosts, stage size, or constraints of reality. Our objectscan simulate physical interactions, but are notnecessarily bound by the laws of physics. Similarly,special effects become not only possible butlimitless; pyrotechnics are perfectly safe and humanteleportation becomes possible. Acknowledging theparallels with video dance, it is possible to add anyvideo effect that can be computed in reasonabletime.
Clearly a consideration of all possible augmentationsis beyond the scope of a single piece of work,however here we focus on the effects that are mostlikely to have artistic use within all of the performativearts, including:
2.1.1. Visual effects• full and part-screen washes
• other lighting effects, such as fades
• video filters, for example cine
• temporal adaptions (slowing or stuttering thescene)
• spatial adaptions, such as teleportation ofperformers/real-world objects
• videos out of perspective (replacing the entirescreen with a pre-recorded video)
• videos in perspective (adding pre-recordedvideo to certain parts of the stage)
• adding 3D objects to the stage in knownlocations
• adding 3D objects to the stage in computation-ally determined positions
A physics engine was used for manipulation of theadded 3D objects, such as falling under gravity orshattering (Atkin 2009).
2.1.2. Audio effectsThe sound-scape of an artistic work is consideredintegral and by using mobile devices to view aperformance we allow for augmentation of a globalsound scape with personalised elements.
2.1.3. Haptic effectsHaptic feedback, on the other hand, is not commonwithin performance and was included here todiscover whether it had artistic validity within theaugmented performance.
2.2. Performance
Live augmented dance is a natural combination oflive performance and video dance but cannot betreated as an extension of either. The audience isexpected to view the performance through a mobiledevice, however their field of vision is likely to includeelements of the stage/performer. Each audiencemember has a unique viewpoint and the dance mustmake sense to each of them.
It was considered important to use typical danceelements, including speed and spatial range, tomaintain verisimilitude. Dance is, of course, muchmore dynamic than most other performance typesand if the technologies could effectively work withinthis range then issues should not be found with mostother performance types.
In order to reduce complexity an original work (WhoWe Were, (Golz 2014)) was choreographed usinga solo performer. The performance used a typicalrange of contemporary movements based on theGraham movement vocabulary (Horosko 1991) andused traditional choreographic memes such as arange of levels and speeds. In order to facilitate
Augmenting live performance dance through mobile technology • Golz & Shaw
effects such as teleportation, and to make lightingeffects easier, the performance was held in a purewhite studio with flat white overhead lighting.
2.3. Hardware
Tablet devices (Apple iPad 2) were chosen asthe hardware platform through which the audiencewould augment their view of the performance.The device provides sufficient graphics performancerequired to ensure that the augmentations areapplied repeatedly on every video frame. If theframe rate drops too low, jittering and latency makesthe augmented vision uncomfortable and unrealistic.It is the consensus in the related field of virtualreality that a truly natural experience requires avideo rate of around 120 frames per second (see,for example (Lang 2014)). This is impractical withinthe constraints of the current project, but the fieldis developing rapidly and 120Hz video screensare becoming available for portable devices in thenear future. In the meantime, we are capturing asmuch expression as possible within a relatively high-latency system running at 8-12 frames per second.Latency is observable, but is low enough that severaleffects could be applied simultaneously.
Consideration must also be made for the audiencemember’s interaction with the technology. The AppleiPad 2 weighs around 600g, is slim and easy to holdand easily manoeuvrable. We were expecting theuser to hold it in both hands for a period of around tenminutes, regularly moving it to maintain the locationof the performer roughly within the centre of thescreen.
2.4. Software
The purpose of the software (code-named FullSpectrum Dance Software) is to combine computervision, computer graphics, and the choreographicand artistic experience of the performance.
There are several software frameworks intended foruse by digital artists that encompass these require-ments such as Processing.org, Cinder or Open-Frameworks. Here OpenFrameworks was chosenbecause it is fast and well supported with bothcomputer vision and Apple iOS development.
Several of the augmentations require accuratedetermination of the scene (adding 3D objects,teleportation) and OpenCV (Bradski 2000) was usedfor this aspect of the work. As each camera has itsown view on the scene we were unable to easily usededicated motion capture devices for scene analysis.
All the software used is licensed under an MIT freeopen-source license and can be used commercially
or shared publicly with no restrictions or additionalcosts.
2.4.1. Software pipelineThe Full Spectrum app pipeline is:
1. The tablet device receives an image from itscamera.
2. OpenCV analyses the image to work out thestage layout and the location of the dancer ifrequired.
3. Objects and effects are drawn over the cameraimage using OpenGL.
4. The composite image is shown to the audiencemember holding the tablet.
This pipeline is run once for every frame of videoshown, so it is computationally demanding andrequires both a sufficiently powerful tablet device andefficient software.
2.4.2. Integration of 3D objectsAdding 3D objects to the real-world stage so theyare in perspective and appropriately lit is complex, asthere is no simple way for a computer to decipher theperspective of a scene from a 2D image of it. Markersmade of decorating tape were placed on the real-world stage at measured positions and sizes. TheOpenCV POSIT algorithm was used to calculate atransformation between these points in the 2D imageto a known 3D position.
Figure 1: The markering system used by the FullSpectrum Dance Software
2.5. Audience
Due to the nature of the technology there are likely tobe limitations of the effective number and positioningof audience members. Given the effective viewingangle/depth of the device and limits on the detectionof the markering system, we were interested in whataudience/stage set-up would provide the optimalviewing experience.
We identified the extrema positions of the dance and,viewing through the tablet device, identified the bestviewing location. Here we considered three factors:
Augmenting live performance dance through mobile technology • Golz & Shaw
• Ability to see the performer in their entirety
• Enough of the markering system to bedetected to apply an effect
• Performer should take up between 10% and80% of the height of the screen when viewedin landscape mode.
Clearly the last factor is subjective: angular subtendsin live theatre are typically significantly smaller thanthis, however there is an expectation for users oftablets that the point of interest remains of workableheight.
2.5.1. Focus groupThis pilot study is primarily focused on the technicaland artistic considerations of the technology, ratherthan audience engagement, so a very limited focusgroup with three audience members (with variousexperience of live dance) was conducted. Audiencemembers attended a live performance within a dancestudio/performance space within the PerformanceHub at the University of Wolverhampton. Theywere provided with a conceptual overview ofthe performance concept - namely that we wereinvestigating their experience of an augmentedreality performance. They were instructed to viewthe performance through the iPad. No furtherinformation was provided on the performancemeaning nor the nature of the augmentations theywould experience.
Post-performance, audience members were led ina group discussion to discuss their experiencein their own way. The authors did not steerthe discussion towards particular issues, ratherto discover what issues seemed pertinent to theaudience. Clarification was occasionally soughtwhere meaning was not clear.
The conversation was recorded and analysed forrelevance to the research questions.
3. RESULTS
An augmented reality performance with individu-alised viewpoints as viewed through mobile deviceswas successfully created though several technicaland audience related issues were discovered. Aug-mentations fell into one of four categories:
• Full screen effects (including temporal manipu-lation)
• Local effects whose location had been prede-termined
• Local effects whose location had to be deter-mined through computer vision/computation.
• Non-visual effects
For some local effects, such as applying a video filterto part of the stage, could be applied at a knownstage location (see Figure 2) and simply had to beconverted to screen position based on the marker’slocation, whereas effects relative to the performersuch as teleportation required computer vision toanalyse the scene.
All effects were triggered through a master computerand fed to the tablets via local WiFi, rather thantriggering at a known time, or computationallydetermined. Each tablet was, however, responsiblefor calculation of the full effect. This system workedwell and there was no noticeable delay betweentriggering the effect and the effect reaching the tableteach device responded effectively simultaneouslyindicating similar processing times.
3.1. Technical
3.1.1. PerformanceThe software/hardware successfully displayed anaugmented reality performance on each of theaudience’s tablet. Full screen augmentations suchas full screen washes (Figure 2), lighting and videofilters were easy to produce, caused no noticeablelag and worked well artistically.
Figure 2: Split colour/B& W effect at known location
Figure 3: Full screen wash
However augmentations that required use ofOpenCV/POSIT (placed 3D objects as in Figure
Augmenting live performance dance through mobile technology • Golz & Shaw
4 and videos in perspective) showed noticeablelag due to the relative intensity of the calculation.Teleportation, on the other hand, which requireduse of OpenCV without POSIT worked withoutnoticeable delay. From this we conclude that thePOSIT calculation is too computationally intensive tobe carried out within a reasonable frame rate withcurrent technology.
Figure 4: Adding 3D objects at known locations
3.1.2. Non-visual effectsThe tablet was made to produce pre-determinedsounds, as triggered from the master computer.In our choreography whispered conversation wasplayed locally over a global soundtrack and was feltto be, subjectively and based on audience feedback,a dramatic effect. Haptic effects worked in a similarway but seemed less effective and felt subjectivelyout of place with the other augmentations.
3.1.3. Marker OcclusionOcclusion of the markering system by the dancerwas a significant issue. Known stage locationsare sited in space by reference to these systemsand these need to be constantly refreshed dueto potential movement of the viewpoint. When thedancer occluded the markering system the 3D objectstarted to drift (Figure 5)
Figure 5: 3D image becomes misaligned when danceroccludes marker
A way to resolve this might be to chose a markeringsystem in front of the stage rather than at the rear.
3.2. Audience
3.2.1. Location
Figure 6: Stage/audience positioning (not to scale)
The original stage size was 6m deep by 7.25m wide(see Figure 6) with two 1m square markers centredwidth-wise. One was sited on the floor starting 1mfrom the rear of the stage, the other on the rearwall with its base 1m from the ground. Based on theimage size requirements stated earlier (and using nozoom) the audience tablet needed to be around 2mfrom the front of the stage.
Given the tablets field of view of around 44◦(whichis also incidentally about the limit of accuracy forthe POSIT algorithm) this results in an availableaudience width of 2-3m, allowing for around 4-6 audience members. This is quite a restrictiveaudience count.
3.2.2. Focus groupThe work has been performed several timesthough always with very low audience counts,but the following feedback has been consistentacross audiences. Quoted words refer to audiencemembers’ description of their experience
• It was not “odd” or unusual watching aperformance through a mobile device.
• The augmentations were “cool” and “sup-ported” the work.
• There was some kinaesthetic connectionaudience members felt subjectively differentfrom watching a dance on an tablet/PC outsideof the context of a live performance.
• The lag seriously affected those aspectsof the performance. Any lag was deemedunacceptable.
• Many commented that they “wouldn’t want towatch a whole performance through it”. On
Augmenting live performance dance through mobile technology • Golz & Shaw
clarification this was due to the weight of thedevice. No-one mentioned that the weight ofthe device was an issue within the actualperformance (ten minutes in length).
4. CONCLUSIONS
Augmentation of live performance is still in itsinfancy but our work demonstrates that it istechnically feasible using commonplace mobiledevices, no dedicated hardware is required. Arelatively inexpensive tablet device can currentlybe used by small audiences to augment short(less than ten minutes) works and, while intensecomputation remains an issue affecting placementof virtual objects on the stage, there is a wide rangeof augmentations that can currently be deployed togreat artistic effect.
Feedback from limited trials reveals this augmen-tation supported and added to the artistic work.Although more substantial trials are necessary todevelop a deeper understanding of the nature ofengagement, early indications are that there are au-diences willing to engage with this new performanceparadigm and that augmentation of live performanceis a credible concept.
The next stage in the research will utilise morepowerful tablet devices which will overcome manyof the computational challenges addressed throughthis pilot study. We will explore various adaptionsto our markering system to overcome placementissues and indeed the field-of-vision limitations ofthe device, which would open up the work to largeraudiences.
The most significant change likely to affect thiswork is the arrival of smart glasses such as theSpaceGlasses 1 or similar. It is not unreasonable toexpect audiences to wear a pair of AR enhancedglasses for a much longer performance, as theycurrently do for 3D cinema. Combining glasses withthe research described in this paper should result ina dramatic and engaging new way of watching liveperformance.
5. ACKNOWLEDGEMENTS
This work was funded through a grant from ArtsCouncil England (22437756) and we are grateful totheir assistance.
The authors would also like to thank the Universityof Wolverhampton for use of their PerformanceHub, Elizabeth Howell for her artistic input,Charlotte Morewood, our dancer, Matt Timbers for1http://meta.com
photography/videography and Ephemeris Dance fortheir ongoing dedication to changing performancethrough technology.
REFERENCES
Affexity Research (2013). Dansar 01 & 02. Availablefrom: http://affexity.se/post/46328568353/
dansar-01-02-feb-march-2013 (10 June 2014).
Atkin, M. (2009). ofxMSAPhysics. Availablefrom: http://www.memo.tv/ofxmsaphysics/ (1May 2014).
Bradski, G. (2000). The OpenCV Library. Dr. Dobb’sJournal of Software Tools.
Chayka, K. (2011). How augmented reality is goingviral in the art world, from the omi sculpture parkto a 9/11 memorial. Blouin ArtInfo.
Cukrov, C. (2011). Beck’s global augmented-reality art gallery. Available from: http:
//www.psfk.com/2011/07/becks-global-
augmented-reality-art-gallery.html#!Q3E2B
(16 May 2014).
Gilroy, S. W., Cavazza, M., Chaignon, R., Makela,S.-M., Niranen, M., Andre, E., Vogt, T., Urbain,J., Billinghurst, M., Seichter, H., and Benayoun,M. (2008). E-tree: Emotionally driven augmentedreality art. In Proceedings of the 16th ACMInternational Conference on Multimedia, MM ’08,pages 945–948, New York, NY, USA. ACM.
Golz, P. (2014). Who we were. Dance Work.
Horosko, M. (1991). Martha Graham: the evolutionof her dance theory and training, 1926-1991. ACappella books, Atlanta.
Lang, B. (2014). Umbra software positions occlusionculling tech for 120 fps vr gaming. Availablefrom: http://www.roadtovr.com/umbra-
software-occlusion-culling-120-fpt-
virtual-reality-gaming/ (17 July 2014).
Nesta Digital R&D (2014). Digital culture: Howarts and cultural organisations in england usetechnology. Report.
Re+Public (2014). Urban art projects. Available from:http://www.republiclab.com/projects (1 May2014).
Vincs, K., Divers, P., McCormick, J., Skovli, D., andBurt, R. (2011). Choreotopography (remix).
WolfBrown (2011). How dance audiences engage:Summary report from a national survey of danceaudiences. Report.
