Situating the Performer and the Instrument in a Rich Social Context with PESIExtended System

Callum GoddardDepartment of Media

Aalto UniversitySchool of Arts, Design and Architecturecallum.goddard@aalto.fi

Koray TahirogluDepartment of Media

Aalto UniversitySchool of Arts, Design and Architecturekoray.tahiroglu@aalto.fi

ABSTRACT

In this paper we present our solutions to the design chal-lenges of facilitating awareness of actions and develop-ment of self-identities within The notion of ParticipatoryEnacting Sonic Interaction (PESI) project. The PESI sys-tem is a modular framework for participatory music mak-ing with three performers. We present a brief technicaloverview, design considerations and revisions resulting froma user study conducted during the system’s development.Through the development process of the PESI project adesign approach we term: Non-Behaviourally RestrictiveDigital Technology became apparent. In this approach,the shifting focus that embodied agents have in relation tothe environment is accounted for and the development ofsound-action relationships are encouraged. This is achievedthrough providing mappings relating to individual sensorvalues and movement information from motion trackingdata. Our approach to the implementation of the PESI sys-tem can shift the collaborative music activity to a more en-gaging and active experience.

1. INTRODUCTION

Designing systems in which social interaction is the pri-mary focus is challenging as it requires consideration ofadditional factors along side the social interaction. Anawareness and understanding of the evolving nature of theperformer-instrument relationship within specific contextsand cultures is needed to inform how these relationshipsmay be facilitated through the technology. The idea of per-formance ecosystems helps to address these design chal-lenges by emphasising how social factors effect and fa-cilitate changes in the function of technology and music.Through the consideration of performance ecosystems, wehighlight the importance of usage in technology which al-lows for the blurring of phenomenological and epistemicdistinctions between acoustic and digital technology [1,2].In blurring these traditionally held distinctions, we are ableto focus directly on investigating the ideas of social inter-action in collaborative music. Through investigating these

Copyright: c©2013 Callum Goddard et al. This is an

open-access article distributed under the terms of the

Creative Commons Attribution 3.0 Unported License, which permits unre-

stricted use, distribution, and reproduction in any medium, provided the original

author and source are credited.

ideas, we have implemented our own system that is capableof supporting collaborative and creative activities in groupmusic practices.

This paper presents our design approach to The notion ofParticipatory Enacting Sonic Interaction (PESI) project, amodular framework for participative music making withthree performers. The project incorporates new generationmobile phones and group motion tracking technology tocreate an environment in which performers’ individual andsocial actions contribute to and affect the sonic output. Wehave briefly presented previous versions of the PESI sys-tem in [3–5]. The system has been developed on the iOSplatform along with the use of the Microsoft Kinect Sys-tem. Mobile phones enable individual action within thesystem and the Kinect system tracks participants, enablingaugmentation of the social space within the system. Theresult is that the PESI system can facilitate group musicpractices that exploit social action in combination with theuse of everyday devices for allowing musical action.

Compared to other approaches based upon analysis of so-cial behaviours within musical practice, such as [6] and theEU-ICT SIEMPRE project, 1 the PESI project instead hasfocused on facilitating social action through technologywithin a musical context. As such, the design challengesof the PESI system relate to the ideas of awareness andmutual engagement within Human Computer Interaction(HCI). Finding solutions to these challenges has guided thedesign and development process, and in doing so, we haveidentified a design approach which we call: behaviourallynon-restrictive digital technology.

This paper begins with an overview of related work on theperformance ecosystem approach. Section 3 presents thedesign challenges and section 4 provides an technical anddesign overview of the PESI system. Section 5 outlinesour idea of behaviourally non-restrictive digital technol-ogy, which is discussed in section 6, where the approach iscompared with the work described in section 2. The paperis concluded in section 7.

2. RELATED WORK

2.1 Ecological Perspectives Towards Music

The ecological perspective has been used within the designand implementation of interactive digital music systems as

1 http://siempre.infomus.org/

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

368

a way to contextualise and investigate relationships that de-velop when using these systems [7–11]. One approach,which has been given notable attention, is that of perfor-mance ecosystems. Based on ideas suggested by SimonWaters [12], a performance ecosystem is a tool to under-stand current musical activity. The central idea is that “un-derstanding music making as a complex dynamic systemputs it in terms of the process of creation, but also its con-solidation into culture specifically as a social practice em-bodying behaviours, beliefs and actions” [12]. Here musicmaking is seen as an activity that produces artefacts as wellas being part of social practice.

Two projects that embrace the performance ecosystemperspective are the Audible Eco-Systemic Interface (AESI)project [7] and in ‘Infra-instruments’ [8]. Within both theseprojects the notion and design practices of interaction withininteractive systems are questioned. The AESI project ques-tions the performers’ role in interactive systems. In manyinteractive systems the performer is the person providingexternal conditions that the dynamic behaviour of a systemis driven by. The system is depended on the performer.Within the AESI project the need for a performer is re-moved. The relationship of performer, instrument and en-vironment is reduced to only being between the instrumentand the environment. This is accomplished through a feed-back loop in which the system generates the controls fromanalysis of its ambient surroundings.

Similarly, the Infra-instruments project draws compar-isons to and diverges from the approaches used within thedevelopment of ‘hyper’ and ‘meta’ instruments. Hyper andmeta instruments are approaches that extend traditional in-struments’ interactive capabilities, expressive nature andvirtuosity through the addition of sensors. Instead of ex-tending the instrument, Bowers and Archer see value insimplifying and limiting the interaction. Their approachcreates a space within the performance setting for the ad-ditional capabilities of computers [8].

The ecological perspective has also been taken by [9–11] to investigate the relationships between performers andspectators of interactive digital music system performances.Their research has highlighted considerations within thedesign of digital musical instruments (DMIs) to improvespectator experience within these performances. The im-provement of experiences with DMIs and music technol-ogy is also a concern within the area of embodied musiccognition and mediation [13]. Whilst not explicitly draw-ing upon ecological ideas, the focus is upon how we in-teract with music. The ideas developed within the field ofembodied music cognition have been used to inform thedevelopment of an interface exploring musical experienceand creativity [14]. The Musical Paint Machine extendsthe performance space of a player so that the sonic outputof the instrument is visually represented. In this way, addi-tional feedback modalities are introduced to the player asa method of stimulating creativity.

2.2 Blurring Distinctions Between Instruments

The works presented have followed Waters’ main ideas onperformance ecosystems; however, work by Green has ex-

tended these ideas further towards musical creation. Greenemphasises the influence that social factors have in the useof technology and musical practice. When players andinstruments are situated within a social world ‘the cate-gorical distinctions between the acoustic and the digitaldissipate somewhat, and that such differences in practiceare contingent upon the shifting intersections between thetechnical and social’ [2]. Thus the influence that socialfactors have on the usage of technology results in the dis-tinctions 2 between acoustic and digital being blurred.

Accounting for relationships between the different partsof a system, from a conceptual stand point, is one of themain challenges in adopting an ecological perspective. Inadopting Greens contingency view on musical instrumentswe can direct our own focus towards the consideration ofsocial factors and their influences on the design of inter-active systems. In the following section we briefly addressthese challenges in the PESI project by considering and ac-counting for the performers’ relationship to the technologyand to each other within the design of the system.

3. DESIGN CHALLENGES

To account for forms of embodied interaction in the imple-mentation of the PESI system we have investigated the de-sign challenges surrounding the role of meaning in relationto collaborative interactive systems. This has directed ourfocus to concerns relating to intersubjectivity, and the de-sign of technology that facilitates co-operative processes.

Intersubjectivity is the way in which two people can shareunderstanding of the world, or how meaning can be sharedbetween two people [15–17]. To establish and allow forintersubjectivity within a collaborative system, users needto be able to interpret and understand the action of oth-ers. This is required for a communication flow to be es-tablished, thus enabling collaboration within the activity.Similarly participants need to be aware of what others aredoing or have done, also known as having public aware-ness of actions [18]. Facilitating awareness in collaborativesystems is very important in allowing for multiple users tointeract with each other. This is emphasised by both thefields of HCI [15] and Computer-Supported CollaborativeWork (CSCW) [18]. Awareness of actions is also an im-portant design feature in facilitating mutual engagement,allowing for a more socially engaging experience [19, 20].

Issues of awareness within areas of HCI and CSCW andin research into mutual engagement in social music havemainly focused on activities in which participants are notco-located. Within the PESI system the participants areco-located, to account for this we extended the ideas ofawareness and mutual engagement through considering theexperiences of individual users. We draw on ideas of opti-mal experience, Flow [21], for this.

The exact conditions for achieving a Flow state are stillbeing investigated, however, a key component of obtain-ing a Flow state within an activity is the maintenance ofones own personal identity [21]. The importance of self-identity within optimal experiences has prompted us to ex-

2 These are the phenomenological and epistemic distinctions betweenacoustic and digital instrument technology [1]

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

369

Figure 1. Three musicians using the PESI system in a freeform improvisation.

tend ideas of awareness and mutual engagement so thatusers area able to retain the ability to develop and act throughtheir own self-identities within the system. This requiresenabling users to be identifiable as other identities withinthe system, and that these identities can be influenced byothers, through a their own self identities [21]. It also re-quires that each user be able to detach themselves, and theiractions, from the overall interactive system.

We believe that facilitating the building of identities aidsin, and strengthens, awareness within collaborative sys-tems of co-located participants. Therefore, the design chal-lenges of the PESI system are focusing on facilitating bothawareness of actions, and allowing users to create theirown identities within the system.

4. THE PESI SYSTEM

In the PESI system mobile devices run custom softwarethat allows for them to be used as musical instrumentsthat are usable within an improvised musical group perfor-mance [4]. The relationship and interaction between theperformer and their mobile instrument is extended into thephysical and social space through the use of motion track-ing software and group analysis.

4.1 Technical Justifications

From the start of the project, the PESI system was designedwith a modular structure. The modular structure aided indevelopment by allowing for rapid system reconfigurationas well as the testing of ideas. This structure also makesthe system more accessible to others to modify or to ex-tend for their own uses. In that concern, we used readilyavailable technology: Apple iPhones as the mobile phones,Microsoft Kinect as the motion tracking system and PureData 3 for the sound synthesis.

Including mobile phones allows us to emphasise the roleof social communication within the system as they repre-sent communication within our society. iPhones also havethe additional benefit of being association with music prac-tice. iDevices are largely connected with music listeningas well as music production through the ever increasingnumber of musical applications available on the platform.

3 http://puredata.info/

They also contain the technology required by the PESI sys-tem: sensor input feedback mechanisms, enriched com-putational possibilities for sound processing and wirelesscommunication.

The decision to use the Microsoft Kinect system withinthe PESI system was determined in a similar manner tothe iPhones. The Kinect system is a cultural object that isprimarily used within games that require whole body inter-action. It allows us to emphasise the playful nature of thesystem as well as being able to visual track and allow fordetailed evaluation of a group of people.

Pure data was the audio programming language of choicefor the PESI system’s sound synthesis. The primary rea-son pure data was chosen over other audio programminglanguages is due to its portability. The development ofLibpd 4 has made it possible to run Pure Data on many dif-ferent platforms. Within the PESI system this meant thatthe same sound synthesis patch can run on both iPhonesand Laptop Computer. It also sufficiently fulfils the soundsynthesis needs of the PESI system.

4.2 The Initial System and the Design OutcomesArising from it’s User Study

Development of the PESI system has gone through mul-tiple iterations. Work began on a simple system in whichonly mobile phone instruments were augmented with rela-tional parameters generated from a Microsoft Kinect sys-tem. Through a user test we were able to assess the effectextending the controls into the social domain had on theplayability of the mobile phone instrument [3]. This re-sulted in an extension of the system, to allow for furtherideas relating to development of the mobile phone instru-ments to be explored and implemented.

Figure 2. A group of three participants taking part in firstuser study of the PESI system.

The first user study used an initial simple system imple-mentation and was primarily conducted to gain insight intothe effect socially control parameters had on the interac-tion. 21 participants: 8 female and 13 male, aged 25-48were involved in the study. Test participants were dividedinto 7 groups of three players, each group participated ina separate session (see Fig. 2). In each session we askedthe users to use the system in two scenarios. In the first

4 http://libpd.cc

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

370

scenario there were no social parameters effecting the mo-bile instruments, in the second scenario, a parameter cal-culated from the average distance between group membersaltered an amplitude distortion effect. The evaluation ofthe user test included a quantitative survey and qualitativeinterviews. We presented our findings in detail in [3].

In conducting this user study two design issues were high-lighted. The first issue related to the average distance valueof the group, participants found it was very hard to perceiveand understand. Using the individual distance parametersbetween participants was much more understandable. Webelieve this is due to the fact that the interaction is obscuredwhen using an average distance parameter. The behaviouris only revealed when the group synchronises, causing theactions to not be publicly available when the group is notsynchronised. Thus, the difficulties in perception of thedistance parameter’s effect on the sound. We believe thisis due to a breakdown of awareness from the lack of publicavailability of the actions relating to this parameter.

As an additional step we chose to reinforce and separatethe interaction controlled by social parameters and indi-viduals. For this, we decided to have the mobile instru-ments only controllable by the person playing it, and usean extended system for socially controlled sound synthe-sis. These became the on-body and in-space componentsof the PESI system respectively (see section 4.3).

The second issue was a technical problem that arose dueto occlusion. Occlusion occurred when one or more par-ticipants blocked the Kinect’s view of another participantand resulted in the Kinect temporarily losing track of theoccluded participant.

The main problem that occlusion caused was technical.Within our implementation of the final PESI system thereis a requirement to track the identities of performers. Forthe initial user test we were able to mitigate the problemthat occlusion caused by removing the need to user IDs.This was not possible with the final implementation as IDswere required to ensure sonic cohesion between the on-body and in-space components .

Each user is assigned a unique ID value by the Kinectsystem, when the Kinect loses track of a user the ID isunassigned. Once the user reappears a new ID is assignedto that user. Thus, when occlusion occurred, a new ID isassigned by the Kinect to the occluded performer caus-ing the PESI system to miss match the sound synthesisbetween the on-body and in-space components causing abreakdown in sonic cohesion. This further compounds theproblems of awareness, particularly relating to self-identifywithin the system.

The issues relating to occlusion and social mapping thatarose from the user study indicated that the system had notprovided solutions to the design challenges mentioned insection 3, particularly those relating to awareness. How-ever, in revealing these problems potential solutions werealso presented, for example: not using average distancemappings and developing methods to prevent or at least re-duce occlusion. These solutions where implemented whenrefining and continuing the development of the system,which we assessed in a second user study (see section 4.3.3).

Figure 3. PESI extended system module diagram.

4.3 Design Report

The current system has two main parts: on-body and in-space components (see Fig. 3) [4, 5].

4.3.1 The on-body Component

The on-body component consists of a custom native iOSapplication built with Objective-C and using Libpd, de-signed to run on iPhones. This component also includesportable speakers that are directly connected to the mo-bile phones. Running the sound synthesis on the mobilephone allowed for the co-location of the performer’s ac-tion and sonic results, reinforcing the embodiment of theinteraction. Accelerometer, gyroscope and touch data areretrieved from the sensors in the mobile phone. This datais used to generate the parameters for the sound synthesismodules.

The design of the mobile instruments were based uponparticipants’ comments from the initial systems’ user test.The performers have a choice between three different soundmodules. Only one module can be played at a time, how-ever, it is possible to switch between each module whenplaying the mobile phone instrument. Each module has itsown sonic characteristics, associated colour and a ‘tuning’system that lets users alter static parameters within the in-strument for further customisation.

The first instrument Green switches between pulse-widthmodulation (PWM) or wave shaping of a square wave de-pending on the orientation of the device. The tuning sys-tem allows for manipulation of two constant square waveswith PWM. Tilt controls frequency and touch controls tim-bre. The second, Red, and third, Blue, instruments arebased upon a granular synth. Tilting the X axis accelerom-eter controls the grain playback in both, however, in theRed instrument tilting the Y axis accelerometer changesthe grain size. The Y axis on the touch screen allows forthe playback pitch of the Blue instrument to be controlled.The tuning system in both allows for changing default set-tings within the granular synth in each instrument. For de-tailed technical descriptions of each instrument see [5].

4.3.2 The in-space Component

The idea behind the in-space component was to use a gen-erated layer of sound, related but, external to the perform-

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

371

Figure 4. Point-cloud image generated by two Kinect sen-sor bars showing the motion tracking of three performers.

ers individual mobile instruments. Through this layer thesocial interactions within the system are represented andmanifest as sounds.

The in-space component is based around a central con-trol module that receives sensor information from a multi-user motion tracking system as well as from each mobilephone. This information is used to control the sound syn-thesis of the in-space component. The multi-user motiontracking system has been built with two kinect sensor bars(see Fig. 4), the OpenNI 5 library with Processing 6 andopenFrameworks. 7 Multiple Kinects were used as sin-gle Kinect solutions to occlusion proved unreliable, twoKinects provided a robust solution to the ID switching bydramatically reducing occlusion within the system.

This system continuously tracks the spatial positions ofthree performers. From their positional information, rel-ative distances between performers, velocity, accelerationand alignment are calculated. Sensor data from the mo-tion tracking system and each mobile phones is sent to thecentral control module via Open Sound Control (OSC). 8

A robust network module was developed that managed theconnections between each component in the system. Thismodule allows the central control unit to re-establish con-nections, sort sensor information and track the devices thatthe information was sent from.

The sensor data from the mobile phones is used to gen-erate identical copies of the sounds being played by theperformers within the in-space component. These soundsare then processed with a granular synth, controlled by so-cially generated parameters and played back over a multi-channel speaker ring that surrounds the performers. Theprocessed sounds are associated to the instrument they havebeen produced from, thus are unique to each individualperformer, to further reinforce the idea of self-identity.

The granular synthesis is mapped as follows (from theperspective of a performer using the system): the grainsize and density vary depending on the distance to oneperformer and the frequency range of the playback scalesexponentially depending on the distance to the second per-former. The speed of an individuals movement within thespace controls the amplitude of the synth. For further tech-nical details see [5].

5 http://openni.org/6 http://processing.org/7 http://www.openframeworks.cc/8 http://opensoundcontrol.org/

4.3.3 A User Study of the Current System

We invited three skilled-musicians for a user test jam ses-sion with the current PESI system (see Fig. 5). Similar tothe previous user test, the evaluation was based on surveyand interview analysis. Detailed findings from this userstudy were presented in [5]. In summary, musicians com-ments and feedback were positive regarding the on-bodyand in-space component; especially for the sound charac-teristics and the responsiveness of each module. This wassupported with the following comments directly quotedfrom participants:

” ...pretty impressive... I liked the idea ofthe extended system. It gives a nice ambientand supporting feeling that you have soundaround you....”

”.... I have been using sensor based in-struments more with traditional music and tonalstructures . It would actually be very interest-ing to implement this system for a traditionalkind of music that you could improvise with.The system is actually quite interactive andthe instrument is very good. You can do allkinds of stuff with small gestures. This systemcould be open to all kinds of directions.... ”

Figure 5. Skilled musicians performing with the currentversion of the PESI system in the user test jam session.

5. BEHAVIOURALLY NON-RESTRICTIVEDIGITAL TECHNOLOGY

In solving the design issues which occurred in the variousstages of the PESI system’s development, an idea arose,which we term: Behaviourally Non-Restrictive Digital Tech-nology (BNDT). For the in-space component of the PESIsystem to operate, performers are not required to direct orfocus their attention towards it or make explicit changes totheir behaviour. Yet similarly users are able to act throughthe in-space component when their attention is directed to-wards it. When attention is directed towards this compo-nent, it is able to reveal itself to participants, in relationto theirs as well as other users’ actions, allowing them to

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

372

determine how they wish to deal with it. This can be leav-ing the system in ones peripheral perceptions or developingcouplings with it.

We see the approach of BNDT as a method of accountingfor the differing modes of use and shifting of focus that oc-curs between embodied agents and the environment [15].The shifting of focus can also be related to the ideas ofidentity creation, discussed in section 3, as the design ofthe system allows for the detachment of the user from thesystem. In allowing for detachment and coupling, usersare able to reconfigure their focus and understanding ofthe system - which we believe aids in the development ofidentities within the system.

However, accounting for differing modes is only one partof our idea of BNDT. Actions are still an important partof our investigation into design solutions surrounding theideas of intersubjectivity and awareness within collabora-tive digital music systems.

5.1 Being Social in the World

Actions themselves are not a kind of act, they are insteadproperties of individuals at times [22], and are groundedin our movements. We effect objects around us by mov-ing our bodies and we are able to produce different move-ments to fit a large range of circumstances. The effect thesemovements will have is also generally understood, as theresults of movements are dependent upon the context andsituation in which they occur [22].

An important idea relating to the understanding of actionswithin the context in which they arise from, is the notionof accountability. The notion of accountability is a fun-damental feature of the ethnomethodological perspective,and is concerned with what is available to members as sit-uated practice. Members being those who have commonsense understandings relating to the situation [15]. In par-ticular, situated practice is the context in which the actionarises and thus can be part of the means by which actioncan be interpreted. Therefore, actions relating to situatedpractice are understood as normal, rational action by mem-bers. The availability of these actions depends on them be-ing able to be observed by members and reported upon:observable-and-reportable [15].

The methods of understanding an action are also the meth-ods for engagement with it. As such being competent withan action requires that it is engaged in ways that are recog-nised by members. Therefore, an action is needed to beorganised in such a way that it is understood as being ra-tional within the context it has emerged from. In the caseof the PESI project, the focus was on the sound and actionrelationships that occur in the context of music.

5.2 Musical Gestures

The relationship between sound and action is being inves-tigated within the field of musical gesture, which focuseson classifying musically related actions. Within this fieldactions are separated into two categories: sound-producingand sound-accompanying. [23]

Sound-producing actions are the actions used in the pro-cess of making sound. Godøy et al. [23] make a distinc-

tion between excitation action - which triggers the sound,and modulation action - which modifies the sound. Sound-accompanying actions are the other types of actions whichare performed along to sound/music, but are not part of itsproduction; for example playing air guitar or tracing thedynamics of the music. Sound-accompanying actions nor-mally have a readily observable sound matching feature tothem [23]. Within the PESI system we have primarily con-cerning ourselves with sound-producing gestures. How-ever, we also accept the need to be aware of all musicallyrelated action that may arise within the PESI system.

One of the main aesthetic considerations we had in re-lation to the system was that it would not be using tradi-tional acoustic instrument sounds. The sounds being usedwould be synthesised to create new sounds, to be interactedwith and used within a musical context. Therefore, we en-countered the problem of there being no established sound-action relationships for the sounds being synthesised withthe PESI system. This required the design of new sound-action relationships for the PESI system.

We see sound-action relationships as inherently cultural.Associations of gestures that relate to the production ofspecific sounds are developed through the combination ofcultural practice and physical constraints of the musical in-strument [13]. This adds an additional level of complexityif we were to design new sound-action relationships. In-stead we have continued to follow our idea of BNDT anddecided to not predefine any sound-action, or musical ges-tures within the system. Instead we have used movementmappings to sound parameters.

5.3 No-predefined Gestures

We facilitate action through allowing movements to havean effect within the system. Performers can develop theirown actions through the movements that the system cantrack. By combining motion tracking information withsensor data from mobile phones it is possible for manymovements - both individual (through moving the mobiledevice) and social (through movement in space) to be usedto generate sound responses. Actions can then arise in re-sponse to the sounds, caused through individual and socialmovements, and through this, gestures are able to be devel-oped by performers within the system. We believe that thisis possible when the resulting actions are made publiclyavailable, allowing for all actions to be interpreted and ac-counted within the interaction.

In having no-predefined gestures, the idea of BNDT isfurther enforced, and the interaction is able to become fullysituated, aiding in the potential for adoption into practice.As a further result of our approach we see the possibilityto investigate formations of musical gestures from move-ment within a controlled and traceable environment. Ourresearch interest lies in a similar direction to [24] in that itdoes not to look for stated understanding of gesture, but tounderstand how movements are actually employed in theinterfaces that researchers have built.

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

373

6. DISCUSSIONS

When we compare the PESI system to the work discussedin section 2, we can highlight many common themes. Com-paring to the work of Di Scipio [7] and the AESI system,both the PESI system and the AESI system have a focus onthe performer and their role within the system. However,unlike the AESI system which questions if the performeris even needed, we take the opposite approach in which theperformer’s role is strengthened becoming an integral partof the PESI system.

Our approach to having no-predefined gestures could beseen as following Bowers and Archer’s [8] ideas relatingto infra-instruments. In only focusing on movements ouroverall mapping strategies have been reduced and some-what restricted - like those of infra-instruments. We havealso allowed for greater flexibility within the social prac-tices of the PESI system, as well as providing space for thetechnology and computing that enables and enhances themusical activity.

Comparing the work of the PESI project to that of Gure-vich and Trevino, Gurevich and Cavan Fyans and CavanFyans et al. [9–11] and their work directed towards spec-tator experience, we can see benefit in having actions pub-licly available. However, we can foresee a potential chal-lenge faced by spectators of the PESI system. This is dueto actions being created within the use of the PESI system.Those who are not a part of the improvised practice maystruggle from a spectators stand point to grasp or under-stand the actions within the system, as they would not bemembers of the practice. This is a side-effect for primarilyfocusing on facilitating improvised musical practice andnot on performances that feature spectators.

The musical paint machine takes the biggest depart fromthe ideas we have been using with the PESI system. Thisis because it has been designed to investigate musical prac-tice, musical gesture, Flow and stimulation of creativity[14]. Here, an element that has not been part of our de-signs within the PESI system has been used: the additionof multi-modal feedback. Within the PESI system we haveonly focused on sonic feedback; however, the sonic feed-back within the PESI system also provides information onsocial actions of the group, not just on each individualspersonal action. We do see some parallels to the idea ofbehaviourally non-restrictive digital technology within themusical paint machine project. When using the musicalpaint machine one can chose their focus between the in-strument and the feedback provided by the musical paintmachine. This is similar to the PESI system, however, per-formers can chose between the social, personal or neitherto direct their focus.

We acknowledge that there has been little discussion sur-rounding the sounds and music the system creates. Musicitself is an aesthetic form and throughout the design of thesystem we have drawn upon our own aesthetic preferencesto drive the development of the sound synthesis within thesystem. In theory, we see, our approaches implying a gen-erality in assigning sound mappings to the interaction asthe suitably of such mappings is determined through theemergent action-meaning relationship that, we believe, we

have facilitated within the PESI system. The effect this hason aesthetic preferences within a social context requiresfurther exploration and experimentation.

7. CONCLUSIONS

In this paper we presented our solutions to the design chal-lenges that were faced within the development of the PESIextended system. In viewing the technology and practicesthat relate to music from an ecological perspective, the re-lationships between performers themselves and their in-struments became our focus. Drawing upon work withinthe fields of HCI, CSCW and research into Flow the needto account for awareness and creation of identities wereidentified as key considerations when designing the PESIsystem.

Two evaluations were carried out at different stages inthe PESI systems development. It was identified withinthe first evaluation that relational parameters between allthree group members were not as perceivable. We hypoth-esise this was due to the actions of an individual no longerbeing publicly available due to the other group membersobscuring a single members contribution. Within the de-velopment of the in-space component these considerationswhere accounted for and solutions were implemented. Re-lational parameters were relational between each group mem-ber but never related to the whole group. This was done toallow for better public availability of actions as well as toenable a stronger development of self-identity within thesystem. The results from the second evaluation supportour design choices.

Through the development of the PESI system we devel-oped an idea of non-behaviourally restrictive digital tech-nology. Within this approach, the shifting focus that em-bodied agents have in relation to the environment is ac-counted for, and sound-action relationships are encouragedto be developed through providing mappings relating toindividual sensor values and movement information frommotion tracking data. Through the situatedness of the per-former and instrument, we believe that musical gestureswill arise that could not have been anticipated within thedesign of the system due to the influence social factorshave on the usage of the mobile instruments. When dis-cussing these ideas in relation to the related works dis-cussed within section 2 the ideas we have developed canbe see within many of these projects.

The contextualisation of sounds within social settings isstill an open question within the PESI system and withincollaborative digital music systems. Further investigationinto the idea of behaviourally non-restrictive digital tech-nology is also required, not only within the design of digi-tal music systems, but in a wider context of digital technol-ogy. Both these points are to be considered within furtherwork on the PESI system which will be moving to focusupon long-term movement behaviour between performers.

Acknowledgments

This work is supported by the Academy of Finland (project137646).

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

374

8. REFERENCES

[1] T. Magnusson, “Of epistemic tools: musical instru-ments as cognitive extensions,” Organised Sound,vol. 14, no. 2, pp. 168–176, 2009.

[2] O. Green, “Agility and playfulness: Technology andskill in the performance ecosystem,” Orangised Sound,vol. 16, no. 2, pp. 134–144, 2011.

[3] R. Pugliese, K. Tahiroglu, C. Goddard, and J. Nesfield,“A qualitative evaluation of augmented human-humaninteraction in mobile group improvisation,” in Pro-ceedings of the International Conference on New Inter-faces for Musical Expression, University of Michigan,Ann Arbor, May 21 – 23 2012.

[4] N. N. Correia, K. Tahiroglu, and M. Espada, “Pesi: Ex-tending mobile music instruments with social interac-tion,” in Seventh International Conference on Tangi-ble, Embedded and Embodied Interaction (TEI), Workin Progress, Barcelona, Spain, 2013.

[5] K. Tahiroglu, N. N. Correia, and M. Espada, “Pesi ex-tended system: In space, on body, with 3 musicians,”in Proceedings of the International Conference on NewInterfaces for Musical Expression, Daejeon, Korea,May 27-30 2013.

[6] A. Camurri, D. Glowinski, M. Mancini, G. Varni, andG. Volpe, “The 3rd international workshop on socialbehaviour in music: Sbm2012,” in Proceedings of the14th ACM international conference on Multimodal in-teraction, Santa Monica, California, 2012.

[7] A. D. Scipio, “‘sound is the interface’: from interactiveto ecosystemic signal processing,” Organised Sound,vol. 8, no. 3, pp. 269–277, 2003.

[8] J. Bowers and P. Archer, “Not hyper, not meta, not cy-ber but infra-instruments,” in Proceedings of the 2005International Conference on New Interfaces for Mu-sical Expression (NIME05), Vancouver, BC, Canada,2005.

[9] M. Gurevich and J. Trevino, “Expression and its dis-contents: Toward an ecology of musical creation,” inProceedings of the International Conference on NewInterfaces for Musical Expression, New York City, NY,United States, 2007, pp. 106–111.

[10] A. C. Fyans and M. Gurevich, “Perceptions of skillin performances with acoustic and electronic instru-ments,” in Proceedings of the International Conferenceon New Interfaces for Musical Expression, Oslo, Nor-way, 30 May - 1 June 2011.

[11] A. C. Fyans, A. Marquez-Borbon, P. Stapleton, andM. Gurevich, “Ecological considerations for partici-patory design of dmis,” in Proceedings of the 2012Conference on New Interfaces for Musical Expression,University of Michigan, Ann Arbor, May 21 – 23 2012.

[12] S. Waters, “Performance ecosystems: Ecological ap-proaches to musical interaction.” in EMS : Electroa-coustic Music Studies Network Performance, 2007.

[13] M. Leman, Embodied Music Cognition and MediationTechnology. The MIT Press, 2008.

[14] L. Nijs, B. Moen, M. Lesaffre, and M. Leman, “Themusic paint machine: Stimulating self-monitoringthrough the generation of creative visual output usinga technology-enhanced learning tool,” Journal of NewMusic Research, 2012.

[15] P. Dourish, Where the Action Is. The MIT Press, 2004.

[16] H. D. Jaegher and E. D. Paolo, “Participatory sense-making an enactive approach to social cognition,” Phe-nomenoogy and the Cognitive Sciences, 2007.

[17] T. Fuchs and H. D. Jaegher, “Enactive intersubjectivity:Participatory sense-making and mutual incorporation,”Phenom Cogn Sci, vol. 8, pp. 465–486, 2009.

[18] T. Robertson, “The public availability of actions andartefacts,” Computer Supported Cooperative Work,vol. 11, pp. 299–316, 2002.

[19] N. Bryan-Kinns, “Mutual engagement in social musicmaking,” LNICST, vol. 78, pp. 260–266, 2012.

[20] ——, “Mutual engagement and collocation withshared representations,” International Journal ofHuman-Computer Studies, 2012.

[21] M. Csikszentmihalyi, Flow: The psycology of optimalexperiance. HarperCollins Publishers Inc., 1990.

[22] D. Israel, J. Perry, and S. Tutiya, “Actions and move-ments,” in Proceedings of IJCAI, 1991.

[23] R. I. Godøy, A. R. Jensenius, A. Voldsund, K. Glette,M. Høvin, K. Nymoen, S. Skogstad, and J. Tørresen,“Classifying music-related actions,” in Proceedings ofthe 12th International Conference on Music Perceptionand Cognition and the 8th Triennial Conference of theEuropean Society for the Cognitive Sciecnes of Music,July 23-28 2012.

[24] J. Donovan and M. Brereton, “Movements in gestureinterfaces,” in Critiacal Computing 2005 - BetweenSense and Sensibility, the Fourth Decennial AarhusConference, Proceeding of the Workshop : Appraochesto Movement- Based Interaction, 2005.

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

375