Advancements in Actuated MusicalInstruments

DAN OVERHOLT* , EDGAR BERDAHL y and ROBERT HAMILTON z

*Aalborg University, Department of Architecture, Design and Media Technology, Niels Jernes Vej 14, DK-9220 Aalborg, DenmarkE-mail: dano@create.aau.dky,zStanford University, Department of Music, Center for Computer Research in Music and Acoustics, Stanford, California 94305, USAE-mail: yeberdahl@ccrma.stanford.edu; zrob@ccrma.stanford.edu

This article presents recent developments in actuated musicalinstruments created by the authors, who also describe anecosystemic model of actuated performance activities thatblur traditional boundaries between the physical and virtualelements of musical interfaces. Actuated musical instrumentsare physical instruments that have been endowed with virtualqualities controlled by a computer in real-time but which arenevertheless tangible. These instruments provide intuitiveand engaging new forms of interaction. They are differentfrom traditional (acoustic) and fully automated (robotic)instruments in that they produce sound via vibratingelement(s) that are co-manipulated by humans andelectromechanical systems. We examine the possibilitiesthat arise when such instruments are played in differentperformative environments and music-making scenarios, andwe postulate that such designs may give rise to new methodsof musical performance. The Haptic Drum, the FeedbackResonance Guitar, the Electromagnetically Prepared Piano,the Overtone Fiddle and Teleoperation with Robothands aredescribed, along with musical examples and reflections on theemergent properties of the performance ecologies that theseinstruments enable. We look at some of the conceptual andperceptual issues introduced by actuated musical instruments,and finally we propose some directions in which such researchmay be headed in the future.

1. INTRODUCTION

In this article we discuss recent developments of actu-ated musical instruments and the implications theymay hold for the performative ecosystem surroundingthe instruments themselves, the performer and theenvironment in which they are used. Figure 1 repre-sents some of the authors’ developments, in whichelectronic signal processing techniques are applied toactuators physically embedded into the instrumentsthemselves. For example, instead of using audio com-pression algorithms to simulate the prolonged sustainof a plucked string, actuation of the strings and/orbody of the instrument enables physically palpablereal-world behaviours (e.g., with strings that aremagnetically actuated even infinite sustain is possi-ble). All of these instruments are unique – each isimbued with its own form of actuation. However, dueto their tight integration of human (gestural) and

electronic (transducer) actuation, we feel that theseinstruments all share the need for an ecosystemicapproach to performance technique and interaction.

Actuated instruments invite and even require theperformer to engage with programmed models ofmusical interaction that involve one or more layers ofinteractive feedback systems. We seek to incorporateactuation into new musical instrument designs inorder to explore alternative methods of leveraging aperformer’s (slower) consciously controlled gesturesand (faster) pre-learned gestures, by sharing some ofthe control with the machine while still providingproactive and intimate human control of the resultingmusical sounds. In our experience, effectively employedactuation can

> free up some human cognitive bandwidth in order topromote concentration on other facets of musicalendeavour;

> promote combined human and technological cap-abilities that can enable and even compel the per-former to interact in new ways via systemic feedback;for example, an actuated musical instrument canenable a performer to make gestures that wouldotherwise be difficult or impossible;

> allow the performer to physically attend to otheraspects of playing; instead of always having toinject energy into an instrument, the performercan steer external sources of (usually) electricalenergy by controlling when and how they areapplied;

> combine musical gestures from multiple musicalperformers into a networked ecosystem of co-controlled instruments, allowing performers todirectly control or affect the behaviours andcapabilities of other performers; and

> endow a physical instrument with virtual qualitiesthat are adjustable by computer in real-time butwhich are nevertheless tangible, facilitating moreintuitive and possibly even intimate interactions.

Moreover, there exists a continuum of possibleinstrument designs, between real (non-augmented,always requiring human energy input) and virtual(augmented with external energy sources, possibly

Organised Sound 16(2): 154–165 & Cambridge University Press, 2011. doi:10.1017/S1355771811000100

even fully automated) instruments. We define actu-ated musical instruments as those which produce soundvia vibrating element(s) that are co-manipulated byhumans and electromechanical systems. These instru-ments include the capability for control to be exertedby a simple or complex system of external agencyin addition to existent methods of control throughtraditional instrumental performance technique withthe instrument.

1.1. Model: performance ecosystems

Musicians have been playing musical instruments forthousands of years. In most live musical performanceparadigms, a musician employs the natural feedbackprovided by an instrument to help him or her controla performance. For example, consider the modelshown in Figure 2 describing a musician playing amusical instrument. The model is adapted from BillVerplank’s ‘Interaction Design Sketchbook’ (Ver-plank 2003), and we assume here that the musicianuses his or her hand to provide mechanical excitationto the instrument. We acknowledge that a wide rangeof other excitations and/or modifications could ofcourse be provided using the mouth, the remainder ofthe body, and so on, but we do not explicitly depictthem here for the sake of brevity.While it is clear that musicians rely heavily on

auditory feedback for ecosystemic performance, asdiscussed by Di Scipio (Di Scipio 2003), visual andhaptic feedback also play an important role. Forexample, musicians can employ visual and hapticfeedback to help orient their hands in relation to aninterface. Performers may wish to use their sense ofvision for looking at a score or looking at the audi-ence; in contrast, haptic feedback provides localisedinformation at any point on the human body. Fur-thermore, in comparison with other feedback mod-alities, haptic feedback enables the performer torespond the fastest to the behaviour of the musical

instrument. For this reason, we believe that hapticfeedback is essential for a performer who wishes toquickly and intimately interact with an ecosystemicmusical instrument.

Without yet considering environmental or audi-ence intervention, Figure 2 attempts to represent allthe necessary parts to describe a musician’s perfor-mance ecosystem (Berdahl 2009: 2). We include thebrain as the primary link in the composite feedbacksystem where the visual, auditory and haptic feed-back modalities are combined. The brain and thehand are internally interconnected by the humanmotor-system control loop. In order to understandthe complex behaviours enabled by the entire system,all of the parts must be considered. In this sense, themodel is an ecological model. In a well-designedsystem, it can be shown that the whole can becomemore than the sum of its parts, leading to emergentbehaviours. However, computer musicians are wellaware that achieving this wholeness can be challen-ging (Waters 2007: 1–20). Nevertheless, based on ourexperience we believe that actuated musical systemscan exhibit emergence if they leverage the concepts oftangibility, energy steering, enabling new kinds ofperformance interactions and freeing up cognitivebandwidth as outlined in Section 1. In addition tothese discoveries, we hypothesise that composers canobtain a whole that is larger than the sum of the partseither by trial and error, or by considering whichelements of the system the performer controls con-sciously or unconsciously.

Moving beyond traditional musical performanceecologies, wherein performers interact with oneanother through anticipated or improvised musicalgesture, there exists a novel ecosystemic model formusical performance with actuated instruments inwhich physical control of musical instruments can beshared between musical performers, based uponintentional or derived musical gesture or analysis.Actuated instruments capable of external physicalexcitation allow musicians to interact at an entirely

Figure 1. Clockwise from top left: the ElectromagneticallyPrepared Piano, the Haptic Drum, the Feedback Reso-nance Guitar, a teleoperated acoustic shaker held by theFalcon force-feedback device, and the Overtone Fiddle.

Haptic

MusicalInstrument

Auditory

Visual

Position, Velocity, or Related

BRAIN

Figure 2. Musician interacting with a musical instrumentproviding auditory, visual and haptic force feedback.

Advancements in Actuated Musical Instruments 155

new level, both with one another as well as with rule-based or reactive computer systems. By linking musicalinstruments to one another, routing musical outputfrom one as a control signal for another, new musicalpatterns and performance behaviours can emerge. Asbi-directional enactive connections are created betweenperformers, ensembles of actuated instruments trulybecome active musical ecosystems in and of themselves.In this manner, reactive instruments form the basis fora complex control schema, directly enabling co-controland shared haptic interaction between musicians andinstruments themselves.

Sayer introduces the concept of a continuumdescribing how a performer interacts with a musicalfeedback system while improvising (Sayer 2007:1–5). He argues that the continuum spans betweenextremes of reflex-based and volitional control.However, we prefer to expand this by describing thecontinuum as spanning between conscious anddeliberate control by the performer, and involuntarycontrol, where the performer’s response is dominatedby motor control system reflexes, muscle stiffness,body mass, and so forth. Based on knowledge of thehuman motor control system, we can argue that theperformer should be able to consciously controlevents happening sufficiently slowly, whereas themusical instrument could evade deliberate control ofthe performer if it takes unexpected actions fasterthan the conscious reaction time of the human motorcontrol system, which is about 120–180ms (Berdahl2009: 147–9). In other words, if an actuated musicalinstrument reacts irregularly within this initial timewindow, it will surprise the performer and be difficultto control, although this could in fact be desirable insome circumstances! Programmed instrument beha-viours lasting longer than this time window couldtheoretically be compensated for by the performer, ifhe or she is able to act quickly and decisively enoughto master the haptic force feedback. The gesturesproduced by the total ecosystem can be remarkablycomplex, and the composer should consider thequestion: who is really in control anyway?

1.3. Background: prior research in actuated musicalinstruments

1.3.1. Feedback damping of acoustic musical instruments

Force feedback can be applied to an acoustic musicalinstrument in such a rigorous way that it is possibleto actively damp acoustical vibrations. Early work inParis provided some insight into this area. Labora-tory experiments were performed on feedback controlemulation of a virtual bow (Weinreich and Causse1986), feedback control of the end of a column of air(Guerard and Boutillon 1997), and feedback controlemulation of virtual springs, virtual dampers and

virtual masses (Besnainou 1999). More recent workhas been carried out in Paris on controlling a xylo-phone bar in a laboratory (Boutin and Besnainou2008). From these experiments and others, it hasbecome clear that it is especially challenging to dampvibrations or cancel sound. However, damping byfeedback control can be achieved by delivering feed-back as fast as possible, employing sensors andactuators that are collocated (i.e. physically matchedand placed at the same point) and programming thefeedback controller to behave physically like a vis-cous damper (Berdahl 2009: 30, 27–9 and 34–71). Insome previous experiments, piezoelectric actuatorswere employed, which required dangerously largevoltage levels for sufficient actuation. While electro-magnetic actuation can solve the safety problem,these challenges combined explain why there has beenno long history of performance practice with acousticmusical instruments subject to feedback damping.

1.3.2. Inducing self-sustaining oscillations byfeedback control

In contrast, inducing self-sustaining oscillations byfeedback control is technically much easier – typicallyany kind of acoustic feedback with an acoustic delaycausing the actuation and sensing signals to be out ofphase results in self-oscillation for sufficiently largeloop gains. This is the kind of feedback that mostperformers are familiar with as it causes the howlingor squealing effect that affects the performance ofpublic address systems and electric guitars. In addi-tion to its ease of implementation, performers areoften excited by the idea of self-oscillation in thatmusical instruments can be caused to vibrate bythemselves or exhibit infinite sustain. In this case,performers no longer need to supply the energy tomake the instrument produce sound, rather they cansteer energy from external sources. As a consequenceof all of these facts, there is a performance history ofthis kind of feedback control, albeit non-subtle innature. Undoubtedly it will not be possible to noteall prior work, as there are now even commercialproducts for sustaining strings. The E-Bow is a small,portable and electromagnetic virtual bow that can becoupled to a real vibrating guitar string when helddirectly over the string (Heet 1978), and the Sustai-niac is a virtual bow that is mounted in place of oneof the pickups of an electric guitar and constantlyelectromagnetically bows the strings to set them intoself-oscillation unless damped by the hand of themusician (Hoover 2000). Finally, the Moog Guitar,1

released in 2008, incorporates a magnetic feedbacksystem to sustain or damp the strings while playing.

1http://www.moogmusic.com/moogguitar (accessed on 28 October2010).

156 Dan Overholt, Edgar Berdahl and Robert Hamilton

Here we describe some of the notable more aca-demically oriented compositions involving feedbackself-oscillations. As early as 1974, Nic Collins createdfeedback loops in a more generalised ‘musicalinstrument’ using a microphone and a loudspeaker ina room. He writes of Pea Soup that ‘a self-stabilizingnetwork of phase shifters nudges the pitch of audiofeedback to a different resonant frequency every timefeedback starts to build, replacing the familiar shriekwith unstable patterns of hollow tones – a site-specificraga reflecting the acoustical personality of the room’(Collins 2004). Also employing complicated acousticfeedback, Agostino Di Scipio regulated the gain of aroom acoustic feedback system in the Feedback Studyportion of Audible Ecosystemics (Anderson 2005),and Steve Reich’s Pendulum Music was a compellingexample of the use of acoustic feedback in perfor-mance, where metrically phasing tones were createdthrough swinging microphones hung above loud-speakers (Reich 1974: 12–13).More recently, the elaborate Magnetic Resonator

Piano allowed up to fifteen strings to be actuatedsimultaneously. Although only one sensor wasemployed, banks of bandpass filters allowed formultiple sensing signals to be estimated from thesingle sensor measurement. Consequently, fifteenstrings could be electromagnetically sustained some-what independently of one another. The piecesSecrets of Antikythera for a single pianist andd’Amore, in which piano string vibrations were sus-tained to create a ‘harmonic glow’ surrounding thepitches of a viola solo, were performed using theMagnetic Resonator Piano in 2009 (McPherson andKim 2010). These works helped extend previous workby Per Bloland for the more portable Electro-magnetically Prepared Piano (Berdahl et al. 2005;Bloland 2007). Indeed, piano design has evolved overcenturies into an instrument that many people believesounds good, and we believe that sound synthesisedby electromagnetically actuated piano strings conse-quently also tends to sound good. In addition, thestrings’ resonant character promotes new performerinteractions as any energy injected into the strings canrequire multiple seconds to decay back to silence.Feedback systems for inducing self-sustaining

oscillations can either be relatively simple orremarkably complex. As with formally chaotic sys-tems, small changes in the parameters can result inlarge changes in the sound. Work with more complexacoustic feedback systems includes Simon Waters’concert flute modified for feedback incorporatingDSP, Stef Edwards’ Radio Pieces, which involvesfeedback to a live radio show through listeners’ tel-ephones placed near their radios, and John Bowers’contributions to drum, long string, electric guitar androom-based feedback instruments, which are alldescribed by Waters (Waters 2007: 1–20).

1.4. Postulations on performative ecosystems

Symbiosism (van Driem 2004) is a theory that looksat language as an organism residing in the humanbrain – language is then recognised as a memeticentity. As humans, we form and propagate languagethrough our speech and writing, while language itselffurnishes the conceptual universe that informs andshapes our thinking. Given that many aspects ofmusical expression can be viewed in this same man-ner, it is interesting to view the characteristics ofmusic itself, along with the tools we use to makemusic (instruments), as being in a similar symbioticrelationship. This parallel can be useful in a con-ceptual exploration of how every musical instrument –from the stone age drum to the laptop computer – isa piece of technology that simultaneously drives andlimits the creative process. Feedback systems havethus always played a role at many different levels andtime spans, and music performance has adapted todynamic cultural environments, showing emergentqualities stemming from the interpenetration of per-former’s ideas, tools, audiences and environments –the entire ecology of musical creation. Actuatedmusical instruments bring to the table programmablephysical behaviours and direct or immediate tactilefeedback from things that were previously virtual-only musical interactions.

Music lives through our minds and bodies, and isaffected by the technologies we use to make and shareit, as well as the environments in which we play it.Hence, it is subject to the strengths and weaknesses ofany technologies in use, including sound-producingmechanisms of all types, notation and compositionalsystems, musical software algorithms, the physicalinterfaces and the real or virtual elements of electro-nically enhanced instruments. It is the interminglingof all of these that are used to create music within aperformative ecosystem. While some may conjecturethat digital technologies today provide such flexibilitythat there is no further need for advancements intraditional or physical musical-instrument designs, itwould surely be unwise to let digital technologies bethe only focus of future developments. They certainlyhave a strong appeal due to their abstract nature andprogrammability, but other technologies (electro-mechanical, materials, acoustics, haptics, etc.) unde-niably offer immediate or tangible benefits thatcomputers alone cannot. The combination of elec-tronic and acoustic elements in a single instrument,some of which are exemplified in the authors’ workhere, show the field rich with possibilities. As it is notyet fully explored, the area is still ripe for exploration,and it is this motivation that has led to the examplesof new actuated musical instruments seen in theremainder of this article – all of which allow gesturalcontrol over dynamic performance ecologies. This is

Advancements in Actuated Musical Instruments 157

not to say that the entire musical performance eco-system (the interpenetration of a musician’s ideas,tools, audiences and environments) can be embodiedin any single instrument, but, rather, that the tech-nology-enhanced instruments discussed herein havebuilt-in performative feedback loops that we hopemay encourage musicians to engage with an ecolo-gical approach to music-making.

2. ACTUATED INSTRUMENTS ANDCOMPOSITIONS

Actuated musical instruments inherit a pre-existingtradition of musical performance practice as well aspioneer a novel and extensible practice of extendedand augmented performance technique. The musicalpossibilities of such instruments inhabit a space bothfamiliar and challenging, building on historicallyvalidated traditions while simultaneously breakingnew sonic ground and new paradigms of interactivity.

The classification of ‘actuated musical instrument’thus implies an idiomatic extension of performancetechnique, as the physical instrumental system itselfserves as the starting point for technology-aidedmodification: the set of performance techniques andinteraction methods traditionally associated with agiven instrument are generally extended withoutreduction. The adoption of adaptive algorithms couldeven, for example, lead to instruments that changetheir response to player inputs over time according tousage patterns.

2.1. The Overtone Fiddle

An extension of the prior Overtone Violin (Overholt2005), the Overtone Fiddle by Dan Overholt is anelectronic violin that incorporates electronic sensors,integrated DSP and feedback actuation. An embed-ded tactile sound transducer (see Figure 3, lower left)vibrates the body of the Overtone Fiddle, allowingperformer control and sensation via traditional violintechnique, as well as extended playing techniquesincorporating shared man/machine control. A mag-netic pickup system is mounted to the end of thefiddle’s fingerboard in order to detect only the signalsfrom the vibrating strings, avoiding the vibrationsfrom the body of the instrument. This focused sensingapproach allows less restrained use of DSP-generatedfeedback signals, because there is less direct leakagefrom the actuator to the sensor.

Signals can be injected directly into the mainacoustic body of the instrument via a voice-coil typeof tactile transducer, as well as into a second acousticbody that hangs below the instrument. This lowerresonating body is made of extremely thin carbon-fibre and balsa wood – materials that would not bestrong enough to support the full string tension of

strings on the main body – thus allowing extremelyefficient transfer of acoustic energy from anotherembedded tactile transducer into this box (which isdesigned as a Helmholtz-resonator). The second bodyof the Overtone Fiddle can also be driven with DSP-generated feedback signals, or indeed any audio sig-nal the performer desires. The tactile transducersinside both the top and bottom resonating bodies aredriven by a battery-powered Class-T amplifier, andthe production of the audio signals with which todrive them is accomplished via real-time software onan iPod Touch mounted on the fiddle (the main bodyof the instrument is designed to accommodate this, asshown in Figure 1). This makes the entire instrumentself-contained, not including the bow and its corre-sponding electronic circuits.

The bow used with the Overtone Fiddle is custommade from a carbon-fibre rod that is lighter in weight(and longer) than a normal violin bow, in order toaccommodate the added mass of a small battery-powered sensor circuit based on the CUI322 (anelectronics prototyping board), along with a wireless802.11 g radio module, and an absolute orientationsensor. A simple BASIC-language program writtenon the CUI32 retrieves the orientation data from thesensor, and translates it into the Open Sound Control(OSC) protocol in order to send it to the iPod Touchon the instrument. The 802.11 g radio module isconfigured to broadcast its own ‘AdHoc’ base sta-tion, which is chosen in the setup of the networkpreferences in the WiFi settings of the iPod Touch,allowing the devices to communicate easily thereafter.

Software used with the Overtone Fiddle is generallywritten in SuperCollider3 (McCartney 2002), PureData

Figure 3. Top: Overtone Fiddle with optional carbon-fibrelower resonating body; bottom: detail of embedded electro-

magnetic actuator and string pickups.

2http://code.google.com/p/cui32 (accessed on 28 October 2010).3http://supercollider.sourceforge.net (accessed on 28 October2010).

158 Dan Overholt, Edgar Berdahl and Robert Hamilton

(using libpd or RjDj4), or MoMu5 the Mobile MusicToolkit (Bryan, Herrera, Oh and Wang 2010), all ofwhich can run in real-time on the iPod Touch. Theinstrument thereby incorporates all of the flexibilityof modern digital signal processing techniques – forexample, altering the timbral structure of the soundbeing produced in response to player input. Sincethese algorithms can be controlled through gesturalinteraction using both the motion sensors in the iPod(accelerometers, gyroscopes, etc.) and the orientationsensor on the bow, the Overtone Fiddle promotesnew performer interactions beyond those supportedby traditional acoustic violins. For instance, in animprovisational performance by the authors (seen inthe video accompanying this article: Movie example 1),6

the timbre of the instrument’s sound is made to changedramatically when bowed rapidly.While the sound quality of a traditional acoustic

instrument is fixed by its physical design, actuatedmusical instruments can malleably simulate even phy-sically impossible acoustic properties, such as a violinwith hundreds or thousands of sympathetic strings. Forthe performer and the audience, however, the completesound produced by an actuated musical instrumentsuch as the Overtone Fiddle is perceptually contiguous,as both the natural acoustic and the digitally processedwaveforms are ‘folded together’ – both producedlocally. The music is generated through immediateperformative gestures on the instrument, and musicianscan use the embedded technology to produce interestingtimbres that might never have been heard before.Nonetheless, the internal actuation is caused by exci-tation from digitally processed signals, so this aspect ofthis sound is due to virtual computerised agents. Theinstrument can also make use of wireless networks toincorporate itself into a wider ecology of musicalendeavours, enabling shared instrumental controlschemes, as previously mentioned in section 1.1.Algorithms are used to adjust the body vibrations of

the acoustic part of the instrument, actively changingthe harmonics (overtones) heard in the musical tonequality. Consequently, the acoustic properties of theinstruments are adjustable, instead of being perma-nently defined by the woodworking skills of a luthier.In other words, the force feedback from the actuatorcan cause the wooden body to exhibit new dynamicbehaviours, and it is this programmable force feedbackthat distinguishes the Overtone Fiddle from priorinstruments such as the Variax from Line 6 (Celi,Doidic, Fruehling and Ryle 2004) and the ChameleonGuitar (Zoran and Maes 2008). This feedback actua-tion promotes the continuing evolution of musical

instruments today, and aims to see what might resultfrom the combination of some of today’s mostadvanced technologies with traditional instrument-making and musical skill and practice. It is hoped thatsuch combinations, by enabling performative feedbackloops, will encourage performers more often to engagewith the entire musical performance ecosystem whenincorporating such technology-enhanced instrumentsinto their musical practices.

2.2. The Feedback Resonance Guitar

In the case of the Feedback Resonance Guitar byEdgar Berdahl (see Figure 4), audio signals routedinto the instrument’s dual embedded electromagnetsdrive the activation and subsequent resonance ofgiven frequencies on each of the six strings of theguitar. By varying the source of input – using sourcessuch as pre-recorded or computer-mediated electro-nic signals, live third-party instrumental audiostreams, or the guitar’s own output – significantlydifferent practices of instrumental performativepractice become possible, including self-sustainingoscillations using acoustic feedback, exciting instru-ment resonances by a computer-mediated system,and the control of and communication between athird-party instrument and the guitar itself.

Figure 4. The Feedback Resonance Guitar, played by RobertHamilton.

4http://www.rjdj.me (accessed on 28 October 2010).5http://momu.stanford.edu/toolkit (accessed on 28 October 2010).6http://ccrma.stanford.edu/, eberdahl/AMI/video/EveningActuated.m4v (accessed on 28 October 2010).

Advancements in Actuated Musical Instruments 159

2.2.1. Inducing self-sustaining oscillations

As described in Section 1.3.2., the artificial extensionof a guitar string’s resonance or the sustain of a givennote on the instrument can be achieved by feedingoutput from the guitar’s pickups into the electro-magnetic resonators driving the excitation of theinstrument. Through gating and limiting of the exci-tation signal, a performer can control the amount ofsustain generated by the instrument, evokingresponses that can range from a subtle lengthening ofa given plucked note to a fast and powerful self-oscillating response. A note may be sustained for anarbitrarily long period of time, enabling the perfor-mer to steer the actuated energy – a musical inter-action that would otherwise require some otherexternal energy source.Highly amplified self-oscillating ‘feedback’ tones

hold an idiomatic place in the historical performancepractice of electric guitars, with rock-based examplesranging from early Beatles recordings (Emerick andMassey 2006) to the iconic controlled-chaos of JimiHendrix’s live performance techniques. In traditionalelectric guitar systems, the unique sonic qualities ofthese tones are achievable through the completion ofa complicated analogue feedback loop, most com-monly created with high-volume loudspeaker outputsbeing transmitted through the air, intro the instru-ment’s body, through the strings and subsequentlyinto the pickups. As such, a low-volume recreation ofquickly growing self-oscillation tones in live perfor-mance has always proven difficult to achieve for liveelectric guitar performance.Feedback loops created by routing output from

various combinations of the Feedback ResonanceGuitar’s seven acoustic pickups into its actuatorsallow for the generation of compelling feedbacktimbres with no correlation to the performancesystem’s output volume. In this fashion, the levelsof input signal driving the feedback system canbe throttled through the use of digital or analoguelimiters in the signal path, creating a highly con-trollable feedback performance system capable ofsubtle performance control (Berdahl and Smith2006).In live performance, the Feedback Resonance

Guitar has been used as a self-supplying feedbacksystem in performances of yof giants by RobertHamilton (Sound example 1), The Metaphysics ofNotation by Mark Applebaum and Siren Cloud byChris Chafe. In these compositions, the combinationof induced self-oscillations in combination withmulti-channel output and spatialisation has beenused to create rich timbres and textures that are at thesame time idiomatic to the guitar and sufficientlynovel to provoke questions and commentary fromguitarists and audience members alike.

2.2.2. Computer-mediated actuation

Audio signals generated by interactive and reactivereal-time computer software systems under algorithmicor performer-mediated control enhance active physicalinstrumental systems by granting them access to prop-erties and functionalities normally associated withcomputer systems. Since actuated musical instrumentstransform the actuation signal into a physical responsethrough the vibration of notes on a set of strings,membranes or the instrument’s body, a tangible inter-action exists between the performer and the ‘virtual’computer system. For example, audio output from acomputer system fed into the actuated fiddle or guitarcan cause the strings to vibrate, a physical reactionclearly felt by the performer’s hands. Without control-ling the performer’s instrument or diminishing his orher inherent choices in adjusting the length of thestrings, such a haptic interaction instead indicatesthrough a tangible interface to the performer that agiven pitch location is at that moment a region ofinterest for the virtual computer system – the performerhears the actuated notes; however, the actuated noteswill be quiet if they do not correspond to the perfor-mer’s currently selected string lengths. The choice ofwhether or not to follow this suggestion is left to theperformer, and the communication occurs at a muchmore intimate level between the performer and thecomputer-mediated system than simply hearing normalaudio output from loudspeakers attached to the system.Once the computer-mediated actuation becomes suf-

ficiently complex, it can be helpful to have a convenientinterface such as an iPod touch to adjust its parameters.As a brief aside, let us consider the Overtone Fiddle, forwhich soundscape actuation signals have been imple-mented. One useful iPod mapping allows the performerto press the screen to capture a sound sample from thestrings. This sound sample is then applied as anactuation signal, in short harmonising drone segmentsas a background over which the performer can continueto play the fiddle. Finally, another mapping generatessemi-autonomous percussion sounds, while simulta-neously processing the sound of the strings in real-timeusing a band-pass filter with a cut-off frequency that isdependent upon the tilt angle of the violin.Another use of computer-mediated sound as input

can be found in an interactive iPhone-driven controlpatch used for improvisation with the FeedbackResonance Guitar, wherein a bank of five oscillatorsis routed into the guitar, the pitch and relativeamplitude of each being controlled in real-time by aniPhone Touch interface mounted on the guitar bodyitself (see Figure 5). In this scenario, touch locationson the lengthwise-axis of the multitouch display aremapped to oscillator frequency, while motion on thewidthwise axis is mapped to oscillator amplitude.Individual oscillators can be enabled or disabled

160 Dan Overholt, Edgar Berdahl and Robert Hamilton

through a bank of toggle controls on the touch screenitself. In this manner, a performer can easily andidiomatically sweep through harmonic frequencies ofthe guitar with one hand, while fretting desired pit-ches with the other hand.

2.2.3. Live audio signals

In Sections 2.2 and 2.3 the instrument designs fit theaugmented reality paradigm because the musicalinstruments are real, but their capabilities are aug-mented by coupling to some virtual elements viafeedback control. Now we present a new applicationof the Feedback Resonance Guitar, which subscribesto the augmented virtuality paradigm. As before, theactuators on the Feedback Resonator Guitar serve asportholes to a virtual environment; however, in thisapplication, elements in the virtual environment areavatars representing other real objects.Consider employing musical output from live

acoustic or electronic instrumental performers asactuation signals to an acoustic musical instrument.This kind of acoustical connection can create a newlevel of haptic ensemble communication during amusical performance. This actuated sound made

tangible occurs in parallel to traditional methods ofintra-ensemble communication (visual and auralcues) or replaces traditional communications insituations where such communication is difficult orimpossible such as telematic or networked musicalperformance. In this manner, rich musical ecosystemsbased on shared instrumental control are created,allowing musical communication based on hapticexcitation to drive novel emergent musical ideas andbehaviours. This kind of haptic communicationthrough tangible sound enables musicians to interactthrough the haptic modality, which we believe has sofar been underutilised in the field.For instance, in yof giants for double bass and

Feedback Resonance Guitar by Robert Hamilton(Sound example 1), cues for half-step key-switches inan improvisational structure are triggered by theplaying of low cue notes on the double bass, which arein turn routed into the Feedback Resonance Guitar.When a cue is received by the guitar, not only arenodes representing the root (i.e. first octave overtone)suddenly expressive and heavily weighted areas onwhich to play the guitar, but each node representing anovertone of that cue pitch too suddenly becomesheavily weighted. While only one instrument in thisduo is currently an actuated instrument, even suchone-sided co-control of an actuated instrument createsan exciting ecosystemic musical space, wherein theperformers are connected through their shared controlof the Feedback Resonance guitar.In the work six in one hand for seven guitars and six

network performers, also by Robert Hamilton, liveaudio signals are routed into the Feedback ResonanceGuitar from six guitarists located in various locationsaround the world (Figure 6), sending individual audiooutputs into the guitar across low-latency high-band-width network connections (Caceres and Chafe 2008).Output from remote guitars is visualised within a fullyrendered three-dimensional environment and manipu-lated in real-time by virtual performers interacting with3D artefacts in the virtual space (Hamilton 2008, 2010).By essentially filtering audio input and musical gesturesfrom seven performers through the musical lens of theFeedback Resonance Guitar, with subsequent manip-ulation by virtual performers in a virtual space, thecomplex musical ecosystem realised within the instru-ment itself is made visible and tangible. Such feedbackbetween performers would be practically impossible orcertainly impractical without actuation, yet it enablesthe performers to engage in new kinds of musicalinteractions.

2.3. Teleoperation with Robothands

The prior section demonstrated how the FeedbackResonance Guitar can serve as a porthole into avirtual environment, for both augmented reality and

Figure 5. iPhone control interface mounted on the FeedbackResonance Guitar.

Advancements in Actuated Musical Instruments 161

augmented virtuality applications. Here we considerconstructing a porthole into a virtual musical envir-onment that is so accurate, it is like a hand. For this,we use a general haptic force-feedback device, such asthe three degree-of-freedom (3-DOF) NovInt Falconshown in Figure 7. However, the virtual world is a bigplace! In other words, the three-dimensional space isexpansive, and the performer must explore it in orderto find virtual objects with which to interact. Creatinghigh-fidelity virtual worlds with virtual objects tofacilitate efficient exploration requires great attentionto programming detail (Cadoz, Luciani and Florens1981 and 1993). For this reason, we should study howto place objects in the virtual world with reducedprogramming effort.

We therefore suggest creating avatars in virtualenvironments that are connected to real objects

instead of programmed. For instance, see the slave‘robothand’ shown in Figure 7 (left) that is holdingan acoustic shaker. When a (stiff) virtual spring isconnected between the master and a slave robot invirtual space, the performer can remotely play theshaker through the robothand while receiving forcefeedback from the shaker (Berdahl, Smith and Nie-meyer 2010).

We find it striking to observe percussion instru-ments controlled by robots that move as if they werebeing played by a real human performer makingessential as well as ancillary gestures. The sound islifelike, and perhaps even more importantly it istangible. Furthermore, while the percussion instru-ments are indeed actuated by robots, the performercan touch the instruments to interact with themdirectly if they are placed next to the performer. Forinstance, by holding onto one of the percussioninstruments, the human performer can change themechanical load seen by the robot, so the robot willcause the percussion instrument to move in a filteredbut still somehow lifelike manner.

One intriguing question is, ‘How can one perfor-mer control distinct musical parts on multiple slaverobothands simultaneously?’ One could employ arti-ficial intelligence algorithms to cause the slaverobothands to behave semi-autonomously, but we aremore interested in intimate control musical instru-ments. Therefore, in the piece Edgar Robothands:When The Robots Get Loose, Berdahl employs atechnique that he terms mechanical looping to controlthe tambourine, snare drum and two shakers by wayof four slave robots, as shown in Figure 8. During

Figure 6. Conceptual schematic of Robert Hamilton’s six in one hand.

Figure 7. Left, a slave robothand holding an acousticmusical shaker, and right, a performer interacting with the

master NovInt Falcon force-feedback device.

162 Dan Overholt, Edgar Berdahl and Robert Hamilton

teleoperation, the trajectory of the gestures made byBerdahl into the master robot are recorded intowavetable loops in Max/MSP7 using the HSP toolbox(Berdahl, Kontogeorgakopoulos and Overholt 2010).At later times during the piece, these loops are playedback into the slaves through virtual springs, provid-ing for mechanical (real world) synthesis with anespecially human feel, even when the loops are playedback irregularly or at superhuman speeds. An audiorecording of the piece as performed on 17 February2010 for the CCRMA Winter Concert (Soundexample 2) demonstrates that robots can performactions with repeatability, accuracy and speed thatmay be unparalleled by human users, creating phy-sical and lifelike sounds that are nevertheless new andthat were not possible before. The net result is co-control of the percussion instruments via ecosystemicperformance that would not be the same withouteither the performer or the Robothands.

2.4. Haptic Drum

The last musical instrument in this paper fits withinthe augmented reality paradigm, and is a percussioninstrument that applies feedback control to changedrum-roll gestures. When a drumstick is droppedupon a normal drum, it typically bounces two toseveral times before coming to rest due to frictionalforces. Percussionists use this bouncing action to playdrum rolls on traditional percussion instruments atrates of up to 30Hz (Hajian, Sanchez and Howe1997: 2294–9), even though humans cannot move thehand back and forth unassisted at such fast rates. TheHaptic Drum explores the possibilities that arisewhen friction is partially counteracted by actuation.For example, the Haptic Drum prototype v1 being

played in Figure 9 consists primarily of a drum padattached to the cone of a woofer by a lightweight,cardboard cylinder. Every time that a stick strikes thedrum pad, a pulse is sent to the loudspeaker, whichadds energy to the drumstick vibrations. As a con-sequence, the Haptic Drum can enable the performerto make drum-roll gestures that would otherwise bedifficult or impossible, such as drum rolls at fast ratessuch as 70Hz with a single drumstick (Berdahl 2009:153–74). These rates are so fast that the drum rollbegins to sound pitched at the drum-roll rate.

The pulse height, which is constant, is selected bythe position of the top knob on the Haptic Drum (seeFigure 9). The knob in the middle controls the shapeof the loudspeaker pulses, which is only of minorimportance, and the third knob controls thedebounce threshold for the impact detector. Byreducing the debounce threshold far enough, it ispossible to cause the Haptic Drum to emit multiplepulses for each drumstick impact, or the system caneven become unstable, depending on how the per-former manipulates the drumstick. However, theenergy in the system does not grow without boundsbecause the pulse height is constant (Berdahl 2009:153–74). This feature enables the performer to steerthe drumstick vibrational energy provided by theHaptic Drum power source to create new gesturesand patterns. In particular, when the performeremploys two drumsticks simultaneously, it becomespossible to obtain complex bifurcation phenomena asthe performer modulates the stiffness and angle of hisor her hands. An online video demonstration isavailable that illustrates the most basic new gesturesand musical interactions that are enabled by theHaptic Drum.8

A sound synthesiser can be connected so that adrum sample is triggered each time the drumstick

Figure 8. Edgar Robothands: When The Robots Get Loose.

Figure 9. Berdahl playing Haptic Drum prototype v1 (left)using two drumsticks simultaneously.

7http://cycling74.com (accessed on 28 October 2010).

8https://www.youtube.com/user/eberdahl810#p/a/u/1/Yn0CQnl0PEQ(accessed on 28 October 2010).

Advancements in Actuated Musical Instruments 163

strikes the Haptic Drum. In our opinion, the resultingsynthesised sound is perceived as physical becauseeach triggered sound sample is due to a real drum-stick striking a real drum pad. However, a snaredrum sample played back at 70Hz may sound bizarreto the Western ear – it may even be perceived as amachine gun sound. For this reason, Berdahl oftenemploys non-Western drum samples.

Multiple pieces have been written for the HapticDrum. For example, It’s Like A Car9 (excerpt inSound example 3) is a recording of a live improvi-sation by Edgar Berdahl and Frankie Siragusainvolving the Haptic Drum and traditional percus-sion instruments including drum set, maracas, xylo-phone, egg and FM radio. It was premiered at theCCRMA Transitions Concert at Stanford Universityin September 2009. A similar approach is taken in thepiece Sike!,10 which incorporates the Haptic Drum,the Feedback Resonator Guitar and NovInt Falcon-controlled harmony synthesiser voices, as well as anacoustic drum set.

3. CONCLUSION

Actuated musical instruments produce sound viavibrating element(s) that are co-manipulated byhumans and electromechanical systems. We believethat effectively employed actuation can enable theperformer to interact in new ways. For instance,actuation can provide energy from an external sourcethat the performer can feel and steer. More generally,actuation can make virtual qualities tangible in phy-sical instruments, allowing designs that can free upsome cognitive bandwidth. During musical perfor-mance, shared control of actuated instruments allowsfor the emergence of novel musical behaviours, asrich polyvalent control schema evolve into complexmusical ecosystems, generated and sustained by thecomplex interactions between interconnected perfor-mers and instrumental systems alike.

Towards these ends we have described some of ourongoing explorations into the world of actuatedmusical instruments, including the development ofnew instruments and the performative ecosystemsthey bring about. While some of these instrumentsare so new (especially the Overtone Fiddle) that theyhave not been used in a large number of perfor-mances as yet, we have portrayed them in a recentensemble performance An Evening of ActuatedMusical Instruments,11 a trio performance with theFeedback Resonance Guitar, the Haptic Drum and

the Overtone Fiddle (see Figure 10). We have alsostarted an online presence ‘Actuated Musical Instru-ments Guild’ at www.actuatedinstruments.com thatwe hope will serve as a community hub where relatedwork can be shared.

We postulate that the two most promising direc-tions coming out of this research might be:

1) a revival in the fairly stagnant evolution oftraditional instruments; heading towards a ‘newrenaissance’ of instrument development focusedon actuated instruments and their related dis-course; and

2) further advances in the area of teleoperation ofacoustic instruments, telematic performance andthe expanded performance ecosystems these typesof approaches enable.

We feel that it is especially advantageous to have thesound localised with the action, and also good for theinstrument to be untethered while performing andeasily portable. Future actuated musical instrumentswill tightly incorporate sound elements coming fromembedded signal processing, and enable more inti-mate control of the combination of digital algorithmswith the natural physics of interaction. They willaccomplish this by bringing digital behaviours outinto the real world, allowing them to be used asmusical inflections via gestural interaction. In sodoing, they will enhance the expressivity of electronicand acoustic instruments alike, opening up newdirections in music.

REFERENCES

Anderson, C. 2005. Dynamic Networks of Sonic Interac-tions: An Interview with Agostino Di Scipio. ComputerMusic Journal 29(3) (Fall): 11–28.

Berdahl, E. 2009. Applications of Feedback Control to MusicalInstrument Design, PhD thesis, Stanford University.Available online: http://ccrma.stanford.edu/,eberdahl/berdahl-thesis.pdf (accessed on 28 October 2010).

Berdahl, E. and Smith, J., III. 2006. Some Physical AudioEffects. 9th International Conference on Digital Audio

Figure 10. Berdahl, Overholt and Hamilton playing theHaptic Drum, Overtone Fiddle and Feedback ResonanceGuitar in An Evening of Actuated Musical Instruments.

9https://ccrma.stanford.edu/,eberdahl/CompMusic/ItsLikeACar.mp3(accessed on 28 October 2010).10https://ccrma.stanford.edu/,eberdahl/CompMusic/Sike!.mp3(accessed on 28 October 2010).11http://ccrma.stanford.edu/,eberdahl/CompMusic/Evening.m4v(accessed on 28 October 2010).

164 Dan Overholt, Edgar Berdahl and Robert Hamilton

Effects (DAFx), Montreal, Canada, 16–20 September,pp. 165–8.

Berdahl, E., Backer, S. and Smith, J., III. 2005. If I Had AHammer: Design and Theory of an Electromagneti-cally Prepared Piano. Proceedings of the InternationalComputer Music Conference, Barcelona, Spain, 5–9September.

Berdahl, E., Kontogeorgakopoulos, A. and Overholt, D.2010. HSP v2: Haptic Signal Processing with Extensionsfor Physical Modeling. Proceedings of the Haptic AudioInteraction Design Conference, Copenhagen, Denmark,16–17 September, pp. 61–2.

Berdahl, E., Smith, J., III and Niemeyer G. 2010.Mechanical Sound Synthesis: And the New Applicationof Force-Feedback Teleoperation of Acoustic MusicalInstruments. Proceedings of the 13th InternationalConference on Digital Audio Effects (DAFx–10), Graz,Austria, 6–10 September.

Besnainou, C. 1999. Transforming the Voice of MusicalInstruments by Active Control of the Sound Radiation.Proceedings of the International Symposium on ActiveNoise and Vibration Control, Fort Lauderdale, USA.

Bloland, P. 2007. The Electromagnetically Prepared Pianoand its Compositional Implications. Proceedings of theInternational Computer Music Conference (ICMC),Copenhagen, Denmark, pp. 125–8.

Boutin, H. and Besnainou, C. 2008. Physical Parameters ofan Oscillator Changed by Active Control: Applicationto a Xylophone Bar. Proceedings of the 11th Interna-tional Conference on Digital Audio Effects, Espoo, Fin-land, 1–4 September.

Bryan, N.J., Herrera, J., Oh, J. and Wang, G. 2010. MoMu:A Mobile Music Toolkit. Proceedings of the Interna-tional Conference on New Interfaces for MusicalExpression. Sydney, Australia.

Caceres, J.-P. and Chafe, C. 2009. JackTrip: Under theHood of an Engine for Network Audio. Proceedings ofInternational Computer Music Conference, Montreal,Canada.

Cadoz, C., Luciani, A. and Florens, J.-L. 1981. Synthesemusicale par simulation des mecanismes instrumentaux.Revue d’acoustique 59: 279–92.

Cadoz, C., Luciani, A. and Florens, J.-L. 1993. CORDIS-ANIMA: A Modeling and Simulation System for Soundand Image Synthesis – The General Formalism. Com-puter Music Journal 17(1) (Spring): 19–29.

Celi, P., Doidic, M., Fruehling, D. and Ryle, M. 2004.Stringed Instrument with Embedded DSP Modeling. USPatent No. 6,787,690.

Collins, N. 2004. Pea Soup. New York: Apestaartje, CD.Di Scipio, A. 2003. ‘Sound is the Interface’: From Inter-

active to Ecosystemic Signal Processing. OrganisedSound 8(3): 269–77.

Emerick, G. and Massey, H. 2007. Here, There andEverywhere: My Life Recording the Music of the Beatles.New York: Gotham Books.

Guerard, J. and Boutillon, X. 1997. Real Time Acoustic WaveSeparation in a Tube. Proceedings of the InternationalSymposium on Musical Acoustics, Edinburgh, UK, 19–22August, pp. 451–6.

Hajian, A., Sanchez, D. and Howe, R. 1997. Drum Roll:Increasing Band-width through Passive ImpedanceModulation. Proceedings of the IEEE InternationalConference on Robotics and Automation, Albuquerque,USA, 20–25 April, pp. 2294–9.

Hamilton, R. 2008. q3osc: or How I Learned to StopWorrying and Love the Game. Proceedings of theInternational Computer Music Association Conference,Belfast, UK.

Hamilton, R. 2010. UDKOSC Development Wiki. http://ccrma.stanford.edu/wiki/UDKOSC (accessed in October2010).

Heet, G. 1978. String Instrument Vibration Initiator andSustainer. US Patent 4,075,921.

Hoover, A. 2000. Controls for Musical Instrument Sustai-ners. US Patent 6,034,316.

McCartney, J. 2002. Rethinking the Computer MusicLanguage: SuperCollider. Computer Music Journal26(4): 61–8.

McPherson, A. and Kim, Y. 2010. Augmenting theAcoustic Piano with Electromagnetic String Actuationand Continuous Key Position Sensing. Proceedings ofthe 2010 International Conference on New Interfaces forMusical Expression (NIME 2010), Sydney, Australia.

Overholt, D. 2005. The Overtone Violin: A New ComputerMusic Instrument. Proceedings of the InternationalComputer Music Conference (ICMC 2005), Barcelona,Spain, 5–9 September.

Reich, S. 1974. Writings About Music. Nova Scotia: Pressof the Nova Scotia College of Art and Design.

Sayer, T. 2007. Using Technology to Ride the Reflex/Volitional Continuum in Improvised Musical Perfor-mance. Proceedings of the 2007 International ComputerMusic Conference. San Francisco: International Com-puter Music Association.

van Driem, G. 2004. Language as Organism: A BriefIntroduction to the Leiden Theory of Language Evolu-tion. In Ying-chin Lin, Fang-min Hsu, Chun-chih Lee,Jackson T.-S. Sun, Hsiu-fang Yang and Dah-ah Ho(eds.), Studies on Sino-Tibetan Languages: Papers inHonor of Professor Hwang-cherng Gong on his SeventiethBirthday (Language and Linguistics Monograph SeriesW–4). Taipei: Institute of Linguistics, Academia Sinica,pp. 1–9.

Verplank, B. 2003. Interaction Design Sketchbook. Classlecture for Music 250A, Stanford University, availableonline: http://ccrma.stanford.edu/courses/250a/lectures/IDSketchbook.pdf (accessed on 28 October 2010).

Waters, S. 2007. Performance Ecosystems: EcologicalApproaches to Musical Interaction. Proceedings of theElectroacoustic Music Studies Network. Leicester: DeMontfort, pp. 1–20.

Weinreich, G. and Causse, R. 1986. Digital and AnalogueBows: Hybrid Mechanical-Electrical Systems. Proceed-ings of the IEEE International Conference on Acoustics,Speech, and Signal Processing, vol. 11, pp. 1297–9.

Zoran, A. and Maes, P. 2008. Considering Virtual andPhysical Aspects in Acoustic Guitar Design. Proceedingsof New Instruments for Musical Expression (NIME)Conference, Genova, Italy, 5–7 June.

Advancements in Actuated Musical Instruments 165