Embed Size (px)
Citation preview
Instant Instrument Anywhere: A Self-Contained CapacitiveSynthesizer
David B. GerhardUniversity of Regina
3737 Wascana ParkwayRegina, Saskatchewan, Canada
Brett ParkUniversity of Regina
3737 Wascana ParkwayRegina, Saskatchewan, Canada
ABSTRACTThe Instant Instrument Anywhere (IIA) is a small devicewhich can be attached to any metal object to create anelectronic instrument. The device uses capacitive sensingto detect proximity of the player’s body to the metal ob-ject, and sound is generated through a surface transducerwhich can be attached to any flat surface. Because the ca-pacitive sensor can be any shape or size, absolute capacitivethresholding is not possible since the baseline capacitancewill change. Instead, we use a differential-based movingsum threshold which can rapidly adjust to changes in theenvironment or be re-calibrated to a new metal object. Weshow that this dynamic threshold is effective in rejectingenvironmental noise and rapidly adapting to new objects.We also present details for constructing Instant InstrumentsAnywhere, including using smartphone as the synthesis en-gine and power supply.
KeywordsCapacitive Sensing, Arduino
1. INTRODUCTIONInstant Instrument Anywhere (IIA) is a small electronicdevice which can be connected to almost any metal ob-ject and attached to any surface, turning that object intoan electronic musical instrument with the surface as thespeaker. IIA is inspired by the “LED throwie” [3], a small,low-cost, battery-powered magnetic LED used for graffitiand street art. The idea of being able to augment any sur-face with coloured light is taken one step further with theIIA, which can augment any surface it is attached to in or-der to create music. IIA is also inspired by the youtubevideo entitled “Homemade Synthesizer - KITCHEN MU-SIC1”, (Fig. 1a) where a composer seems to construct a syn-thesizer out of standard household objects and repurposedelectronics. The video author admitted that the video wasfake, but many of the objects he uses are plausible. Basedon his video, we first constructed a simple capacitive-sensingpaper-cup instrument2 (Fig. 1b) and after experimentingwith different materials and different ways of processing thesignals and turning them into music, we settled on a small
1http://www.youtube.com/watch?v=tCeYMpu3oXg2http://www.youtube.com/watch?v=dLHrn5v772E
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.NIME’12, May 21 – 23, 2012, University of Michigan, Ann Arbor.Copyright remains with the author(s).
Arduino-based system using a surface transducer, with al-ligator clips to attach capacitive sensing probes to nearbymetal objects.
Figure 1: (a) Kitchen music; and (b) our first IIA
The result is a musical instrument which is attached toa flat surface (which becomes the resonator for the trans-ducer), with probes from the instrument attached to a nearbymetal object (which becomes the capacitive sensor). Fig-ure 2 shows the exterior of the prototype IIA device.
(a)
(b)
(c)
(d) (e)
Figure 2: Prototype IIA device. (a) random metalobject; (b) Additional unused probes; (c) Surfacetransducer; (d) power supply jack; (e) USB portfor programming.
2. BACKGROUNDThe Arduino has become a staple of the hacker/maker com-munity, enabling the integration of embedded processinginto many standalone music projects over the last five years.Many opportunities exist for locally manipulating MIDI [1]and OSC data, and building new interfaces. Recently, theArduino platform has started using an ATmega8u2 (16u2in the Uno R3 version) as the USB interface. This advance-ment has opened up many new opportunities for MIDI con-nectivity (see Section 4).
2.1 Capacitive SensingElectrical field sensing has been a popular topic of investiga-tion in the music interface field for decades [8]. Capacitivesensing works by measuring the influence of the human bodyon a capacitive plate. A conductive surface or object canbe given an electrical charge which can be affected by theair and nonconductive surfaces around it. When a personbrings their hand close to the charged object, the storagecharacteristics of the object change and that change can bemeasured by an electrical circuit. Most capacitive sensingsystems use an additional nonconductive surface surround-ing the charged object (like glass on smartphones) that al-lows the user to touch the surface without discharging theobject. For this reason, there is a common misconceptionthat capacitive sensing requires contact, where in fact onlyproximity is required.
Capacitive touch systems have been used in music syn-thesis systems in a number of different instances beginningwith the Theremin. The Snyderphonics Manta [10] waspresented at NIME last year. It uses a touch-based iso-morphic hexagonal keyboard [6] similar to the MUSIX iOSsoftware3. Traditional instruments, such as a guitar [4] anda piano [7], have been augmented using capacitive sensorsto provide additional information such as touch location.New instruments like the T-Stick [5] use capacitive touchsensors as one type of sensor among many to create originalexperiences and sounds.
2.2 CircuitbendingA significant community has recently emerged around theconcept of circuit bending [2]. Circuit bending is the re-purposing of electronic toys, instruments, and other deviceswith the goal of creating new and interesting sounds. In asimilar manner, the IIA uses found metallic objects in orderto create new and interesting instruments.
3. PROTOTYPE CONSTRUCTION
VS1053
Metal Object 1 Arduino or
ATmega 32u4
CapSense Library
SurfaceTransducer
Surface
dC/dt andthreshold
make notesynth
Amplifier(LM358)
C
👋
MOk
out
in1
ink
Figure 3: System bock diagram
Figure 3 shows the block diagram of the prototype sys-tem as constructed. The core of the system is a micropro-cessor which is set up to detect changes in capacitance andtranslate them into note messages. Our prototype uses anArduino Uno with an ATmega328 microprocessor, but we
3http://www.shiverware.ca/musix.html
have also successfully deployed the codebase on a standaloneATmega32u4-based breakout board, which is less expensiveand has a smaller footprint than the Arduino Uno.
3.1 Sensor ConstructionBecause the goal of the IIA is to turn any metal object intoa capacitive sensor, a small circuit must be constructed toallow the capacitance to be measured by the CapSense li-brary4. A set of alligator clips are attached to I/O pins onthe microprocessor through a very large resistor (≈ 1MΩ).The large resistor means that the RC time constant of thecircuit is large, and can be measured in software using pulsesexpressed through the “output” pin. The “input” pin mea-sures the voltage on the metal object, which is the sameas the voltage across the capacitor. The time lag betweenthe voltage changes can be used to measure the RC timeconstant, and thus the capacitance of the metal object thatthe circuit is attached to.
3.2 Capacitive Sensing with CapSenseThe main use-case for the Instant Instrument Anywherewas to have a person approach a surface, attach the trans-ducer to the surface, clip probes on to a nearby piece ofmetal, and start to make sounds with no reconfigurationrequired. Traditional capacitive sensing requires the free-space capacitance of the sensor to be know beforehand andto use a threshold above that free-space capacitance to in-dicate a trigger event. Because we cannot know what metalthe IIA will be attached to, we cannot know the free-spacecapacitance and are not able to use an absolute threshold tocalculate triggers. Instead, we used a relative capacitancemeasure to detect how fast the capacitance was changing.
Figure 4 shows an example trace of the data coming fromthe CapSense library on the ATmega328 microprocessor.Note that C is not the measured capacitance of the system,but a time-based value related to RC.
The value of C relates to the time it takes to charge anddischarge the system. Each probe connected to a metal ob-ject is in series with a large resistor (1MΩ) which brings theRC time constant into a measurable range. The output pinprovides +5v to the system, which charges the RC circuit.When the input pin associated with the probe crosses thehysteresis input threshold of the ATmega328 (+3v rising,+2v dropping), the CapSense library measures the time fora complete 4-stage charge-discharge cycle to be performed:1: The circuit is charged past +3v on the input pin; 2:the input pin is set high, pinning the capacitor to +5v; 3:the input pin is disconnected from +5v and the capacitorbegins to discharge through the resistor; and 4: the inputpin is set low, fully discharging the capacitor to 0v. Thiscomplete cycle happens n times (in our case, n = 30) anda single measured C value is related to time to complete ncharge-discharge cycles. The charge/discharge time is actu-ally measured in number of CPU clock cycles multiplied bya looping factor which is determined empirically.
Referring again to Fig. 4, points a, c, d, and f are inter-preted as note-on events, while b and d are note-off events.Note that at point c, the environmental capacitance haschanged and a new baseline is present. A standard thresh-old would fail at this point, but our dynamic threshold con-tinues to accurately detect events and reject the backgroundnoise. Additionally, after point g, we can see a significantvariation in the background capacitance which is not causedby the player moving in the vicinity of the sensor. This vari-ation is too slow to cause an event, but a strict thresholdwould likely trigger on these values. The background capac-
4http://arduino.cc/playground/Main/CapSense
C
dCdt
dCdt
∑
Sample Number
a fd
4
b c ge
50100150200250
-100-500
50100150
-100-500
50100150
0 20 40 60 80 100 120 140 160
Figure 4: Processing of measured data. C = Mea-sured values from CapSense; dC/dt = differential;and
∑4(dC/dt) = moving four-tap sum of the differ-
ential, thresholded at ±50.
itance can change due to any number of factors includingstatic discharge, humidity, and people in the room.
In a second interaction mode (termed “hold”), a playercan start a note by quickly moving their hand toward themetal object, then slowly move their hand away preventinga release. If the player moves their hand slowly enough, thechange in capacitance will not surpass the change thresholdand the note will be sustained until triggered again.
3.3 Sound SynthesisThe sound synthesis module is the VS10535 sound decoderchip. This device, from VLSI solutions, contains decodersfor MP3, Ogg, AAC, and WMA as well as a bank of GeneralMIDI instruments for real-time synthesis. Our prototypeonly makes use of the General MIDI synthesis engine. Wedecided not to use an analog synthesis engine in order tokeep the size and power constrains down. The synthesizeroutput is routed to a HIAX25C10 8Ω frog transducer.
3.4 Constructed prototypeFigure 5 shows a photograph of the internal constructionof the prototype. SparkFun industries makes an Arduino“shield” that breaks out the VS1053 synthesizer chip andcontains some prototyping space, so we used that to buildthe amplifier circuit and attached a resistor for each of fourconnection probes.
3.4.1 Null PlateAllowing four connection probes meant that the prototypedevice could be connected to up to four metal objects. Wediscovered, however, that when the probes were not con-nected to a metal object, they continued to function as sen-
5http://www.vlsi.fi/en/products/vs1053.html
(c)(e)
(d)(b)(a)
Figure 5: IIA prototype internals. (a) Connectionprobes; (b) null plate; (c) VS1053 protoboard; (d)Resistors on the receive pins; (e) LM358-based am-plifier (wired to transducer) and headphone jack.
sors, causing unwanted touch events. In order to mitigatethis issue we added a copper plate to the outside of the de-vice, soldered to the output of the circuit pin. When not inuse, probes are connected to this “null” plate causing themto instantly charge and discharge, which indicates to thesoftware that they should be ignored without slowing downthe sensing algorithm.
3.4.2 Probe ShieldingA second consideration in creating the prototype itself isthat the wires used to attach to the metal objects do them-selves have capacitance, which can influence the result of themeasurement. This was discovered when playing multipleobjects which were near to each other: in some instances, ifthe wires connected to the objects were touching or tangled,multiple instruments would sound at the same time due tocrosstalk, or worse, false positives would appear.
out
in Cs Co
(a) (b)
Figure 6: Shielded probe. (a) shielded cable has aninherent capacitance between the core wire and theouter wrapping. (b) Grounding the outer wrappingcauses the capacitance of the shield CS to be in par-allel with the capacitance CO of the metal object.
By switching to shielded wires (see Fig. 6), the probewires are no longer reactive to environmental capacitanceand will not interact with each other or the player. Thiswas proven by measuring the reaction of the circuit with andwithout the shielding connected to ground while repeatedlytouching the outside of the wire once every two seconds for12 seconds.
We can see from this figure that without shielding, thewire acts as a sensor and will transmit noise into the systemif the user touches the wire. With the shielding, this noiseis excluded from the system. The overall capacitance of thesystem is increased to CS + CO with CS being constant,which increases RC and thus the time it takes to charge ordischarge the circuit. Our measurements indicated that ittakes about twice as long (≈ 18 ms vs ≈ 9 ms) to chargethe system with the shielded cables attached to a metal ob-ject. The size and dimensions of the metal object will ofcourse affect the overall capacitance of the system, increas-ing or decreasing the time required to charge the systemand receive a measurement.
150
170
190
210
230
250
150
170
190
210
230
250
0 3000 6000 9000 12000time (ms)
C (o
pen
shie
ld)
C (g
roun
ded
shie
ld)
(a)
(b)
Figure 7: (a) Shield is not grounded and wire capac-itance can be affected; (b) shield is grounded andprobe is protected from environmental capacitance.
4. SMARTPHONE-BASED IIAAlthough the goal of the Instant Instrument Anywhere isto make a standalone device, there is a significant advan-tage of pairing the capacitive touch sensing module witha smartphone. Modern smartphones are able to run moreadvanced software synthesizers; they contain a headphonejack to output audio and small speakers which can be aug-mented acoustically with devices such as the iPhoneAmpli-fier6. Additionally, the phone battery could power all thecomponents for all-day operation. Figure 8 shows an exam-ple of a smartphone IIA.
Metal Object
ATmega 32u4Capsense👋
Smartphone
Configure and Calibrate
Synthesis and Control
USB
Speakerphones
Figure 8: Smartphone-based Instant InstrumentAnywhere, with the smartphone fulfilling the role ofconfiguration, synthesis, power source and speaker.
An further advantage of using a smartphone for the IIA isthat a custom app can be created to provide access to con-figuration settings relating to the digital capacitive sensing.For example, the app could allow the user to modify thedC/dt threshold, or route different probe values to differentcontrol parameters (aftertouch, bend etc.) or choose dif-ferent notes or effects for each sensor being played. Thisincreases the utility and reconfigurability of the system.
A smartphone version of the IIA was constructed as a
6http://www.iphoneamplifier.net/
proof of concept. An Arudino Uno was loaded with theCapSense library as well as a USB driver that made the8u2 chip appear as a General MIDI device. The device wasthen connected, via USB and the apple camera connectionkit, to an iPhone running synthesis software. The result wasa peripheral device connected to the iPhone which providescapacitive touch sensing. This is the device shown in ourvideo response to the Kitchen Synthesizer.
5. FUTURE WORKBuilding on the portability and DIY construction of theseinstruments, we plan to create a single-PCB device that con-tains all of the circuitry necessary for the IIA. This singleboard device would contain an ATmega32u4, a sound syn-thesizer, and an amplifier circuit. It will have connectionterminals for a power source, a speaker, probes, and USBfor programming and MIDI out. This could be the heartof any new instrument that uses capacitive sensing as thefundamental interface characteristic. It will also serve as aplatform for a true acoustic throwie (with a small battery,speaker, and a magnetic probe).
6. REFERENCES[1] D. Diakopoulos and A. Kapur. HIDUINO: A firmware
for building driverless USB-MIDI devices using theArduino microcontroller. In A. R. Jensenius, A. Tveit,R. I. Godøy, and D. Overholt, editors, Proceedings ofthe 11th international conference on New interfacesfor musical expression (NIME 2011), pages 405–408,Oslo, Norway, 2011.
[2] R. Ghazala. Circuit-Bending: Build your own alieninstruments. Wiley, 2005.
[3] Graffiti Lab. LED Throwies. Online, last checked:Febuary 1, 2011, [http://www.graffitiresearchlab.com/blog/projects/led-throwies/].
[4] E. Guaus, T. Ozaslan, E. Palacios, and J. Arcos. Aleft hand gesture caption system for guitar based oncapacitive sensors. Proceedings of the 10thinternational conference on New interfaces for musicalexpression (NIME 2010), pages 238–243, 2010.
[5] J. Malloch and M. M. Wanderley. The T-Stick: frommusical interface to musical instrument. InProceedings of the 7th international conference onNew interfaces for musical expression (NIME 2007),pages 66–70, New York, NY, USA, 2007. ACM.
[6] S. Maupin, D. Gerhard, and B. Park. IsomorphicTesselations for Musical Keyboards. In Proceedings ofthe 8th Sound and Music Computing Conference(SMC 2011), 2011.
[7] A. McPherson and Y. Kim. Design and Applicationsof a Multi-Touch Musical Keyboard. Proceedings ofthe 8th Sound and Music Computing Conference(SMC 2011), 2011.
[8] J. Paradiso and N. Gershenfeld. Musical applicationsof electric field sensing. Computer Music Journal,21(2):69–89, 1997.
[9] G. Roma and A. Xambo. A tabletop waveform editorfor live performance. Proceedings of the 8thinternational conference on New interfaces formusical expression (NIME 2008), 2008.
[10] J. Snyder. The Snyderphonics Manta and a NovelUSB Touch Controller. In A. R. Jensenius, A. Tveit,R. I. Godøy, and D. Overholt, editors, Proceedings ofthe 11th international conference on New interfacesfor musical expression (NIME 2011), pages 413–416,Oslo, Norway, 2011.
