An Augmented Snare DrumELEC3580 LEVEL 3 PROJECT

Robin Gray Simon Lindsell Rachael Minster

IntroductionAims of the project

1) Use a camera to analyse the expressive patterns created on the drum skin during a drum brush performance.

2) Use a range of electrical sensors to enhance and improve the accuracy when detecting drum hits. This data will be used in conjunction with the visual data.

3) To emulate drum brush techniques used on a traditional acoustic drum controlling either audio samples or synthesis. At present, the use of drum brushes on commercial electronic drum kits is very restricted.

4) To use sensor data to control and manipulate parameters of a visual feed. This will be in real-time as an accompaniment to a performance.

System Overview

Electrical Sensors and

Camera

Data Processing

and Manipulation

Audio and Video Output

Modules

Parameter Mapping

Initial System Design

Camera(Capture brush movement

and position) ‏

Piezoelectric Transducer(capture drum hits and

velocity) ‏

Flex Sensor(capture pressure

appliedto the brush) ‏

Flex Sensor(capture pressure

appliedto the brush) ‏

PIC Microcontroller(A/D convertion

serial transmition) ‏

Processing in

MAX/MSP and JITTER

Serial Connection

USB Connection

Visual/Audio output

Footswitches ‏

Electrical Sensors and

Camera

Data Processing

and Manipulation

Audio and Video Output

Modules

Parameter Mapping

Initial testing - Camera

Initial testing discovered that our idea was viable, and that lighting the camera from above would help define the brushes.

This included:

• Clear and frosted drum skins• Different webcams and handheld camcorders• Various lighting conditions

Lens choice

A wide angled lens would allow the camera to be placed closer to the drum, this was initially tested with a door viewer

The final system consisted of three 38mm lenses mounted in a plastic tube

Initial Testing - Subtraction

Initial Image Processing

•Thresholding could be used to help differentiate the drum brushes.

•Unused areas of the video could be masked, removing further noise and un-needed background

•Necessary to differentiate between the two brushes.

Image Processing - Segmentation• Separates each frame into labelled objects

• This can be used to identify brushes on the skin

• Problems – Apparent ‘jumping’ of brushes between frames

– The merging of brushes when they touch

Motion Tracking 1 - Distance

•If the object travels greater than a set distance the distance to the other object can be measured.

•If this is smaller we can presume that the image has ‘jumped’ and can then re-label the image

Motion Tracking 2 – Colour Tracking

Requires using two different coloured brushes. A simple multiband thresholding method was used to detect objects within a colour range, differentiating between the two brushes.

Motion Tracking 3 - Orientation

120°

65°

It was noted that orientation of brushes remained constant throughout and could be used to track brushes.

The orientation can then be extracted and used to detect the left and right brushes.

Data Acquisition Overview

Sampler

Half rectifcation and smoothing

ADC

Encoder and serial transmitter

Data decoding and manipulation in

MAX/MSP

Webcam

Signal processing

Signal processing

Comparator

Piezo

Flex-sensor in left brush

Flex-sensor in right brush

PIC16F877A

PC

interrupt

Footswitches

Sensors – Piezo

+

-

Inve rte r

~

15V

-15V

10kΩ

200kΩ10kΩ

1µF10kΩ

Re ctifcation and Smoothing

Inve rting Amplife r

+

-

15V

-15V

10kΩ

Comparitor

+

-

15V

0V

15V

100kΩ

50Ω

• Mounted on the skin using tape and later a fabric pocket

• Hit used to trigger the PIC’s interrupt

• Signal processing involved amplification, smoothing and thresholding.

• A monostable circuit was used to prevent multiple triggering

•The piezo’s signal was also processed using a separate amplifying and smoothing circuit to give a hit’s velocity

Sensors – Flex Sensors

+

-

15V

-15V

15V

1kΩ

1kΩ

Diffe re ntial Amplife r

+

-7kΩ

10kΩ+

-

+

-

1kΩ

1kΩ

47kΩ

fex sensor

(11V to 8V depending on fex sensor)

4k7Ω

5k6Ω

(8V fxed)15V

-15V

15V

15V

-15V

15V

-15V

• Flex sensors were embedded inside the brushes to measure the force applied by the drummer.

• Signal Processing is used to convert the variable voltage from the flex sensor into a voltage between 0V and 5V.

Sensor Circuits – Footswitches and PCB

10kΩ

resetfootswitch 1

from piezo velocity circuitfrom flex sensor 1 circuitfrom flex sensor 2 circuit

from monostable

serial data out

footswitch 2

4MHz

• Two footswitches were used to provide the performer with additional control of the output system

• These signals were input into the digital inputs of the PIC

• The other sensor signals were input into the analogue inputs of the PIC

PIC Program Overview

Read ADC for analogue input 1 (fex-sensor1)

START

Break data into 2 nibbles and convert into ASCII equivalent

Transmit header ‘m’ and 2 nibbles

Read ADC for analogue input 2 (fex-sensor2)

Break data into 2 nibbles and convert into ASCII equivalent

Transmit header ‘n’ and 2 nibbles

Is RB1 high (footswitch 1)?

Break piezo data into 2 nibbles and convert into ASCII equivalent

Transmit header ‘p’ and 2 nibbles

Yes

No

Transmit ‘S’

Read ADC for analogue input 0 (piezo velocity)

Is RB2 high (footswitch 2)?

Yes

NoTransmit ‘T’

Check Flags

START ISR

Yes

NoIs timer0 flag set?

ERROR

Output ‘X’

Reset timer0 to FF

Reset flags

END ISR

PIC data encoding

START

Read frst 4-bits of data

Is high nibble <

10?

+0x30

+0x37

Read second 4-bits of data

Is low nibble <

10?

+0x30

+0x37

END

Conve rts to as cii characte rs e quivale nt of he x value

(nibl = data % 16)

(nibh = data / 16)

Serial data transmission

109 53 88 53 110 65 55 112 56 69 109 53 53 110 65 55 112 58 52 m 5 X 5 n A 7 p 8 E m 5 5 n A 7 p 9 4

109 53 53 110 65 56 112 56 55 83 109 53 53 110 65 65 112 52 66 83 m 5 5 n A 8 p 8 7 S m 5 5 n A A p 4 B S

Characters transmitted before sensor data to identify sensor.

Data stream can be searched for identifiers and sensor data extracted

Two nibbles need to be combined to give the 8 bit sensor value

Example Datastream:

PIC data decoding

Mapping Overview

Flex Sensor 1

Flex Sensor 2

Piezo Camera

Brush area

Brush x position

Brush y position

Flex value

Flex value

Velocity Hit

Drum Synthesise

r

PC – data processing in Max/MSP and J itter

Audio Looper

Drum Sample

Triggering

Abstract Synthesise

r

Video Manipulation

inputs

outputs

Foot-switche

s

The Audio Interface 1 - Drum Sample Playback

•Numerous audio samples from a snare drum were recorded to create a large bank of samples.

•When the piezo detects a hit, a sample of a similar velocity is selected at random

•A change in flex sensor data triggers a looping sweep sample.

The Audio Interface 2 - Drum Synthesiser

•The harmonic content of snare drum samples was analysed and the results used to determine the components of the synthesiser.

•The interrupt data from the piezo is used to trigger the “hit” sound, whilst the velocity data controls its amplitude.

•Motion tracking data used to control the overall volume of sweeps.

The Audio Interface 3 – Abstract Synthesiser

•The user can also play the drum as a melodic instrument, using the sensor data to control pitch and timbre.

•The piezo velocity is used to control the frequency and volume of the hit.

•For the sweep sounds, the x co-ordinates of the brushes control the pitch of the sound heard, whilst the y co-ordinates control the volume.

Audio Interface 4 – Record and Playback Module

•The first foot switch allows the performer to record samples in real-time, which are stored in a buffer.

•Recordings can be made either internally, or from an external source.

•The second footswitch can be used to set the playback speed

•The user can play over the top of the recorded sample by switching on another of the audio modules.

•The patch bay allows the user control over data mapping

Video Modules

•Provides a graphical output to enhance the performance using images from the webcam and an external source.

•Maps the sensor data to video processing effects such as hue, brightness and saturation.

Latency Testing

• Average latency of 193ms

• Jitter of 120ms

Validation

A questionnaire was given to a sample of drummers and non-drummers to evaluate the success of the project.

Results comparing drummers and non-drummers

Overall Mean Result

Conclusions

• Sensors were successfully used to capture data from a drum performance, including:

• the velocity of hits, • the flex of the brushes• the patterns created on the drum skin• footswitches

• Data was mapped in Max/MSP, allowing for the manipulation of audio and video parameters.

• The final system provides the drummer with control over the output through the following modules

• drum sample playback• realistic synthesiser• abstract synthesiser• record and playback of samples• video manipulation

• Validations were carried out to evaluate the functionality of the Augmented Drum.

• Results reflected the success of the video output and the drum as a novel surface.

Further Developments

•System latency improvements

•The refinement of the realistic synthesiser and video module.

•The addition of alternative sensors including inclinometers and accelerometers

•Adaptation to allow for the detection of hand gestures as a control