Motion in Sound: Motion in Sound: Designing Sound for Interactive Dance PerformanceDesigning Sound for Interactive Dance Performance

Dr. Dan HoskenDr. Dan Hosken

Associate Professor of MusicAssociate Professor of Music

California State University, NorthridgeCalifornia State University, Northridge

Presented at: Presented at:

ATMI 2006ATMI 2006

San Antonio, TXSan Antonio, TX

September 16, 2006September 16, 2006

PurposePurpose

Present a somewhat simplified and useful Present a somewhat simplified and useful approach to creating for the interactive approach to creating for the interactive dance medium dance medium

Facilitate collaboration between students Facilitate collaboration between students of dance and students of musicof dance and students of music

Objectives:Objectives:

Give an overview of the hardware and Give an overview of the hardware and software components of a camera-based software components of a camera-based interactive dance/music systeminteractive dance/music system

Present a loose taxonomy of motion Present a loose taxonomy of motion parameters and mapping typesparameters and mapping types

Suggest some useful mappings between Suggest some useful mappings between motion parameters and sound element motion parameters and sound element parametersparameters

Illustrate these mappings using examples of Illustrate these mappings using examples of my recent work with the Palindrome IMPGmy recent work with the Palindrome IMPG

General System OverviewGeneral System Overview

Camera trained on dancer(s) is connected Camera trained on dancer(s) is connected to computerto computer

Video Analysis Software abstracts motion Video Analysis Software abstracts motion data in realtimedata in realtime

Motion Data are passed to Sound SoftwareMotion Data are passed to Sound Software Sound Software maps incoming motion Sound Software maps incoming motion

data to sound element parameters in data to sound element parameters in realtimerealtime

Overview w/ bad clipartOverview w/ bad clipart

video computervideo computer audio computeraudio computer

ethernetethernet

Sound Generation SoftwareSound Generation Software

Max/MSPMax/MSP (Cycling ‘74) (Cycling ‘74) PDPD (Miller Puckette)—free! (Miller Puckette)—free! Supercollider (J. McCartney)—free!Supercollider (J. McCartney)—free! ReaktorReaktor (Native Instruments) (Native Instruments) ……and any software that can receive and any software that can receive

data and produce sound in realtimedata and produce sound in realtime

Video Analysis SoftwareVideo Analysis Software

EyeConEyeCon (Frieder Weiss) (Frieder Weiss) EyesWebEyesWeb (eyesweb.org)—free! (eyesweb.org)—free! JitterJitter (Cycling ‘74) (Cycling ‘74)

SoftVNS (David Rokeby)SoftVNS (David Rokeby) CyclopsCyclops (Eric Singer/Cycling ‘74) (Eric Singer/Cycling ‘74) TapTools (Electrotap)TapTools (Electrotap) cv.jit (Jean-Marc Pelletier)cv.jit (Jean-Marc Pelletier) Eyes (Rob Lovel)—free!Eyes (Rob Lovel)—free!

Objectives (redux):Objectives (redux):

Give an overview of the hardware and software Give an overview of the hardware and software components of a camera-based interactive components of a camera-based interactive dance/music systemdance/music system

Present a loose taxonomy of motion parameters and Present a loose taxonomy of motion parameters and mapping typesmapping types

Suggest some useful mappings between motion Suggest some useful mappings between motion parameters and sound element parametersparameters and sound element parameters

Illustrate these mappings using examples of my Illustrate these mappings using examples of my recent work with the Palindrome IMPGrecent work with the Palindrome IMPG

Definitions (1)Definitions (1)

Motion ParameterMotion Parameter: made up of specified data : made up of specified data abstracted from part or all of video, e.g., abstracted from part or all of video, e.g., HeightHeight WidthWidth DynamicDynamic

Sound ElementSound Element: a distinct, coherent sonic : a distinct, coherent sonic behavior created by one or more synthesis or behavior created by one or more synthesis or processing techniques, e.g., processing techniques, e.g., A Low Drone created by FM SynthesisA Low Drone created by FM Synthesis Time-stretched text created by GranulationTime-stretched text created by Granulation Percussive patterns created by Sample PlaybackPercussive patterns created by Sample Playback

Definitions (2)Definitions (2)

Sound Element ParameterSound Element Parameter: a parameter of a : a parameter of a synthesis/processing technique, e.g.,synthesis/processing technique, e.g., Modulation Frequency of a simple FM pairModulation Frequency of a simple FM pair Grain Size of a granulated sound fileGrain Size of a granulated sound file Ir/regularity of tempo in a rhythmic patternIr/regularity of tempo in a rhythmic pattern

MappingMapping: the connection between a motion : the connection between a motion parameter and a sound element parameter, parameter and a sound element parameter, e.g.,e.g., HeightHeightmodulation frequency of FMmodulation frequency of FM WidthWidthgrain size of granulated sound filegrain size of granulated sound file DynamicDynamicIrregularity of tempoIrregularity of tempo

Definitions (3)Definitions (3)

SceneScene: a group of motion parameters, : a group of motion parameters, sound elements, and mappings sound elements, and mappings between thembetween them

EyeCon Interface (1)EyeCon Interface (1)

Field: can measure height or width or dynamic or…

Touchlines: detect crossing and position on line

EyeCon Interface (2)EyeCon Interface (2)Fields and lines are mapped to MIDI data (or OSC)

Sequencer steps through “scenes”

Taxonomy of Motion ParametersTaxonomy of Motion Parameters

Body Parameters: position independent, Body Parameters: position independent, “attached” to body“attached” to body HeightHeight WidthWidth DynamicDynamic

Stage Parameters: position dependent, Stage Parameters: position dependent, “attached” to stage“attached” to stage Left-right positionLeft-right position TouchlinesTouchlines Extremely Narrow FieldsExtremely Narrow Fields

Parameter Type ExamplesParameter Type Examples

Stage Parameter (position): Scene 3 from Stage Parameter (position): Scene 3 from Brother-Sister SoloBrother-Sister Solo Julia Eisele, dancer/choregrapherJulia Eisele, dancer/choregrapher Stuttgart, June 2005Stuttgart, June 2005

Body Parameter (Dynamic): Body Parameter (Dynamic): ConversationConversation Robert Wechsler, dancer/choreographerRobert Wechsler, dancer/choreographer Julia Eisele, dancerJulia Eisele, dancer Stuttgart, June 2005Stuttgart, June 2005

Primary/Secondary MappingsPrimary/Secondary Mappings

Primary Mapping: controls dominant sonic Primary Mapping: controls dominant sonic featurefeature

Secondary Mapping: …is secondary…Secondary Mapping: …is secondary… Example: Scene 3 from Brother-Sister SoloExample: Scene 3 from Brother-Sister Solo

Primary mapping: positionPrimary mapping: positionposition in sound position in sound “landscape”“landscape”

Secondary mapping: dynamicSecondary mapping: dynamicdisturbance of disturbance of dronedrone

Secondary mapping: widthSecondary mapping: widthloop size/speed of loop size/speed of segment within sound filesegment within sound file

Sound Element mappings (1)Sound Element mappings (1)

A Human Conversation (in progress)A Human Conversation (in progress) Scene 7-8:Scene 7-8:

DynamicDynamicGranulated Text (playback rate)Granulated Text (playback rate) Scene 9:Scene 9:

Dynamic (left)Dynamic (left)Granulated Text (playback rate)Granulated Text (playback rate) Dynamic (right)Dynamic (right)Granulated Text (playback rate)Granulated Text (playback rate)

A Human ConversationA Human Conversation

Robert Wechsler (Palindrome), Robert Wechsler (Palindrome), choreographer/dancerchoreographer/dancer

J’aime Morrison (CSUN), J’aime Morrison (CSUN), choreographer/dancerchoreographer/dancer

Dan Hosken, composer and sound Dan Hosken, composer and sound programmerprogrammer

Work session, CSUN, June 23, 2006Work session, CSUN, June 23, 2006

Sound Element mappings (2)Sound Element mappings (2)

Perceivable Bodies (Emily Fernandez)Perceivable Bodies (Emily Fernandez) Scene 3a:Scene 3a:

PositionPositionGranulated Text (position in file) [Primary]Granulated Text (position in file) [Primary] WidthWidthGranulated Text (grain duration)Granulated Text (grain duration) DynamicDynamicLow FM Drone (mod frequency)Low FM Drone (mod frequency)

Scene 3b:Scene 3b: PositionPositionPhase Voc File (position in file) [Primary]Phase Voc File (position in file) [Primary] WidthWidthPhase Voc file (loop length/rate)Phase Voc file (loop length/rate) DynamicDynamicLow FM Drone (mod frequency)Low FM Drone (mod frequency)

Scene 4:Scene 4: DynamicDynamicGranulated Noise (density) [Primary]Granulated Noise (density) [Primary] DynamicDynamicGranulated Noise (position in file)Granulated Noise (position in file)

Perceivable BodiesPerceivable Bodies

Emily Fernandez, Emily Fernandez, choreographer/dancerchoreographer/dancer

Frieder Weiss, projections and Frieder Weiss, projections and interactive programminginteractive programming

Dan Hosken, composer and sound Dan Hosken, composer and sound programmerprogrammer

World Premiere at Connecticut College, World Premiere at Connecticut College, April 1, 2006April 1, 2006

dan.hosken@csun.edudan.hosken@csun.edu

Examples shown can be found:Examples shown can be found:http://www.http://www.csuncsun.edu/~dwh50750/Papers-Presentations/.edu/~dwh50750/Papers-Presentations/

Full Pieces can be found:http://www.http://www.csuncsun.edu/~dwh50750/Music/.edu/~dwh50750/Music/

Other Examples of Palindrome’s work:http://www.palindrome.de/

Max/MSP ScreenshotMax/MSP Screenshot

PD ScreenshotPD Screenshot

Reaktor ScreenshotsReaktor Screenshots

Eyecon ScreenshotEyecon Screenshot

EyesWeb ScreenshotEyesWeb Screenshot

Jitter ScreenshotJitter Screenshot

Cyclops ScreenshotCyclops Screenshot