Sound and Music Computing Research:

Trends and Challenges

Xavier SerraMusic Technology Group

Universitat Pompeu Fabra, Barcelona

Xavier Serra - 2008

Introduction

Xavier Serra - 2008

Music communication framework

Symbolic representation

Temporal controls

Source sound

Sound field

PerceptionCognition

Listener

Composer

Performer

InstrumentRoom

“Musical” Knowledge base

“Physical” Knowledge base (Moorer, 1990)

Xavier Serra - 2008

SMC research disciplines

Music

Psychology

PhysicsEngineering

SMC

TheoryPerformance CompositionComputer

Science

ArtificialIntelligence Psychomusicology

Cognition

Psychoacoustics

Acoustics

Device design

Digital signal

processing

Digitalhardware

Programming

(Moorer, 1990)

Xavier Serra - 2008

New context for SMC research

CONTENT(databases)

COMMUNITY(social tools)

MUSIC3.0

CREATION(production

tools)

collaborativecreativity

contentsharing

contentprocessing

Xavier Serra - 2008

Basic Research Challenges

Xavier Serra - 2008

Understanding sounding objects

String vibration Membrane vibration

Xavier Serra - 2008

Understanding perception

Diagram of inner ear

Xavier Serra - 2008

Understanding cognition

(Peretz and Coltheart, 2003)

research

Xavier Serra - 2008

Current Trends and Challenges

Xavier Serra - 2008

Sound synthesis

Current trends:

Audio content processing

Physical modeling based on measurements

Gesture-sound modelingTraditional approach to sound

synthesis

Xavier Serra - 2008

Content processing

Vocaloid (Yamaha & MTG-UPF, 2005)

Xavier Serra - 2008

Physical modeling from measurements

Measuring impulse responses from a violin

Various sensors on a violin

Xavier Serra - 2008

Gesture-sound modeling

Finger position

Playing state estimator

String

Bow- bridge distance

Bow pressing force

Bow acceleration

Bow velocity

Bow position

Audio

Performed scores

Violin synthesisPachelbel Canon

Xavier Serra - 2008

Sound/Music description

(Lesaffre et alt., 2003)

Current trends:

Search and retrieval

Combining bottom-up and top-down approaches

Xavier Serra - 2008

Search and retrieval

Harmonic analysis (Gomez, 2006)

MusicSurfer

Xavier Serra - 2008

Bottom-up and top-down

Music recommendation system (Celma, 2006)

Xavier Serra - 2008

Interaction

Mathews with the Radio Drum

Current trends:

Understanding performance expression

New musical interfaces

Xavier Serra - 2008

Understanding performance

High-qualityrecordings

analyze

encode

extract

obtain

Analyze

Machinerepresentation

Structure of pieces

SymbolicdescriptionExpressive

aspects ofrecordings

Machinelearning

Models

Synthesizedscore

Automatic expression

Xavier Serra - 2008

New interfaces

Reactable (Jordà, 2006)

Xavier Serra - 2008

Broad Challenges

Xavier Serra - 2008

Bridging the semantic gap

Signalfeatures

ContentObjects

HumanKnowledge

semantic gap

Audio(music recordings)

Text(lyrics, editorial text,

press releases, …)

Image(video clips, CD covers,

printed scores, …)

frequency

duration

spectrum

intensity

pitchtimbreloudness time

melody

harmony

rhythm

source

dynamics

emotions

genre

understanding

musicscores

graphicstyle

similarity

personalidentity

memories

opinions

nouns

adjectivesscenes

signstags

labels

motions

shotrhythm

expectations

colorsshapes

textures

contrastsverbs

sentences

articles

numbers

semanticfeatures

Music information plane

Xavier Serra - 2008

Signalfeatures

ContentObjects

HumanKnowledge

semantic gap

Audio(music recordings)

Text(lyrics, editorial text,

press releases, …)

Image(video clips, CD covers,

printed scores, …)

frequency

duration

spectrum

intensity

pitchtimbreloudness time

melody

harmony

rhythm

source

dynamics

emotions

genre

understanding

musicscores

graphicstyle

similarity

personalidentity

memories

opinions

nouns

adjectivesscenes

signstags

labels

motions

shotrhythm

expectations

colorsshapes

textures

contrastsverbs

sentences

articles

numbers

semanticfeatures

Bridging the semantic gap

Signalprocessing

Machinelearning

Webmining

MusictheoryStatistical

modeling

Music information plane

Xavier Serra - 2008

Signalfeatures

ContentObjects

HumanKnowledge

semantic gap

Audio(music recordings)

Text(lyrics, editorial text,

press releases, …)

Image(video clips, CD covers,

printed scores, …)

frequency

duration

spectrum

intensity

pitchtimbreloudness time

melody

harmony

rhythm

source

dynamics

emotions

genre

understanding

musicscores

graphicstyle

similarity

personalidentity

memories

opinions

nouns

adjectivesscenes

signstags

labels

motions

shotrhythm

expectations

colorsshapes

textures

contrastsverbs

sentences

articles

numbers

semanticfeatures

Bridging the semantic gap

Signalprocessing

Machinelearning

Webmining

MusictheoryStatistical

modeling

Computationalneuroscience

Multimodalprocessing

Musiccognition

Ontologies

Reasoningrules

Computationalmusicology

Textunderstanding

Music information plane

Xavier Serra - 2008

Understanding music making

(Leman, 2007)

Xavier Serra - 2008

Understanding communication

Score A

Score B

Score C

Score D

Performer A

Performer B

Performer C

Performer D

Instrument A

Instrument B

Instrument C

Instrument D

Audience indiv. 1

COMPOSITION

Audience indiv. 2

Audience indiv. 3

Compositional channel: musical message + role in the performance (solo, accompanist, etc…)

Sonic channel

Instrumental channel: sound-producing and modifying movements / actions, haptic feedback

Visual channel

AUDIENCE

PERFORMANCE

Xavier Serra - 2008

Understanding social interaction

http://freesound.org