Sound and Music Computing Research: Trends and and Music Computing Research: Trends and Challenges Xavier…

  • Published on
    05-Jul-2018

  • View
    212

  • Download
    0

Embed Size (px)

Transcript

  • Sound and Music Computing Research:

    Trends and Challenges

    Xavier SerraMusic Technology Group

    Universitat Pompeu Fabra, Barcelona

  • Xavier Serra - 2008

    Introduction

  • Xavier Serra - 2008

    Music communication framework

    Symbolic representation

    Temporal controls

    Source sound

    Sound field

    PerceptionCognition

    Listener

    Composer

    Performer

    InstrumentRoom

    Musical Knowledge base

    Physical Knowledge base (Moorer, 1990)

  • Xavier Serra - 2008

    SMC research disciplines

    Music

    Psychology

    PhysicsEngineering

    SMC

    TheoryPerformance CompositionComputer

    Science

    ArtificialIntelligence Psychomusicology

    Cognition

    Psychoacoustics

    Acoustics

    Device design

    Digital signal

    processing

    Digitalhardware

    Programming

    (Moorer, 1990)

  • Xavier Serra - 2008

    New context for SMC research

    CONTENT(databases)

    COMMUNITY(social tools)

    MUSIC3.0

    CREATION(production

    tools)

    collaborativecreativity

    contentsharing

    contentprocessing

  • Xavier Serra - 2008

    Basic Research Challenges

  • Xavier Serra - 2008

    Understanding sounding objects

    String vibration Membrane vibration

  • Xavier Serra - 2008

    Understanding perception

    Diagram of inner ear

  • Xavier Serra - 2008

    Understanding cognition

    (Peretz and Coltheart, 2003)

    research

  • Xavier Serra - 2008

    Current Trends and Challenges

  • Xavier Serra - 2008

    Sound synthesis

    Current trends:

    Audio content processing

    Physical modeling based on measurements

    Gesture-sound modelingTraditional approach to sound

    synthesis

  • Xavier Serra - 2008

    Content processing

    Vocaloid (Yamaha & MTG-UPF, 2005)

  • Xavier Serra - 2008

    Physical modeling from measurements

    Measuring impulse responses from a violin

    Various sensors on a violin

  • Xavier Serra - 2008

    Gesture-sound modeling

    Finger position

    Playing state estimator

    String

    Bow- bridge distance

    Bow pressing force

    Bow acceleration

    Bow velocity

    Bow position

    Audio

    Performed scores

    Violin synthesisPachelbel Canon

  • Xavier Serra - 2008

    Sound/Music description

    (Lesaffre et alt., 2003)

    Current trends:

    Search and retrieval

    Combining bottom-up and top-down approaches

  • Xavier Serra - 2008

    Search and retrieval

    Harmonic analysis (Gomez, 2006)

    MusicSurfer

  • Xavier Serra - 2008

    Bottom-up and top-down

    Music recommendation system (Celma, 2006)

    file:///home/xavier/MTG/presentacions-MTG/Foafing.wmv

  • Xavier Serra - 2008

    Interaction

    Mathews with the Radio Drum

    Current trends:

    Understanding performance expression

    New musical interfaces

  • Xavier Serra - 2008

    Understanding performance

    High-qualityrecordings

    analyze

    encode

    extract

    obtain

    Analyze

    Machinerepresentation

    Structure of pieces

    SymbolicdescriptionExpressive

    aspects ofrecordings

    Machinelearning

    Models

    Synthesizedscore

    Automatic expression

  • Xavier Serra - 2008

    New interfaces

    Reactable (Jord, 2006)

  • Xavier Serra - 2008

    Broad Challenges

  • Xavier Serra - 2008

    Bridging the semantic gap

    Signalfeatures

    ContentObjects

    HumanKnowledge

    semantic gap

    Audio(music recordings)

    Text(lyrics, editorial text,

    press releases, )

    Image(video clips, CD covers,

    printed scores, )

    frequency

    duration

    spectrum

    intensity

    pitchtimbreloudness time

    melody

    harmony

    rhythmsource

    dynamics

    emotions

    genre

    understanding

    musicscores

    graphicstyle

    similarity

    personalidentity

    memories

    opinions

    nouns

    adjectivesscenes

    signstags

    labels

    motions

    shotrhythm

    expectations

    colorsshapes

    texturescontrastsverbs

    sentences

    articles

    numbers

    semanticfeatures

    Music information plane

  • Xavier Serra - 2008

    Signalfeatures

    ContentObjects

    HumanKnowledge

    semantic gap

    Audio(music recordings)

    Text(lyrics, editorial text,

    press releases, )

    Image(video clips, CD covers,

    printed scores, )

    frequency

    duration

    spectrum

    intensity

    pitchtimbreloudness time

    melody

    harmony

    rhythmsource

    dynamics

    emotions

    genre

    understanding

    musicscores

    graphicstyle

    similarity

    personalidentity

    memories

    opinions

    nouns

    adjectivesscenes

    signstags

    labels

    motions

    shotrhythm

    expectations

    colorsshapes

    texturescontrastsverbs

    sentences

    articles

    numbers

    semanticfeatures

    Bridging the semantic gap

    Signalprocessing

    Machinelearning

    Webmining

    MusictheoryStatistical

    modeling

    Music information plane

  • Xavier Serra - 2008

    Signalfeatures

    ContentObjects

    HumanKnowledge

    semantic gap

    Audio(music recordings)

    Text(lyrics, editorial text,

    press releases, )

    Image(video clips, CD covers,

    printed scores, )

    frequency

    duration

    spectrum

    intensity

    pitchtimbreloudness time

    melody

    harmony

    rhythmsource

    dynamics

    emotions

    genre

    understanding

    musicscores

    graphicstyle

    similarity

    personalidentity

    memories

    opinions

    nouns

    adjectivesscenes

    signstags

    labels

    motions

    shotrhythm

    expectations

    colorsshapes

    texturescontrastsverbs

    sentences

    articles

    numbers

    semanticfeatures

    Bridging the semantic gap

    Signalprocessing

    Machinelearning

    Webmining

    MusictheoryStatistical

    modeling

    Computationalneuroscience

    Multimodalprocessing

    Musiccognition

    Ontologies

    Reasoningrules

    Computationalmusicology

    Textunderstanding

    Music information plane

  • Xavier Serra - 2008

    Understanding music making

    (Leman, 2007)

  • Xavier Serra - 2008

    Understanding communication

    Score A

    Score B

    Score C

    Score D

    Performer A

    Performer B

    Performer C

    Performer D

    Instrument A

    Instrument B

    Instrument C

    Instrument D

    Audience indiv. 1

    COMPOSITION

    Audience indiv. 2

    Audience indiv. 3

    Compositional channel: musical message + role in the performance (solo, accompanist, etc)

    Sonic channel

    Instrumental channel: sound-producing and modifying movements / actions, haptic feedback

    Visual channel

    AUDIENCE

    PERFORMANCE

  • Xavier Serra - 2008

    Understanding social interaction

    http://freesound.org