August 12, 2004IAML - IASA 2004 Congress, Olso1 Music Information Retrieval, or how to search for (and maybe find) music and do away with incipits Michael

August 12, 2004 IAML - IASA 2004 Congress, Olso 1

Music Information Retrieval, or how to search for (and maybe find) music and do away with

incipits

Michael FingerhutMultimedia Library and

Engineering BureauIRCAM – Centre Pompidou


Why Music Information Retrieval

Increased availability of musical contents in digital form (remotely)

Networks

... actions, methods and procedures for recovering stored data to provide information on music.

Need for…

Faster methods for processing contents and producing "meaning"

Computing power

Increased availability of musical contents in digital form (locally)

Storage

ImplicationTechnical Advance

pattern extraction, form recognition…

synchronization, summarization

indexation

pitch, voice extraction

work

publicationspaper

symbolic

OCR

score

about

sound

audio programs

text

fingerprint

structure

taste, mood

user

musicology

data

multimediaaudio

data

rights owner

DR

M

annotation

performer

composer

performance (live, recorded)

info

rmat

ion

mu

sic

sto

red

dat

a

metadata

genre

automatic

playlists

librarian

lyrics

abstract

digital

symbolic

physical

semantic

author

interaction

cognitive, social

theory

feature extraction,

identification

motives

conc

ept

→ s

ign

→ s

igna

l →

si

gn

→

conc

ept

textual

features

characteristicsmusical

acoustical

interpretation similarity form orchestration

melody key harmonypolyphony tempo instrumentation

chordssegments dynamics rhythm

…durationpitch timbre intensity spatial room

analysis

A (much simplified) MIR map

similarity

metadata


A typology of MIR

Preprocessing• OCR, digitization, compression• Encoding, notation• Feature extraction• Segmentation• Instrument recognition• Voice recognition

Indexation• Identification• Clustering• Classification

Extraction• Melody, key, harmony, rhythm…

Structural analysis• Polyphony• Repetition• Similarity• Summarization

Organization• Databases, systems, networks• Compression• Synchronization• Metadata

Search• Objective criteria

• Metadata indices (name, title, period, genre, instrumentation…)

• Full-text (with or without semantic tags)• Query by example (audio excerpt, melody,

contour, rhythm, tonality, harmony…)• Similarity• Acoustical characteristics

• Subjective criteria• Mood• Taste

Retrieve, deliver, use• Browsing• Playlists• Using and reusing (annotate, combine, transform)• Rights management (recognition, watermarking…)

Usability• Evaluation• User studies


Database of 100,000 documents

Common methods in MIR

Modeling• Start from a theory

• Look for patterns Statistical

• Look for patterns

• Build a theory Evaluation

• Relevance (recall, precision)

• Performance (speed, friendliness)

• Standardized test collections

1000 relevant documents 400 retrieved

documents

700 100300

Recall: 30% (300 out of 1000)

Precision: 75% (300 out of 400)

query


MIR as a multidisciplinary domain

000 – General• 000 Generalities & computer science• 001 Knowledge

• 001.4 Research

• 003 Systems• 004 Data processing

• 004.5 Storage • 004.6 Interfacing and communications

• 005 Computer programming• 005.4 Systems programming and programs • 005.7 Data in computer systems

005.8 Data security

• 006 Special computer methods • 006.3 Artificial intelligence • 006.4 Computer pattern recognition

• 006.5 Computer sound synthesis • 020 Library and information sciences• 025 Library operations

• 025.3 Bibliographic analysis and control • 025.4 Subject analysis and control • 025.5 Services to users • 025.6 Circulation

100 – Philosophy and Psychology• 150 Psychology• 152 Perception

• 152.1 Sensory perception • 152.3 Movements and motor functions, facial expressions, laterality • 152.4 Emotions and feelings • 152.5 Physiological drives • 152.8 Quantitative threshold, discrimination, reaction-time studies

• 153 Mental processes and intelligence• 153.7 Perceptual processes

300 Social sciences• 300 Sociology and anthropology

• 302.2 Communication

• 306 Culture and Institutions

• 310 General Statistics

• 340 Law• 341.7 Law of international cooperation

• 380 Commerce

• 384 Communications; Telecommunications

500 – Natural Science and Mathematics• 510 Mathematics

• 516 Geometry

600 – General Technology• 620 Engineering & allied operations

• 620.2 Sound and related vibrarions

• 621.3 Electric, electronic, magnetic, communications, computer engineering

• 681 Precision instruments & other devices

700 – The Arts• 780 Music

• 780.7 Education, performances

• 780.92 musicians, composers

• 781 General principles & musical forms• 781.1 Aesthetics, appreciation, taste

• 781.2 Elements of music

• 781.3 Composition

• 781.4 Techniques of music

• 781.5 Kinds of music

• 781.8 Musical forms


Take One, or SummarizingDave Brubeck’s Take Five (5m25s)

all 1:47:35 A small bit of each segment

each 0:14:82 One segment of each type

longest 0:03:25 Longest segment

most freq. 0:03:25 Most frequent segmentsimilarity matrixThe warmer the color (red = warmest), the more similar

1. Find which parts sound like other parts (timbre similarity)

2. Extract segments

3. Produce summaries

6

5

4

3

2

1

Courtesy of Geoffroy Peeters, Ircam

0m0s

0m0s

5m25s

5m25s


A tool to catalog and extract audio CD contents for online distribution

Automatic identification of CDs• Compute CDDB of the CD

Metadata retrieval and correction• Query Internet CDDB for metadata

• Allow correction

Extraction and compression Transfer to a Web server












What’s the use…

Increased collections• Quantity

• Variety Help in organizing

• Cataloguing and indexing Help in finding

• Search and retrieval

Contribute knowledge• Metadata, indexation

• Relevance

• Users Getting involved

• ISMIR annual conferenceswww.ismir.net• Be informed

• Publish, speak

• Review

• Organize

Documents

August 12, 2004IAML - IASA 2004 Congress, Olso1 Music Information Retrieval, or how to search for (and maybe find) music and do away with incipits Michael