overview intro content ACA summary course outline

MUSI-6201 — Computational Music AnalysisPart 2: Introduction

alexander lerch

November 4, 2015

overview intro content ACA summary course outline

introductionoverview

text bookChapter 1: Introduction (pp. 1–6)

sources: slides (latex) & Matlab

github repository

lecture contentwhat is audio content analysis?what are typical applications?what is audio content?what are the processing blocks of a typical ACA system?

overview intro content ACA summary course outline

introductionoverview

text bookChapter 1: Introduction (pp. 1–6)

sources: slides (latex) & Matlab

github repository

lecture contentwhat is audio content analysis?what are typical applications?what is audio content?what are the processing blocks of a typical ACA system?

overview intro content ACA summary course outline

introductionoverview

text bookChapter 1: Introduction (pp. 1–6)

sources: slides (latex) & Matlab

github repository

lecture contentwhat is audio content analysis?what are typical applications?what is audio content?what are the processing blocks of a typical ACA system?

overview intro content ACA summary course outline

introductionoverview

text bookChapter 1: Introduction (pp. 1–6)

sources: slides (latex) & Matlab

github repository

lecture contentwhat is audio content analysis?what are typical applications?what is audio content?what are the processing blocks of a typical ACA system?

overview intro content ACA summary course outline

introductionoverview

text bookChapter 1: Introduction (pp. 1–6)

sources: slides (latex) & Matlab

github repository

lecture contentwhat is audio content analysis?what are typical applications?what is audio content?what are the processing blocks of a typical ACA system?

overview intro content ACA summary course outline

introductionaudio content analysis — terminology

goalextract information about the content of audio data

terminologymusic information retrieval (MIR):

analysis and retrieval of music databoth audio and symbolic data

machine listening & computer audition

focus on the recognition and understanding of music

computational auditory scene analysis (CASA)

focus on human perception & cognition, understanding of theauditory scene

overview intro content ACA summary course outline

introductionaudio content analysis — terminology

goalextract information about the content of audio data

terminologymusic information retrieval (MIR):

analysis and retrieval of music databoth audio and symbolic data

machine listening & computer audition

focus on the recognition and understanding of music

computational auditory scene analysis (CASA)

focus on human perception & cognition, understanding of theauditory scene

overview intro content ACA summary course outline

introductionaudio content analysis — research field

interdisciplinarydigital signal processingmachine learning / data miningmusicologypsycho-acoustics. . .

communityISMIR: ismir.net

annual conferencescumulative list of conference papersISMIR-Community mailing listMIREX: MIR Evaluation eXchange

related publicationsconferences: ICASSP, ICME, SMC, DAFx, ACM MM, . . .journals: TASLP, Computer Music, JNMR, JAES, . . .

overview intro content ACA summary course outline

introductionaudio content analysis — research field

interdisciplinarydigital signal processingmachine learning / data miningmusicologypsycho-acoustics. . .

communityISMIR: ismir.net

annual conferencescumulative list of conference papersISMIR-Community mailing listMIREX: MIR Evaluation eXchange

related publicationsconferences: ICASSP, ICME, SMC, DAFx, ACM MM, . . .journals: TASLP, Computer Music, JNMR, JAES, . . .

overview intro content ACA summary course outline

introductionaudio content analysis — research field

interdisciplinarydigital signal processingmachine learning / data miningmusicologypsycho-acoustics. . .

communityISMIR: ismir.net

annual conferencescumulative list of conference papersISMIR-Community mailing listMIREX: MIR Evaluation eXchange

related publicationsconferences: ICASSP, ICME, SMC, DAFx, ACM MM, . . .journals: TASLP, Computer Music, JNMR, JAES, . . .

overview intro content ACA summary course outline

introductionapplications

organization in large databases

search & retrieval, classification, similarity

interfaces to search and retrieval

fingerprinting, query-by-humming systems

music visualizationsymbolic (bars, harmony, score, . . . ), similarity mappings

adaptive processingadaptive effect parametrization or algorithm selection

adaptive interactionplaylist generation, recommendation

overview intro content ACA summary course outline

introductionapplications

organization in large databases

search & retrieval, classification, similarity

interfaces to search and retrieval

fingerprinting, query-by-humming systems

music visualizationsymbolic (bars, harmony, score, . . . ), similarity mappings

adaptive processingadaptive effect parametrization or algorithm selection

adaptive interactionplaylist generation, recommendation

overview intro content ACA summary course outline

introductionapplications

organization in large databases

search & retrieval, classification, similarity

interfaces to search and retrieval

fingerprinting, query-by-humming systems

music visualizationsymbolic (bars, harmony, score, . . . ), similarity mappings

adaptive processingadaptive effect parametrization or algorithm selection

adaptive interactionplaylist generation, recommendation

overview intro content ACA summary course outline

introductionapplications

organization in large databases

search & retrieval, classification, similarity

interfaces to search and retrieval

fingerprinting, query-by-humming systems

music visualizationsymbolic (bars, harmony, score, . . . ), similarity mappings

adaptive processingadaptive effect parametrization or algorithm selection

adaptive interactionplaylist generation, recommendation

overview intro content ACA summary course outline

introductionapplications

organization in large databases

search & retrieval, classification, similarity

interfaces to search and retrieval

fingerprinting, query-by-humming systems

music visualizationsymbolic (bars, harmony, score, . . . ), similarity mappings

adaptive processingadaptive effect parametrization or algorithm selection

adaptive interactionplaylist generation, recommendation

overview intro content ACA summary course outline

introduction(commercial) examples

recommendation, playlist generation

fingerprinting

score following

(multi-) pitch detection

overview intro content ACA summary course outline

introduction(commercial) examples

recommendation, playlist generation

fingerprinting

score following

(multi-) pitch detection

overview intro content ACA summary course outline

introduction(commercial) examples

recommendation, playlist generation

fingerprinting

score following

(multi-) pitch detection

overview intro content ACA summary course outline

introduction(commercial) examples

recommendation, playlist generation

fingerprinting

score following

(multi-) pitch detection

overview intro content ACA summary course outline

audio contentsources

what are the sources of (musical) audio content?

1 score:definition of musical ideas“blue-print” of the musicexamples: melody, key, harmony, rhythmic patterns, . . .

2 performance:unique acoustic renditioninformation in the score is interpreted, modified, added toexamples: (micro-)tempo, dynamics, intonation, . . .

3 production:aesthetic choicesediting & processingexamples: sound quality (EQ, microphone positioning),changes in timing and pitch

overview intro content ACA summary course outline

audio contentsources

what are the sources of (musical) audio content?

1 score:definition of musical ideas“blue-print” of the musicexamples: melody, key, harmony, rhythmic patterns, . . .

2 performance:unique acoustic renditioninformation in the score is interpreted, modified, added toexamples: (micro-)tempo, dynamics, intonation, . . .

3 production:aesthetic choicesediting & processingexamples: sound quality (EQ, microphone positioning),changes in timing and pitch

overview intro content ACA summary course outline

audio contentsources

what are the sources of (musical) audio content?

1 score:definition of musical ideas“blue-print” of the musicexamples: melody, key, harmony, rhythmic patterns, . . .

2 performance:unique acoustic renditioninformation in the score is interpreted, modified, added toexamples: (micro-)tempo, dynamics, intonation, . . .

3 production:aesthetic choicesediting & processingexamples: sound quality (EQ, microphone positioning),changes in timing and pitch

overview intro content ACA summary course outline

audio contentsources

what are the sources of (musical) audio content?

1 score:definition of musical ideas“blue-print” of the musicexamples: melody, key, harmony, rhythmic patterns, . . .

2 performance:unique acoustic renditioninformation in the score is interpreted, modified, added toexamples: (micro-)tempo, dynamics, intonation, . . .

3 production:aesthetic choicesediting & processingexamples: sound quality (EQ, microphone positioning),changes in timing and pitch

overview intro content ACA summary course outline

audio contenttechnical categories

audio content can be structured into 5 technical fundamentalcategories:

1 timbral: related to sound quality

examples: instrument(ation), playing technique, venue, audioprocessing, . . .

2 intensity-related: related to musical dynamics

examples: accents, loudness, . . .

3 tonal: related to pitch

examples: melody, chords, intonation, vibrato, . . .

4 temporal: related to rhythm and tempo

examples: timing, meter, rhythmic patterns, . . .

5 statistical & technical: related to signal properties

examples: amplitude distribution, number of zero crossings,. . .

overview intro content ACA summary course outline

audio contenttechnical categories

audio content can be structured into 5 technical fundamentalcategories:

1 timbral: related to sound quality

examples: instrument(ation), playing technique, venue, audioprocessing, . . .

2 intensity-related: related to musical dynamics

examples: accents, loudness, . . .

3 tonal: related to pitch

examples: melody, chords, intonation, vibrato, . . .

4 temporal: related to rhythm and tempo

examples: timing, meter, rhythmic patterns, . . .

5 statistical & technical: related to signal properties

examples: amplitude distribution, number of zero crossings,. . .

overview intro content ACA summary course outline

audio contenttechnical categories

audio content can be structured into 5 technical fundamentalcategories:

1 timbral: related to sound quality

examples: instrument(ation), playing technique, venue, audioprocessing, . . .

2 intensity-related: related to musical dynamics

examples: accents, loudness, . . .

3 tonal: related to pitch

examples: melody, chords, intonation, vibrato, . . .

4 temporal: related to rhythm and tempo

examples: timing, meter, rhythmic patterns, . . .

5 statistical & technical: related to signal properties

examples: amplitude distribution, number of zero crossings,. . .

overview intro content ACA summary course outline

audio contenttechnical categories

audio content can be structured into 5 technical fundamentalcategories:

1 timbral: related to sound quality

examples: instrument(ation), playing technique, venue, audioprocessing, . . .

2 intensity-related: related to musical dynamics

examples: accents, loudness, . . .

3 tonal: related to pitch

examples: melody, chords, intonation, vibrato, . . .

4 temporal: related to rhythm and tempo

examples: timing, meter, rhythmic patterns, . . .

5 statistical & technical: related to signal properties

examples: amplitude distribution, number of zero crossings,. . .

overview intro content ACA summary course outline

audio contenttechnical categories

audio content can be structured into 5 technical fundamentalcategories:

1 timbral: related to sound quality

examples: instrument(ation), playing technique, venue, audioprocessing, . . .

2 intensity-related: related to musical dynamics

examples: accents, loudness, . . .

3 tonal: related to pitch

examples: melody, chords, intonation, vibrato, . . .

4 temporal: related to rhythm and tempo

examples: timing, meter, rhythmic patterns, . . .

5 statistical & technical: related to signal properties

examples: amplitude distribution, number of zero crossings,. . .

overview intro content ACA summary course outline

audio contenttechnical categories

audio content can be structured into 5 technical fundamentalcategories:

1 timbral: related to sound quality

examples: instrument(ation), playing technique, venue, audioprocessing, . . .

2 intensity-related: related to musical dynamics

examples: accents, loudness, . . .

3 tonal: related to pitch

examples: melody, chords, intonation, vibrato, . . .

4 temporal: related to rhythm and tempo

examples: timing, meter, rhythmic patterns, . . .

5 statistical & technical: related to signal properties

examples: amplitude distribution, number of zero crossings,. . .

overview intro content ACA summary course outline

audio content analysissystem overview

audiosignal

featureextraction

decision,interpretation,classification,

inference

metadata

feature extractiondimensionality reductionmeaningful representation

classificationmap or convert feature tocomprehensible domain

overview intro content ACA summary course outline

audio content analysissystem overview

audiosignal

featureextraction

decision,interpretation,classification,

inference

metadata

feature extractiondimensionality reductionmeaningful representation

classificationmap or convert feature tocomprehensible domain

overview intro content ACA summary course outline

audio content analysissystem overview

audiosignal

featureextraction

decision,interpretation,classification,

inference

metadata

feature extractiondimensionality reductionmeaningful representation

classificationmap or convert feature tocomprehensible domain

overview intro content ACA summary course outline

summarylecture content

what is audio content?

what are the technical categories of interest?

what are the typical processing blocks of an ACA system?

overview intro content ACA summary course outline

summarylecture content

what is audio content?

what are the technical categories of interest?

what are the typical processing blocks of an ACA system?

overview intro content ACA summary course outline

summarylecture content

what is audio content?

what are the technical categories of interest?

what are the typical processing blocks of an ACA system?

overview intro content ACA summary course outline

course outlineoverview 1/2

1 fundamentalsdigital audio signalsconvolution & block based processingFourier transform and filterscorrelation

2 instantaneous featuresaudio pre-processingstatistical and spectral featuresfeature post-processing

3 intensitylevel & loudness

4 tonal analysisfundamental frequencytuning frequencykey and chords

overview intro content ACA summary course outline

course outlineoverview 1/2

1 fundamentalsdigital audio signalsconvolution & block based processingFourier transform and filterscorrelation

2 instantaneous featuresaudio pre-processingstatistical and spectral featuresfeature post-processing

3 intensitylevel & loudness

4 tonal analysisfundamental frequencytuning frequencykey and chords

overview intro content ACA summary course outline

course outlineoverview 1/2

1 fundamentalsdigital audio signalsconvolution & block based processingFourier transform and filterscorrelation

2 instantaneous featuresaudio pre-processingstatistical and spectral featuresfeature post-processing

3 intensitylevel & loudness

4 tonal analysisfundamental frequencytuning frequencykey and chords

overview intro content ACA summary course outline

course outlineoverview 1/2

1 fundamentalsdigital audio signalsconvolution & block based processingFourier transform and filterscorrelation

2 instantaneous featuresaudio pre-processingstatistical and spectral featuresfeature post-processing

3 intensitylevel & loudness

4 tonal analysisfundamental frequencytuning frequencykey and chords

overview intro content ACA summary course outline

course outlineoverview 2/2

5 temporal analysisonset detectiontempo & beatdownbeat & time signature

6 genre, similarity & mood

7 alignmentaudio-to-audioaudio-to-score

8 audio fingerprinting

9 structural segmentation

overview intro content ACA summary course outline

course outlineoverview 2/2

5 temporal analysisonset detectiontempo & beatdownbeat & time signature

6 genre, similarity & mood

7 alignmentaudio-to-audioaudio-to-score

8 audio fingerprinting

9 structural segmentation

overview intro content ACA summary course outline

course outlineoverview 2/2

5 temporal analysisonset detectiontempo & beatdownbeat & time signature

6 genre, similarity & mood

7 alignmentaudio-to-audioaudio-to-score

8 audio fingerprinting

9 structural segmentation

overview intro content ACA summary course outline

course outlineoverview 2/2

5 temporal analysisonset detectiontempo & beatdownbeat & time signature

6 genre, similarity & mood

7 alignmentaudio-to-audioaudio-to-score

8 audio fingerprinting

9 structural segmentation

overview intro content ACA summary course outline

course outlineoverview 2/2

5 temporal analysisonset detectiontempo & beatdownbeat & time signature

6 genre, similarity & mood

7 alignmentaudio-to-audioaudio-to-score

8 audio fingerprinting

9 structural segmentation