Music Structure Analysis · Example: Brahms Hungarian Dance No. 5 (Ormandy) Time (seconds) Music Structure Analysis Example: Brahms Hungarian Dance No. 5 (Ormandy) Time (seconds)

Meinard Müller, Frank Zalkow

Audio Structure Analysis

International Audio Laboratories [email protected], [email protected]

Tutorial T1Fundamentals of Music Processing: An Introduction using Python and Jupyter Notebooks

Music Structure AnalysisExample: Brahms Hungarian Dance No. 5 (Ormandy)

Time (seconds)

Music Structure AnalysisExample: Brahms Hungarian Dance No. 5 (Ormandy)

Time (seconds)

A1 A2 B1 B2 C A3 B3 B4

Music Structure Analysis

Stanzas of a folk song

Intro, verse, chorus, bridge, outro sections of a pop song

Exposition, development, recapitulation, coda of a sonata

Musical form ABACADA … of a rondo

General goal: Divide an audio recording into temporal segments corresponding to musical parts and group these segments into musically meaningful categories.

Examples:

Music Structure Analysis

Homogeneity:

Novelty:

Repetition:

General goal: Divide an audio recording into temporal segments corresponding to musical parts and group these segments into musically meaningful categories.

Challenge: There are many different principles for creating relationships that form the basis for the musical structure.

Consistency in tempo, instrumentation, key, …

Sudden changes, surprising elements …

Repeating themes, motives, rhythmic patterns,…

Feature RepresentationExample: Brahms Hungarian Dance No. 5 (Ormandy)

Time (seconds)

A1 A2 A3B1 B2 B3 B4C

Feature Representation


Feature extractionChroma (Harmony)

Example: Brahms Hungarian Dance No. 5 (Ormandy)

Time (seconds)





G minor G minor

D

GBb

Time (seconds)





G minor G major G minor

D

GBb

D

GB

Time (seconds)

Self-Similarity Matrix (SSM)

General idea: Compare each element of the feature sequence with each other element of the feature sequence based on a suitable similarity measure.

→ Quadratic self-similarity matrix

Self-Similarity Matrix (SSM)Example: Brahms Hungarian Dance No. 5 (Ormandy)







G major

G m

ajor






Slower

Fast

er


Fast

er

Slower


Idealized SSM


Idealized SSM

Blocks: Homogeneity

Paths: Repetition

Corners: Novelty

SSM EnhancementChallenge: Presence of musical variations

Idea: Enhancement of path structure

Fragmented paths and gaps

Paths of poor quality

Regions of constant (low) cost

Curved paths

SSM Enhancement

Time (samples)

Tim

e (s

ampl

es)

Path Enhancement

SSM Enhancement

Time (samples)

Tim

e (s

ampl

es)

Path Enhancement

Diagonal smoothing

SSM Enhancement

Time (samples)

Tim

e (s

ampl

es)

Path Enhancement

Diagonal smoothing Multiple filtering

SSM Enhancement

Time (samples)

Tim

e (s

ampl

es)

Path Enhancement

Diagonal smoothing Multiple filtering Thresholding (relative) Scaling & penalty

SSM Enhancement

Time (samples)

Tim

e (s

ampl

es)

Further Processing

Path extraction

SSM Enhancement

Time (samples)

Tim

e (s

ampl

es)

Further Processing

Path extraction Pairwise relations

100 200 300 400

1

Time (samples)

234567

SSM Enhancement

Time (samples)

Tim

e (s

ampl

es)

Further Processing

Path extraction Pairwise relations Grouping (transitivity)

100 200 300 400

1

Time (samples)

234567

100 200 300 400Time (samples)

SSM Enhancement

Time (samples)

Tim

e (s

ampl

es)

Further Processing

Path extraction Pairwise relations Grouping (transitivity)

100 200 300 400

1

Time (samples)

234567

Novelty-Based Segmentation Novelty-Based Segmentation

Find instances where musicalchanges occur.

Find transition between subsequent musical parts.

General goals: Idea (Foote):

Use checkerboard-like kernelfunction to detect corner pointson main diagonal of SSM.

Novelty-Based Segmentation

Idea (Foote):



Idea (Foote):



Idea (Foote):



Idea (Foote):



Idea (Foote):


Novelty function using


Idea (Foote):




SSM Enhancement

V1 V2 V3 V4 V5 V6 V7 V8 OBI

Example: Zager & Evans “In The Year 2525”

SSM EnhancementExample: Zager & Evans “In The Year 2525”Missing relations because of transposed sections

SSM EnhancementExample: Zager & Evans “In The Year 2525”Idea: Cyclic shift of one of the chroma sequences

One semitone up

SSM EnhancementExample: Zager & Evans “In The Year 2525”Idea: Cyclic shift of one of the chroma sequences

Two semitones up

SSM EnhancementExample: Zager & Evans “In The Year 2525”Idea: Overlay Transposition-invariant SSM& Maximize

Audio Thumbnailing Audio Thumbnailing



General goal: Determine the most representative section(“Thumbnail”) of a given music recording.

V1 V2 V3 V4 V5 V6 V7 V8 OBI

Example: Zager & Evans “In The Year 2525”

Thumbnail is often assumed to be the most repetitive segment

Audio Thumbnailing

Two steps Paths of poor quality (fragmented, gaps) Block-like structures Curved paths

1. Path extraction

2. Grouping Noisy relations(missing, distorted, overlapping)

Transitivity computation difficult

Both steps are problematic!

Main idea: Do both, path extraction and grouping, jointly One optimization scheme for both steps Stabilizing effect Efficient

Audio Thumbnailing

Main idea: Do both path extraction and grouping jointly

For each audio segment we define a fitness value

This fitness value expresses “how well” the segmentexplains the entire audio recording

The segment with the highest fitness value isconsidered to be the thumbnail

As main technical concept we introduce the notion of a path family

0 50 100 150 2000

20

40

60

80

100

120

140

160

180

200

−2

−1.5

−1

−0.5

0

0.5

1

Fitness Measure

Enhanced SSM

Fitness Measure

Consider a fixed segment

Path over segment

Fitness Measure


Path over segment Induced segment Score is high

Path over segment

Fitness Measure

Path over segment



A second path over segment Induced segment Score is not so high

Fitness Measure

Path over segment



A second path over segment Induced segment Score is not so high

A third path over segment Induced segment Score is very low

Fitness Measure

Path family


A path family over a segmentis a family of paths such thatthe induced segments do not overlap.

Fitness Measure

Path family

This is not a path family!



Fitness Measure

Path family

This is a path family!



(Even though not a good one)

Fitness Measure

Optimal path family


Fitness Measure

Optimal path family


Consider over the segmentthe optimal path family,i.e., the path family havingmaximal overall score.

Call this value:Score(segment)

Note: This optimal path family can be computedusing dynamic programming.

Fitness Measure

Optimal path family


Consider over the segmentthe optimal path family,i.e., the path family havingmaximal overall score.

Call this value:Score(segment)

Furthermore consider theamount covered by theinduced segments.

Call this value:Coverage(segment)

Fitness Measure

Fitness


P := R :=

Score(segment)Coverage(segment)

Fitness Measure

Fitness


Self-explanation are trivial!

P := R :=


Fitness Measure

Fitness



Subtract length of segment

P := R :=


- length(segment) - length(segment)

Normalize( )

Fitness Measure

Fitness



Subtract length of segment

Normalization

P := R :=


- length(segment)- length(segment)

]1,0[]1,0[

Normalize( )

Fitness Measure

Fitness


F := 2 • P • R / (P + R)Fitness(segment)

Normalize( ) Normalize( )

P := R :=


- length(segment)- length(segment)

]1,0[]1,0[

Thumbnail

Segment center

Segm

ent l

engt

h

Fitness Scape Plot

Segment length

Segment center

Fitness

Thumbnail

Segment center

Fitness Scape Plot

Fitness(segment)

Segment length

Segment center

Fitness

Segm

ent l

engt

h

Thumbnail

Segment center

Fitness Scape PlotFitness

Segm

ent l

engt

h

Thumbnail

Segment center

Fitness Scape Plot

Note: Self-explanations are ignored → fitness is zero

Fitness

Segm

ent l

engt

h

Thumbnail

Segment center

Fitness Scape Plot

Thumbnail := segment having the highest fitness

FitnessSe

gmen

t len

gth

ThumbnailFitness Scape Plot


Fitness


Fitness




Fitness




Fitness




Scape Plot


Scape Plot

Coloring accordingto clustering result(grouping)


Scape Plot


Coloring accordingto clustering result(grouping)


Structure Analysis

Conclusions

Representations

Structure Analysis

AudioMIDIScore

Conclusions

Representations

Musical Aspects

Structure Analysis

TimbreTempoHarmony

AudioMIDIScore

Conclusions

Representations

Segmentation Principles

Musical Aspects

Structure Analysis

HomogeneityNoveltyRepetition

TimbreTempoHarmony

AudioMIDIScore

Conclusions

Temporal and Hierarchical Context

Representations

Segmentation Principles

Musical Aspects

Structure Analysis

HomogeneityNoveltyRepetition

TimbreTempoHarmony

AudioMIDIScore

Conclusions Links SM Toolbox (MATLAB)

http://www.audiolabs-erlangen.de/resources/MIR/SMtoolbox/

MSAF: Music Structure Analysis Framework (Python)https://github.com/urinieto/msaf

SALAMI Annotation Datahttp://ddmal.music.mcgill.ca/research/salami/annotations

LibROSA (Python)https://librosa.github.io/librosa/

Evaluation: mir_eval (Python)https://craffel.github.io/mir_eval/

Deep Learning: Boundary DetectionJan Schlüter (PhD thesis)

Book: Fundamentals of Music Processing

Meinard MüllerFundamentals of Music ProcessingAudio, Analysis, Algorithms, Applications483 p., 249 illus., hardcoverISBN: 978-3-319-21944-8Springer, 2015

Accompanying website: www.music-processing.de

Book: Fundamentals of Music Processing

Meinard MüllerFundamentals of Music ProcessingAudio, Analysis, Algorithms, Applications483 p., 249 illus., hardcoverISBN: 978-3-319-21944-8Springer, 2015

Accompanying website: www.music-processing.de

Documents

Music Structure Analysis · Example: Brahms Hungarian Dance No. 5 (Ormandy) Time (seconds) Music Structure Analysis Example: Brahms Hungarian Dance No. 5 (Ormandy) Time (seconds)