Representation and Segmentation of Melodies in Indian Art …...S. Gulati, J. Serrà, K. K. Ganguli,...

Representation and Segmentation of Melodies in Indian Art Music

3rd CompMusic Workshop, IIT-Madras, Chennai, India 13th Dec, 2013

Sankalp Gulati

Supervisor: Prof. Xavier Serra PhD Candidate, MTG-UPF, Barcelona, Spain

Tonic Identification

Rhythm Analysis

Melodic Feature Extraction

Melody Representation

Melody Segmentation

Melodic Similarity

R�g characterization

& Music similarity

measures

Data extracted knowledge Expert knowledge

Redundancy reduction

Pattern Extraction

Listening tests

Dunya integration

Audio Metadata

Annotations

Melodic motivic discovery Evaluation

Melodic motives representation

Melodic motives similarity graph

•  Rhythmic Variations (overall

timing, non-linear time variations)

•  Extent of pitch variations •  Melodic segmentation

information

•  Nyas swaras •  Motif transpositions •  Salient melodic movements

Block diagram of the proposed semi-supervised methodology for melodic motif discovery

Tonic Identification

Rhythm Analysis

Melodic Feature Extraction

Melody Representation

Melody Segmentation

Melodic Similarity

R�g characterization

& Music similarity

measures

Data extracted knowledge Expert knowledge

Redundancy reduction

Pattern Extraction

Listening tests

Dunya integration

Audio Metadata

Annotations

Melodic motivic discovery Evaluation

Melodic motives representation

Melodic motives similarity graph

•  Rhythmic Variations (overall

timing, non-linear time variations)

•  Extent of pitch variations •  Melodic segmentation

information

•  Nyas swaras •  Motif transpositions •  Salient melodic movements

Block diagram of the proposed semi-supervised methodology for melodic motif discovery

Tonic Identification

Rhythm Analysis

Melodic Feature Extraction

Melody Representation

Melody Segmentation

Melodic Similarity

R�g characterization

& Music similarity

measures

Data extracted knowledge Expert knowledge

Redundancy reduction

Pattern Extraction

Listening tests

Dunya integration

Audio Metadata

Annotations

Melodic motivic discovery Evaluation

Melodic motives representation

Melodic motives similarity graph

•  Rhythmic Variations (overall

timing, non-linear time variations)

•  Extent of pitch variations •  Melodic segmentation

information

•  Nyas swaras •  Motif transpositions •  Salient melodic movements

Block diagram of the proposed semi-supervised methodology for melodic motif discovery

Tonic Identification: JNMR article

Sengupta, R., Dey, N., Nag, D., DaĶa, A. K., & Mukerjee, A. (2005). Automatic tonic ( SA ) detection algorithm in Indian classical vocal music. In National Symposium on Acoustics (pp. 1–5).

Ranjani, H. G., Arthi, S., & Sreenivas, T. V. (2011). Carnatic music analysis: Shadja, swara identification and raga verification in Alapana using stochastic models. Applications of Signal Processing to Audio and Acoustics (WASPAA), IEEE Workshop , 29–32.

Salamon, J., Gulati, S., & Serra, X. (2012). A multipitch approach to tonic identification in Indian classical music. In Proc. of Int. Conf. on Music Information Retrieval (ISMIR) (pp. 499–504). Gulati, S., Salamon, J., & Serra, X. (2012). A two-stage approach for tonic identification in Indian art music. In 2nd CompMusic Workshop (pp. 119–127).

Bellur, A., Ishwar, V., Serra, X., & Murthy, H. (2012). A knowledge based signal processing approach to tonic identification in Indian classical music. In 2nd CompMusic Workshop (pp. 113–118).

Gulati, S., Bellur, A.,Salamon, J., Ranjani, H.G., Ishwar, V., Murthy, H. A. and Serra, X., ‘Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation’, Journal of New Music Research (in press).

Tonic Identification: Datasets

Gulati, S., Bellur, A.,Salamon, J., Ranjani, H.G., Ishwar, V., Murthy, H. A. and Serra, X., ‘Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation’, Journal of New Music Research (in press).

Tonic Identification: Results

Gulati, S., Bellur, A.,Salamon, J., Ranjani, H.G., Ishwar, V., Murthy, H. A. and Serra, X., ‘Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation’, Journal of New Music Research (in press).

Tonic Identification: Analysis

Performance as a function of length Performance as a function of category

Error type as a function of dataset

Gulati, S., Bellur, A.,Salamon, J., Ranjani, H.G., Ishwar, V., Murthy, H. A. and Serra, X., ‘Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation’, Journal of New Music Research (in press).

Tonic Identification

Rhythm Analysis

Melodic Feature Extraction

Melody Representation

Melody Segmentation

Melodic Similarity

R�g characterization

& Music similarity

measures

Data extracted knowledge Expert knowledge

Redundancy reduction

Pattern Extraction

Listening tests

Dunya integration

Audio Metadata

Annotations

Melodic motivic discovery Evaluation

Melodic motives representation

Melodic motives similarity graph

•  Rhythmic Variations (overall

timing, non-linear time variations)

•  Extent of pitch variations •  Melodic segmentation

information

•  Nyas swaras •  Motif transpositions •  Salient melodic movements

Block diagram of the proposed semi-supervised methodology

Melody Extraction and Representation

q  Melody (lead artist/voice) §  Pitch (Fundamental Frequency, F0) §  Timbre §  Loudness

Salamon, Justin, and Emilia Gómez. "Melody extraction from polyphonic music signals using pitch contour characteristics." Audio, Speech, and Language Processing, IEEE Transactions on 20.6 (2012): 1759-1770.

Rao, Vishweshwara, and Preeti Rao. "Vocal melody extraction in the presence of pitched accompaniment in polyphonic music." Audio, Speech, and Language Processing, IEEE Transactions on 18.8 (2010): 2145-2154.

De Cheveigné, A., & Kawahara, H. (2002). YIN, a fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 111, 1917.

Original Audio

Predominant Voice

F0 of the Predominant Voice

Loudness and Timbral Facets

Loudness and Timbre

q  Zwicker, E. (1977). Procedure for calculating loudness of temporally variable sounds. The Journal of the Acoustical Society of America, 62(3), 675–682.

q  Röbel, A., & Rodet, X. (2005). Efficient spectral envelope estimation and its application to pitch shifting and envelope preservation. In Proc. dafx.

Predominant F0 Frequency estimation

Synthesize predominant melodic source

Loudness feature extraction

Timbre feature extraction

Audio

Evaluation: Essentia Pitch Extraction q  Predominant melody extraction (F0)

§  6 Hindustani music pieces ~45 mins

Bogdanov, D., Wack, N., Gómez, E., Gulati, S., Herrera, P., Mayor, O., … Serra, X. (2013). Essentia: an audio analysis library for music information retrieval. In Proc. of int. society for music information retrieval conf. (ISMIR) (pp. 493–498).

Salamon, Justin, and Emilia Gómez. "Melody extraction from polyphonic music signals using pitch contour characteristics." Audio, Speech, and Language Processing, IEEE Transactions on 20.6 (2012): 1759-1770.

Rao, Vishweshwara, and Preeti Rao. "Vocal melody extraction in the presence of pitched accompaniment in polyphonic music." Audio, Speech, and Language Processing, IEEE Transactions on 18.8 (2010): 2145-2154.

Loudness and Timbre: Motif Detection q  Motif similarity: DTW

§  mnDP (#72 instances) §  DnDP (#56 instances)

q  # Combinations: §  Postives: 4096 §  Negative: 4032

q  Dataset: IIT Bombay §  Raga: Alhaiya Bilawal §  # Performances: 5 §  # Artists: 4

Ross, J. C., & Rao, P. (2012). Detection Of Raga-Characteristic Phrases From Hindustani Classical Music Audio. In 2nd CompMusic Workshop (pp. 133–138).

ROC: Pitch

ROC: Timbre

ROC: Loudness

Tonic Identification

Rhythm Analysis

Melodic Feature Extraction

Melody Representation

Melody Segmentation

Melodic Similarity

R�g characterization

& Music similarity

measures

Data extracted knowledge Expert knowledge

Redundancy reduction

Pattern Extraction

Listening tests

Dunya integration

Audio Metadata

Annotations

Melodic motivic discovery Evaluation

Melodic motives representation

Melodic motives similarity graph

•  Rhythmic Variations (overall

timing, non-linear time variations)

•  Extent of pitch variations •  Melodic segmentation

information

•  Nyas swaras •  Motif transpositions •  Salient melodic movements

Block diagram of the proposed semi-supervised methodology

Melody Segmentation q  Task, context and music style/tradition

dependent q  Do we need it? (motif discovery)

§ At what stage of processing?

q  Melodic motifs <> nyas svar

q  Melodic segmentation: estimating boundaries of nyas svars

Ross, J. C., & Rao, P. (2012). Detection Of Raga-Characteristic Phrases From Hindustani Classical Music Audio. In 2nd CompMusic Workshop (pp. 133–138).

Nyas Svar Segmentation

Nyas Svar Segmentation

Nyas Svar Segmentation

Nyas Svar Segmentation

S. Gulati, J. Serrà, K. K. Ganguli, and Xavier Serra, “LANDMARK DETECTION IN HINDUSTANI MUSIC MELODIES” Submitted to ICASSP 2013

1 2 3 4 5 6 7 8 9 10

850

900

950

1000

1050

1100

1150

T1

Sn

Sn-1

Sn+1

ε ρ1 T2 T3

ρ2

T4 T7 T8 T9

T5 T6

Nyās segment

Time (s)

Fund

amen

tal F

requ

ency

(cen

ts)

Tonic Identification

Rhythm Analysis

Melodic Feature Extraction

Melody Representation

Melody Segmentation

Melodic Similarity

R�g characterization

& Music similarity

measures

Data extracted knowledge Expert knowledge

Redundancy reduction

Pattern Extraction

Listening tests

Dunya integration

Audio Metadata

Annotations

Melodic motivic discovery Evaluation

Melodic motives representation

Melodic motives similarity graph

•  Rhythmic Variations (overall

timing, non-linear time variations)

•  Extent of pitch variations •  Melodic segmentation

information

•  Nyas swaras •  Motif transpositions •  Salient melodic movements

Block diagram of the proposed semi-supervised methodology

Nyas Identification

S. Gulati, J. Serrà, K. K. Ganguli, and Xavier Serra, “LANDMARK DETECTION IN HINDUSTANI MUSIC MELODIES” Submitted to ICASSP 2013

Predominant melody extraction Tonic identification

Audio

Nyās svars

Melody representation

Histogram computation

Melody segmentation

Svar identification Segmentation

Local Feature extraction Contextual Local + Contextual

Segment classification Segment fusion Segment classification and fusion

Nyas Identification q  Local Features (#9)

§  Segment length

§  Mean, variance of pitch values

§  Mean, variance of the differences in adjacent peak locations in pitch sequence

§  Mean, variance of peak amplitudes of pitch sequence

§  Temporal centroid

§  Flatness measure (output of segmentation method)

Nyas Identification q  Local Features (#3)

§  Segment length

§  Mean, variance of pitch values

§  Mean, variance of the differences in adjacent peak locations in pitch sequence

§  Mean, variance of peak amplitudes of pitch sequence

§  Temporal centroid

§  Flatness measure (output of segmentation method)

Nyas Identification q  Contextual Features (#24)

§  Segment length / (longest segment length in breath phrase)

§  Segment length / (length of the breath phrase)

§  Segment length / (length of the previous segment)

§  Segment length / (length of the following segment)

§  Duration between the ending and succeeding silence

§  Duration between the starting and preceding silence

§  All local features of adjacent segments

Nyas Identification q  Contextual Features (#15)

§  Segment length / (longest segment length in breath phrase)

§  Segment length / (length of the breath phrase)

§  Segment length / (length of the previous segment)

§  Segment length / (length of the following segment)

§  Duration between the ending and succeeding silence

§  Duration between the starting and preceding silence

§  All local features of preceding segments

Nyas Identification: Baselines q  Segmentation

§  Piece-wise linear segmentation

q  Classification § DTW + kNN classification

q  Several random baselines

Nyas Identification: Dataset q  20 recordings of Hindustani music

§  15 Polyphonic: CompMusic collection §  5 Monophonic: Kaustuv’s recordings

q  Unique artists: 8 q  Unique Rāgs: 16 q  Number of nyās segments: 1257 q  Duration of nyās segments

§  Range: 150 ms – 16.7 s § Mean: 2.4 s § Median: 1.4 s

Nyas Identification: Results

Local features with

Proposed Segmentation

S. Gulati, J. Serrà, K. K. Ganguli, and Xavier Serra, “LANDMARK DETECTION IN HINDUSTANI MUSIC MELODIES” Submitted to ICASSP 2013

Nyas Identification: Results

Local features with

Proposed Segmentation

S. Gulati, J. Serrà, K. K. Ganguli, and Xavier Serra, “LANDMARK DETECTION IN HINDUSTANI MUSIC MELODIES” Submitted to ICASSP 2013

Tonic Identification

Rhythm Analysis

Melodic Feature Extraction

Melody Representation

Melody Segmentation

Melodic Similarity

R�g characterization

& Music similarity

measures

Data extracted knowledge Expert knowledge

Redundancy reduction

Pattern Extraction

Listening tests

Dunya integration

Audio Metadata

Annotations

Melodic motivic discovery Evaluation

Melodic motives representation

Melodic motives similarity graph

•  Rhythmic Variations (overall

timing, non-linear time variations)

•  Extent of pitch variations •  Melodic segmentation

information

•  Nyas swaras •  Motif transpositions •  Salient melodic movements

Block diagram of the proposed semi-supervised methodology for melodic motif discovery

Melodic Motif Discovery: Brute-force

Tonic Identification

Rhythm Analysis

Melodic Feature Extraction

Melody Representation

Melody Segmentation

Melodic Similarity

R�g characterization

& Music similarity

measures

Data extracted knowledge Expert knowledge

Redundancy reduction

Pattern Extraction

Listening tests

Dunya integration

Audio Metadata

Annotations

Melodic motivic discovery Evaluation

Melodic motives representation

Melodic motives similarity graph

•  Rhythmic Variations (overall

timing, non-linear time variations)

•  Extent of pitch variations •  Melodic segmentation

information

•  Nyas swaras •  Motif transpositions •  Salient melodic movements

Block diagram of the proposed semi-supervised methodology for melodic motif discovery