Making machines that make music

Making Machines that Make Music

Srihari Sriraman nilenso

should we listen to some now?

Why I do thisI Sing, I do computers

Bleeding edge research Dreamy ambitions

Interesting By-products

What this talk is about

Melody Modelling Synthesis

Generation

Melody Modelling

SynthesisGeneration

Melody ˈmɛlədi

noun a sequence of single notes that is musically satisfying; a tune.

Carnatic Music

Carnatic Music

Kalyani, Extempore MS Gopalakrishnan, Violin

Khamas, Thillana Abhishek Raghuram

Carnatic Music

South Indian classical music Ragas, Gamakams

Vocal Tradition Rich in compositions

Extempore / Manodharma

Tanpura

Veena Mridangam

The foundations of musical abstractions

in Carnatic music

ShruthiTonic note

Choice of artist Swarams are relative to this

LayaRhythm concepts

Similar to time signatures A rather mature system

Sa Ri Ga Ma Pa Da SaNi

Swarams

Sa Ri Ga Ma Pa Da SaNi

S R G M P D SN

R1 R2 R3

G1 G2 G3 M1 M2

D1 D2 D3

N1 N2 N3

Notation

Pronunciation

Variations

SwaramsThe 12 semitones

Elements of a raga Simples are sung

Prescriptive notation

Rāgā

Kaapi, Extempore TM Krishna

RagaHas a name

Rule to ascend Rule to descend

Not necessarily symmetric Not necessarily linear Grouped into families

RagaHas a name

Rule to ascend Rule to descend

Not necessarily symmetric Not necessarily linear Grouped into families

Demo

of the fundamental abstractions.

Tools & Libraries

Fuzzy searchFor indic languages

Needs to be fast Primary stitching mechanism Helps with multi-source data

A quick recapPlay the scale of a raga

Fuzzy find a raga Play a phrase

Play a phrase in the context of a raga Play some prescriptive notation

But..

..that doesn’t sound like Carnatic music, does it?

Synthesis

Enter Melographs

Me

Machine

another phrase

Me

Machine

Prescriptive vs Descriptive

Gamakams

Sphuritam

Orikai

Jaaru

Kampitam

Sphuritam Nokku Ravai

Kandippu Ullasitam Etra-jaru

Iraka-jaru Odukkal

Orikai Vali

Kampitam

Gamakams in SSP

Gamakams in SSP

Gaayaka

| S, N D | N S R G |

((P S,,)) , ((S , S>>> S)) -((D. S. D)) ((S , S>> S))- S R ((G<< G , ,))

Subramanian, 2009 Database of phrases

Automatic Gamakam feature – guided

Modelling Gamakams

Me

Machine

Back to this…

PASR

Srikumar 2013 Pitch, Attack, Sustain, Release

Vector specifies the PASR vars for each prescriptive note

Me

Machine

Rendering PASR…

Rendering PASR…

Generation

Random | Within a raga

Random | Within a raga

Get data

Kosha An Open Carnatic Music Database

http://github.com/ssrihari/kosha

Study data

Melographs

Melographs

kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram

kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram

Pitch Histograms

Pitch Histograms

kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg

Pitch HistogramsKalyani - Vocal Kalyani - Violin

Mohana - MandolinMohana - Vocal

Revati - Vocal Revati - Instrumental

Extract Music Information

Midi Histogram

Normalised Midi Histogram

Tonic note identification

Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012) A knowledge based signal processing approach to tonic identification in indian classical music.

Bellur, A., and H. A. Murthy (2013) Automatic tonic identification in classical music using melodic characteristics and tuning of the drone.

Srihari, S. (2016) * Pick the most frequent note, it mostly just works.

* not really, no

Tonic note identification

Tonic note identification

Swaram HistogramKalyani

S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S

Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S

Revati S, R1, M1, P, N2, S. S., N2, P, M1, R1, S

Mohana S, R2, G3, P, D2, S. S., D2, P, G3, R2, S

Generation with weighted probabilities

In comparison with random

Random

Single Swaram

Weighted

Melody insights #1Tonic note is prominent

Sa, and Pa have higher and sharper peaks Other note peaks are blunt

Probabilities of all swarams in a raga are not the same Probabilities across octaves are not the same

Two swaram probabilities

Two swaram probabilities

Prominence of adjacency Encoded rules of Arohanam and Avarohanam

Two swaram probabilities

Single swaram vs Two swarams

Two Swaram

Weighted

Single Swaram

Weighted

Melody insights #2Swarams close to each other are more melodious The rules of Arohanam, Avarohanam are encoded

We begin to see gamakams Sometimes, the in-between is worse than either extreme

Three swaram probabilities and more

A simple markov chain

First Order Matrix

https://en.wikipedia.org/wiki/Markov_chain#Music

Markov Chains in Music

https://github.com/rm-hull/markov-chains

Second Order Matrix

Markov Chains in Music

Melody insights #3Generic markov chains don’t really work

LSTMs also don’t work, probably

By-products

Automatic Transcription

Automatic Transcription

(:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1 :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 :..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1 :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3 :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1 :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3

:..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)

Raga Identification

Goodness of fit test

:base mohanam-base :samples mohanam-files (12.39 3.84 11.14 6.46 9.88 7.02 9.41 12.61 13.22 1.58)

:base mohanam-base :samples kalyani-files (10.95 28.66 25.61 15.26 27.32 21.53 16.42 18.58 24.80 23.80)

:base mohanam-base :samples revati-files (46.56 57.19 65.69 55.21 38.61 78.10 56.27 42.99 70.92 58.39)

Raga Identification

Revati sample vs

Revati base

Mohana sample vs

Revati base

What nextModel insights as melodic abstractions

Use synthesis models with generative music Experiment with Rhythm Synthesise Human Voice

Deep learning (Recurrent variational auto encoders)

Is this music though?

Behag

Dasarapada Abhishek Raghuram

Making Machines that Make Music

Srihari Sriraman nilenso