36
LyriSys: An Interactive Support System for Writing Lyrics Based on Topic Transition Kento Watanabe 1 Yuichiroh Matsubayashi 1 , Kentaro Inui 1 , Tomoyasu Nakano 2 , Satoru Fukayama 2 , Masataka Goto 2 1 Graduate School of Information Sciences, Tohoku University, Japan 2 National Institute of Advanced Industrial Science and Technology (AIST), Japan

LyriSys: An Interactive Support System for Writing Lyrics Based on Topic Transition

Embed Size (px)

Citation preview

LyriSys: An Interactive Support System

for Writing Lyrics Based on Topic TransitionKento Watanabe1

Yuichiroh Matsubayashi1, Kentaro Inui1, Tomoyasu Nakano2, Satoru Fukayama2, Masataka Goto2

1Graduate School of Information Sciences, Tohoku University, Japan 2National Institute of Advanced Industrial Science and Technology (AIST), Japan

Difficulty of Writing Lyrics

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 2

It is difficult to consider many techniques at the same time.

Select words whose syllables correspond to the melody notes.

[Austin+2010, Ueda 2010]

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

I re-mem-ber our love And I re-mem-ber eve- ry word un- spo- ken1 3 1 1 1 1 3 2 1 3Syllable Counts

Lyrics

Difficulty of Writing Lyrics

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 3

It is difficult to consider many techniques at the same time.

[Austin+2010, Ueda 2010]

Block1 Block2 Block3I drop into the oceanMy friend the stormy seaAs long as the blindnessWon't leave me tonight

I felt the wrath of the chosen oneI wanted forgiveness but was given noneLet all the lights scream in my eyesThat 's how I want it to fee

I fell in loveI fell in love

Topic:Dark

Topic:Scene

Topic:Sweet Love

• Verse-Bridge-Chorus segment (called Block).• Text in each block has an topic.• The topic transition constitutes a story.

Development of a Lyrics Writing Support System

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 4

System enables rich interaction by the topic transition model

Topic Transition Model

Language Model

Sweet love

Scene

Life

Exciting

Ardent Love

……ni

ght

day

light …love

know

baby … pr

obab

ility

wordword

prob

abili

ty

SYSTEM INTELIGENCE

Over 100000 lyricsI drop into the oceanMy friend the stormy seaAs long as the blindnessWon't leave me tonight…

TRAINING DATA

USER INPUT

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

Syllable counts

e.g., “Time”→”Life”→”Love”

Number of blocksNumber of lines

Structure of an entire song…

Sequence of topicse.g.,1-3-1-1

SYSTEM OUTPUTRecommended lyrics

If tomorrow never comesI remember the dayYou realize time flies

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

Unsupervised Learning

Trial-and-Error

System Introduction

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 5

Example of Created Lyrics

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 6

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . . 1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .If you rea- lly un- der- stand I pro- mise you’-ll be An ex- traor- di- nar- y man

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

If you really understand I promise you'll be An extraordinary man

I don't really understand I promise you'll be Oh supernatural love

Example of lyrics when the user uses our system

Fully automatically recommended lyrics

The love phrases “I promise…” and ”supernatural love” were recommended when the topic was ⟨sweet love⟩.

1 1 2 3 1 2 2 1 1 5 1SyllableLyrics

Inputted Topic: Sweet Love

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 7

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 8

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 9

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 10

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 11

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 12

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 13

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 14

Writing Support System: LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 15

Implementation of LyriSys

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 16

Recommendation Algorithm

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 17

LyriSys searches the word strings so that the word probability is large according to the beam search.

IYou

So…

1 syllable words

need

don’twhen…

1 syllable words

always

reallybetter…

2 syllable words

remember

everything

understand…

3 syllable words

: Word probability 𝑃(𝑤𝑜𝑟𝑑)|𝑤𝑜𝑟𝑑)+,, 𝑡𝑜𝑝𝑖𝑐)

Example of beam search when syllable count is “1-1-2-3” and topic is “sweet love”.

How does the system handle topic

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 18

Block1 Block2 Block3I drop into the oceanMy friend the stormy seaAs long as the blindnessWon't leave me tonight

I felt the wrath of the chosen oneI wanted forgiveness but was given noneLet all the lights scream in my eyesThat 's how I want it to fee

I fell in loveI fell in love

Topic:Dark

Topic:Scene

Topic:Sweet Love

𝑃(𝑡𝑜𝑝𝑖𝑐3|𝑡𝑜𝑝𝑖𝑐,) 𝑃(𝑡𝑜𝑝𝑖𝑐4|𝑡𝑜𝑝𝑖𝑐3)

∏𝑃(𝑤𝑜𝑟𝑑|𝑡𝑜𝑝𝑖𝑐,) ∏𝑃(𝑤𝑜𝑟𝑑|𝑡𝑜𝑝𝑖𝑐3) ∏𝑃(𝑤𝑜𝑟𝑑|𝑡𝑜𝑝𝑖𝑐4)

The story is represented by two kinds of probabilities

Transition probability

Word probability

Enhanced Hidden Markov Model [watanabe+ 2014]

How to Get Probabilities ?

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 19

𝑃(𝑤𝑜𝑟𝑑|𝑡𝑜𝑝𝑖𝑐)𝑃(𝑡𝑜𝑝𝑖𝑐)|𝑡𝑜𝑝𝑖𝑐)+,)Enhanced HMM [Watanabe+2014]

(1) Transition prob: (2) Word prob:

……

nigh

tda

ylig

ht …love

know

baby …

prob

abili

ty

wordword

SceneExcitingSlangTime

Sweet love

LifeForeign

DarkReligious

Ardent love

Scen

eEx

citin

gSl

ang

Tim

e

Swee

t lov

e

Life

Fore

ign

Dar

kRe

ligio

us

Ard

ent l

ove

Transition Matrix

𝑡𝑜𝑝𝑖𝑐)

𝑡𝑜𝑝𝑖𝑐 )+,

Prev

ious

73 5 4 1 7 4 5 1 1 1

6 62 3 1 15 1 5 1 5 1

9 5 42 6 8 9 13 1 5 1

1 1 3 85 1 5 3 1 2 1

6 9 2 1 72 1 3 1 7 1

8 1 6 8 1 71 4 1 1 16 5 5 2 7 2 67 1 4 1

1 1 0 1 1 1 1 93 1 1

2 6 2 2 13 0 4 0 69 1

2 4 2 3 4 1 3 0 4 78 (%)

System automatically learns (1) typical topic transitions and (2) semantically appropriate wordings

from a large collection of human lyrics.

prob

abili

ty

Over 10000 lyrics

Unsupervised learning

I drop into the oceanMy friend the stormy seaAs long as the blindnessWon't leave me tonight…

User Feedback

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 20

User Feedback: SetupnWriting Japanese Lyrics

p5 Japanese (One user was a school teacher of music).pWe randomly selected songs from RWC music database.

n4 Tasks1. Base line : Without interface2. Method 1: Fully Automatic lyrics generation

Users can only select topics and input syllables,cannot edit/select the generated lyrics.

3. Method 2: Interaction without topicSystem calculates the simple N-gram probability,cannot handle topic.

4. Proposed : LyriSys2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 21

User Feedback: Results

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 22

Task Positive Comments Negative Comments(Base line)Without system

In comparison to other tasks, it was comfortable in not specifying the number of syllable.

It was difficult to come up with the words that satisfy the melody of song, because of lacking in vocabulary.

(Method 1)Automatically generation

It was easy to write the lyrics because I didn’t need to determine which words to use.

I sometimes felt boring because users couldn’t edit the generated lyrics.

(Method 2)Interactionwithout topic

It was useful to select the candidate of lyrics when the generated result was partially good.

It was difficult to write the lyrics that represent the story, because only a limited variety of words are generated.

(Proposed)LyriSys

1) In comparison to the previous method 2, selecting topics made it easy to write the lyrics that specifies my intention. 2) The generated lyrics are more expressive than the result of other interface because of the consideration of topic.

1) The list of the 10 topics was too restricted and coarse-grained. 2) Although the system generates an abstract story, I thought that it would be interesting if the system could generate a concrete story.

Conclusions and Future Work

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 23

Our interaction designSystem is designed as a recommendation system.User can input the structure of entire an song and the story.

Our system intelligenceSystem automatically learns typical topic transitions and semantically appropriate wordings from a large collection of human lyrics.

Novel lyric-writing system: LyriSys

It might be too much of a burden for the user to specify the number of syllable counts.We plan to introduce extended functions on the Web.

Next Step

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 24

Appendix

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 25

Related worksMany researchers have studied rhyme of lyrics

[Nichols+09 Genzel+10, Abe+10 Ramakrishnan+10, Berbieri+12].

Prior studies report systems that can generate only a single line of lyrics independently of the rest [Abe+2012]

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 26

Our system has a strong advantage in capturing topic transition (i.e., story) of an entire lyrics.

Story Estimation

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 27

What topics were learned ?

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 28

z Label Representative words in each topic: top words from 𝑃(𝑤𝑜𝑟𝑑|𝑧)1 Sweet love love,know,baby,want,need,make,wan,feel,one,tell,give,more,heart,good,only,hold,'cause,please,

kiss,mine,stay,true,cry,crazy,touch,someone,miss,somebody,mean,woman,nobody,enough,care

2 scene night,day,light,eye,fall,dream,sun,sky,rain,shine,fly,into,home,star,walk,blue,wind,burn,fire,moon,christmas,cold,morning,watch,sleep,tear,dark,water,close,summer,open,remember,river,

3 exciting yeah,down,come,hey,gon,ooh,back,rock,keep,right,dance,tonight,stop,here,run,roll,alright,whoa,everybody,round,turn,ready,move,music,ride,party,bring,doo,c'mon,slow,beat,high,gim,ohh

4 slang nigga,shit,fuck,bitch,cause,money,niggaz,'em,ass,hit,real,hoe,game,wit,big,street,fuckin,bout,gettin,rap,hood,gun,block,motherfucker,dick,check,thug,catch,smoke,tryin,young,throw,straight

5 time have,all,time,say,never,see,way,take,think,life,find,try,thing,leave,too,look,nothing,lose,believe,live,always,everything,mind,change,long,something,wait,much,break,end,wrong,word,inside

6 Ardent love get,like,girl,off,body,put,boy,shake,hot,show,work,club,floor,lady,drop,cuz,sexy,got,sex,jump,low,lookin,freak,top,shawty,damn,pop,aint,boom,bounce,chick,dont,hair,thang,lil,shorty,push

7 life out,well,little,call,old,play,friend,talk,new,town,bad,use,car,kid,mama,drive,sit,lot,door,pay,drink,meet,pretty,house,buy,first,daddy,fun,guy,wear,next,write,phone,bed,school,hang,trouble,blues

8 foreign que,por,con,amor,como,una,quiero,para,sin,esta,pero,todo,solo,las,cuando,hay,soy,corazon,voy,vida,del,porque,los,tengo,bien,ella,estoy,ser,vez,hoy,aqui,les,este,puedo,siempre,dale,tan,quien

9 dark die,hand,head,lie,blood,face,dead,kill,fight,death,fear,hell,black,while,hate,wall,line,cut,scream,skin,bleed,pull,speak,become,bone,devil,full,thought,sick,blind,human,chain,build,stone,breath,

10 religious dem,god,man,lord,yuh,sing,nah,world,jesus,king,child,nuh,hear,gal,name,fus,song,praise,people,heaven,bear,inna,soul,pray,holy,war,free,pon,peace,hallelujah,glory,joy,thank,bless,mary,mek

Enhanced HMM

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 29

Generation process

For each topic z = 1,… , 𝐽:Draw 𝜃>→~𝐷𝑖𝑟 𝛼Draw 𝜙>~𝐷𝑖𝑟(𝛽)

For each lyrics 𝑚 = 1,2, … ,𝑀:For each Block 𝑏 = 1,2, … , 𝐵J:

Draw 𝑧K~𝑀𝑢𝑙𝑡𝑖 𝜃>NOP→For each word 𝑤 in block 𝑏:

Draw 𝑤~𝑀𝑢𝑙𝑡𝑖(𝜙>N)𝑤

𝑁,

𝑀

𝑧,

𝑁3 𝑁4𝑤

𝑧3

𝑤

𝑧4

𝜙>𝐽

𝜃>→𝐽

𝛽

𝛼

Plate notation of enhanced HMM

𝑃 𝐿𝑦𝑟𝑖𝑐𝑠J =UV𝜃>NOP→V𝜙>NWNX

YN

Z[,

\]

K[,

>_``

Generation probability of lyrics

topic

Transition distribution

Word distribution

DatasetnPurchasable at online shop.

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 30

$260

Development of a Lyrics Writing Support System

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 31

Our interaction design

System enables rich interaction by the topic transition model

System is designed as a recommendation system.

USER INPUT SYSTEM OUTPUT

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

Syllable counts

e.g., “Time”→”Life”→”Love”

Number of blocksNumber of lines

Structure of an entire song…

Sequence of topics

e.g., 1-3-1-1

Trial-and-Error

Recommended lyrics

If tomorrow never comesI remember the dayYou realize time flies

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

1

hogeGlassy FM Lead=126.

1

& \\17

& D . . . . . . D . . . . . . . . E .". . . D E ." . . . .

24

& . . . . . . . . # ." . # .! . # .! - D E ." . .

. . D E ." . .. .

30

& . . . . . . . . D E ." . . .. . . . . . . . . . . . D . .

35

& . . . . . . . . . . E ." .. . . . . . . . . . . . . . . . .

40

& . . . . . . . . . . . . . . . . . . . . . . . . # . . -44

& . . . . . . . . . . D E ." . . . . . . . . . . .49

& E ." . . . . . . . . . . . . . . . . . . . . E ." . . . . . . . . D54

& D E ." . . . . . . . . . . . . . . D . . . . . . . . .

58

& . . . . . . . . . . - D . . . . .

Development of a Lyrics Writing Support System

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 32

Our system intelligence

System enables rich interaction by the topic transition model

System automatically learns typical topic transitions and semantically appropriate wordings from a large collection of human lyrics.

Topic Transition Model

Language Model

Sweet love

Scene

Life

Exciting

Ardent Love

……

nigh

tda

ylig

ht …love

know

baby …

prob

abili

ty

wordword

…pr

obab

ility

Over 100000 lyricsI drop into the oceanMy friend the stormy seaAs long as the blindnessWon't leave me tonight…

Unsupervised Learning

SYSTEM INTELIGENCE

TRAINING DATA

Result: Automatic lyrics generation

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 33

# Block1: timeI know everything isAnd I remember every word unspokenRemember waiting there inside my headConversation circles

# Block2: timeI know everything isAnd I remember every word every thoughtTomorrow is another story about loveThe situation became

# Block3: lifeAnd I remember everythingCalifornia face realityAnd you realize you're lookingYou'll always rememberAnd you'll always beAnd everybody knows

# Block4: ardent loveI know you're watching every dreamI really don't want nobody sitting on topAnd I remember feeling like letting goI really don't matter

# Block5: sweet loveI remember DecemberI promise you'll remember the first degreeI don't really understandI promise you'll beOh supernatural love

Result: User interaction with system

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 34

# Block1: timeI remember our loveAnd I remember every word unspokenRemember waiting there inside my headRemember I believe

# Block2: timeI know everything endsBut I remember every word every thoughtTomorrow is another story about loveSo appreciate it all

# Block3: lifeYou can remember yesterdayReminiscing on everyoneAnd you realize you're lookingYou'll always rememberAnd every single dayYou probably say

# Block4: ardent loveIt's all because you're wasting timeBut you're the one you're looking out forAnd I remember feeling like letting go'Cause I don't need anybody

# Block5: sweet loveI remember DecemberAnd every little tenderness, and I believeIf you really understandI promise you'll beAn extraordinary man

This result shows that the created lyrics correspond to the input parameters (i.e., syllables and topics)

We can see the sentimental phrases “in my mind” and ”I just wanna be with you” were created when the topic was ⟨Sweet Love⟩.

Example 2 (made by user)

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 35

世界中でこんなにすれ違い.涙を見せず独り振る舞うだけで.あなたのそばに.(We could not understand each other in this world. I didn’t show my tears, I was lonely. I just wanna be with you.)

世界中でこんなにたくさんの.心に決めて思い出した思い出に.あなたのそばに.(So many things in the world. Memories that I remembered in my mind. I just wanna be with you.)

Block 2, Topic: Sweet Love

se- ka- i- ju- u de ko-n- na- ni su-re-chi-ga- i na-mi- da wo mi-se-zu hi-to-ri hu-ru-ma-u da-ke de a- na- ta no so- ba ni

Fully automatically generated lyrics.

Example outcome of an user’s interactions with the system.

This result shows that the created lyrics correspond to the input parameters (i.e., syllables and topics)

We can see the scenic phrases “the way of memories” and ”wet in the rain” were created when the topic was ⟨Scene⟩.

Example (made by user)

2017/3/16 The 22nd annual meeting of the intelligent user interfaces community (IUI2017) 36

思い出の坂道は久しぶりの昼下がりへ.恋人の足跡はアスファルトの雨に濡れる.

o-mo-i- de no sa-ka-mi-chi wa hi- sa-si-bu-ri no hi- ru sa-ga-ri e ko- i- bi-to no a- shi-a- to ha a- su-fa-ru-to no a- me ni nu-re-ru

(Afternoon came to the way of memories after a long time. Lovers' footprint get wet in the rain on asphalt.)

思い出の坂道を.雨上がりの交差点で.思い出の坂道を.心にない雨に濡れて.(The way of memories. At the intersection of the rain. The way of memories. I got wet in the rain without heart.)

Block 1, Topic: Scene

Fully automatically generated lyrics.

Example outcome of an user’s interactions with the system.