Goteborg University Dialogue Systems Lab Dialogue and grammar Göteborg University Dialogue Systems Lab Language Technology in Sweden 2004-05-12 Karin Cavallin,

Goteborg University Dialogue Systems Lab

Dialogue and grammar

Göteborg University Dialogue Systems Lab

Language Technology in Sweden2004-05-12

Karin Cavallin, Robin CooperDavid Hjelm, Staffan Larsson


Speech recognition and NL interpretation & generation

• Options for SR:– Statistical language model– Grammar model

• Options for NL interpretation:– Word/phrase spotting– Grammar & parsing

• Options for NL generation– Template-based– Grammar-based

• Can be combined in various ways


Are grammars useful for dialogue systems?

• Most working systems don’t use a grammar!

• Instead: – keyword or phrase-spotting techniques for

interpretation– template-based generation

• Also, most working systems are built more or less from scratch– do not fully separate application-specific

knowledge from general and reusable knowledge


GoDiS

• A dialogue system built using TrindiKit and based on the information state approach

• Implements a theory of Issue-based Dialogue Management

• Some goals with GoDiS:– Provide a domain-independent theory covering several

dialogue genres– Modularity and reusability of system components– Enable rapid prototyping of new applications

• Focus on dialogue management• These goals have been achieved on the level of

dialogue management...– ... but not for NL processing


SR and interpretation in GoDiS

• Started out as text-based system– Phrase spotting for interpretation;

strings to dialogue moves– Templates for generation; dialogue

moves to strings• Added SR component

– Did not have a corpus, so needed to write a speech recognition grammar by hand


tvGoDiS

• GoDiS application for programming a VCR over the telephone– schedule recordings– remove planned recordings– TV card used for recording television

programs as video files• Small corpus collected of users calling the

system– approx. 100 sessions (calls) and 1100 utterances– misrecognised utterances not yet transcribed– GUI enables immediate access to corpus


Example from tvGoDIS corpus

• TrindiGui displaying a tvGoDiS session

• Audio files linked with utterances

• Access to information state at different stages of the dialogue


Problems with GoDiS

• Mismatch between SR module and interpretation module– correctly recognised speech was

misinterpreted– Leads to strange and unexpected

behaviour (unexpected interactions with dialogue management)


ASR/interpretation mismatch

S> Välkommen till videon! För att få hjälp, säg hjälp.

S> Låt oss se. Vad kan jag hjälpa dig med?

U> jag vill spela in ett program (0.6)

S> Vill du lägga till en inspelning?

U> ja (0.98)S> Låt oss se. Menar du från

noll ett eller till noll ett?

• jag vill spela in ett program– Covered by SR grammar– Not covered by phrase spotting

lexicon– However spela in and ett are

recognized by interpretation module

• Correct interpretation: – jag vill spela in ett program

request(add_rec_job)• Actual interpretation:

– spela in request(add_rec_job

– ett

answer(time(1:00))• System assumes ett specifies

either when the recording should stop or end – thus the follow-up question


Problems with GoDiS

• Mismatch between interpretation module and generation module– System said something that led the

user to believe they could say X – but the system does not understand X


Interpretation/generation mismatch

S> Du kan lägga till en inspelning, ta bort en planerad inspelning, och lista planerade inspelningar. För att börja om, säg börja om. För att sluta, säg sluta.

S> huvudmenyn. Vad kan jag hjälpa dig med?

U> lista inspelningar (0.75)S> Ockey. gå till inspelningar.S> Låt oss se. Vill du lägga till

en inspelning , radera dina inspelningar eller få information om dina inspelningar?

• Help message suggests that the user can say lista inspelningar

• But help message is not up-to-date, and neither is SR grammar.

• Instead interpretation module finds word inspelningar

• The user is asked to choose between roughly the same alternatives that was presented in the help message


• Bugs can of course be fixed...– would take lots of time– no guarantee that new bugs won’t be

introduced– no guarantee that all bugs have shown up

• These solutions are not reusable• Instead, we want a general and

principled solution– enable reuse and rapid prototyping of

applications, including grammars


Solution, part 1: single grammar for SR, interpretation, generation

• No mismatches between the grammars• Only one grammar needs to be written for each

application• However:

– the different modules may require different kinds of grammars, in different formats

• So:– We need to be able to generate all 3 grammars from a

single grammar• However, we may not want the exact same coverage for

SR, interpretation and generation grammars – May not want to understand everything that can be

generated


Solution, part 2: use GF!

• GF is a powerful tool for mutilingual grammar development

• Can be used to generate grammars in various other formats

• Subgrammars can be extracted from larger grammars– SR, interpretation and generation subgrammars may

have different coverage • In addition, GF offers resource grammars in

several languages– This promises to further decrease the time needed

for writing a new application grammar


GF GoDiS Grammar

• Application grammars– User grammar– System grammar– Shared grammar, used by user grammar

and system grammar• Utility grammars

– Weekday, Time, Channel• Application grammars may use

– Utility grammars– Resource grammars


GF VCR grammar

• One abstract syntax– GoDiS dialogue moves as GF abstract trees

• Three concrete syntaxes– Swedish– English – GoDiS dialogue moves in prolog notation

• Statistics: – Abstract syntax : ~420 lines of code– Concrete syntax :

• Prolog: ~440 lines of code• Swedish: ~440 lines of code


Parsing Swedish to abstract syntax

> p -cat=DMove -lang=Swe "spela in från kvart över tolv" request (vcr_add_rec_job_args1 (startTimeToStoreReq

(startTimeToStore (time hour12 minute15))))


Linearising abstract syntax to Prolog notation

> l -lang=Prolog request (vcr_add_rec_job_args1 (startTimeToStoreReq (startTimeToStore (time hour12 minute15))))

request(add_rec_job , answer(start_time_to_store(12 : 15))


Translating Prolog to Swedish

> t -cat=DMove Prolog Swe "request(add_rec_job , answer(start_time_to_store(12 : 15))"

spela in från tolv femtonspela in från tolv och femtonspela in från femton över tolvspela in från femton minuter över tolvspela in från kvart över tolv


Time Grammar

• A multilingual grammar module• Swedish, English, Prolog, started on

French and Catalan.


Constructing concrete syntaxes for time expressions isn’t always trivial

• Abstract: time hour12 Minute30• Prolog: 12 : 30• English: half past twelve / half twelve• Swedish: halv ett

• klockan två • klockan två och femton• ?klockan fjorton• two o’clock• *two fifteen o’clock• *fourteen o’clock

Corpus studies needed


Another problem

• am or pm?• på morgonen • på eftermiddagen• på kvällen• på natten


Combining GF and statistical methods for robustness

• Use GF to generate a corpus to use as a baseline for SLM training

• Use GF-generated SR grammars in initial system and collect corpus to be used for training SLMs for speech recognition

• For recognized out-of-grammar string, try to find nearest in-grammar string; confirm this with user (Gorrell 2003)


Conclusion

• Keeping track of separate SR, interpretation and generation grammars in a dialogue system is time-consuming

• GF offers the possibility of rapid prototyping of grammars for use in dialogue system applications– Generate other grammar formats from GF– Resource grammars for several languages

• GF can be combined with statistical methods for robustness

Documents

Goteborg University Dialogue Systems Lab Dialogue and grammar Göteborg University Dialogue Systems Lab Language Technology in Sweden 2004-05-12 Karin Cavallin,