A Description of Amitiés Modules Developed at SUNY Hilda Hardy Amitiés Consortium Meeting...

A Description of Amitiés Modules Developed at SUNY

Hilda Hardy

Amitiés Consortium MeetingUniversity of Sheffield

May 2, 2002

2

Xinyang Zhang

3

Amitiés System Architecture

Hub

Speech Recognition

Dialogue Manager

Database Server

Text-to-speech Conversion

Nat’l Language UnderstandingFrench

German

Audio Server

Response Generation

FrenchGerman

CustomerDatabase

Engl.

LIMSI

Sheffield

SUNY

4

What we like about Galaxy,MITRE’s Galaxy Communicator SoftwareInfrastructure:• Distributed, message-based hub-and-spoke

infrastructure• Optimized for constructing spoken dialogue systems• Built-in Hub, implemented in C

– Maintains socket connections to servers– Handles message traffic

• Protocol for frames, functions, program files clearly defined

• Excellent on-line tutorial, documentation, many examples

5

What we don’t like about Galaxy (some minor points):

• Difficult to use an object-oriented approach for developing servers

• Not platform-independent• Frequent new versions, problems with memory

management

6

Hub program file: amities.pgmPGM_SYNTAX: extendedSERVICE_TYPE: UICLIENT_PORT: 14500SERVER: DialogueHOST: localhostPORT: 18500OPERATIONS: DoDialogue DoGreetingSERVER: Response_GeneratorHOST: localhostPORT: 10000OPERATIONS: GenerateSERVER: French_GeneratorHOST: localhostPORT: 15000OPERATIONS: GenerateSERVER: German_GeneratorHOST: localhostPORT: 15500OPERATIONS: Generate. . .

server info

port number determined by server

at server startup

7

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; MAIN OUTPUT BODY;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; This program handles the response from the dialogue;; manager which may take one of a number of ;; forms (database tuples to be described, or perhaps;; an already-formatted frame).PROGRAM: DialogueOutputRULE: :output_frame --> Response_Generator.GenerateIN: :output_frameOUT: :output_stringRULE: :output_string -->Display.DoDisplayIN: :output_stringOUT: none!PROGRAM: FrenchDialogOutputRULE: :output_frame --> French_Generator.GenerateIN: :output_frameOUT: :output_string. . .

amities.pgm: example Hub program

name of program, frame dispatch function

condition(s)Hub program

8

Dialogue Manager/Frame Router• Handles task identification

1. Change address

2. Replace card

3. Inquire about account balance

• Handles customer identification • Uses a mixed-initiative approach• Implements layers of abstraction for language and

database independence• Can recover from word recognition errors• Can perform mid-conversation language switching

9

Example frames: NLU to DM

{c NLU.Parse

:token_string (“Christopher” “Smith” “here”...)

:named_entities {c entities

:lname “Smith”

:fname “Christopher”}}

{c NLU.Parse :token_string (“Hello” “I” “wanted” “to” “notify” “you” “of” “a” “change” “of” “address”)}

frame type name

keys

list value

frame value

string values

10

Dialogue Server Dialogue Manager

Task ID Frame Router

User ID Frame Router

1 2

Task1

Keyword profilePrompt sequence

Task2 …

properties file

from NLU via Hub

11

Dialogue Manager: Some Details• Accomplishes goals in any order, even all at once• Loads properties file according to language ID• For task ID: entire utterance need not be recognized

as long as the salient words are identified• Queries database for named entities, uses “longest

common subsequence” dynamic programming technique. Example:

1 2 3 4 5 6

1 too 3 4 5 sick

1 X 3 4 5 X

1 2 3 4 5 6

1 2 tea or 5 6

1 2 X X 5 6

from database

from user

matching pattern

1 2 3 4 5 6

12

Dialogue Manager: Data Representation

A Working-space structure is maintained for each dialogue:– Primary keys, attributes, values, matched patterns– Matching-percentage scores

primary key

score

primary key

score

primary key

score

. . .

attribute

value

match_string

score

attribute

value

match_string

score

. . .

13

Database Server (pgdb_server)• Provides connectivity to a PostgreSQL database• Uses an extended data model: maps meta-attributes

to actual attributes• Sample table constructed from fictitious English,

French and German customer data• Example input from Dialogue Manager:

{c sql_query :query “select title, fname, lname, post_code, house_number from table where house_number = ‘1’ and account_number = ‘1234567887654321’” :maxrows 10 }

14

Database Server: Sample return frame

{c sql_query :query_result {c result

:values ( (“Mr.” “Stephen” “Jones” “EC4 5BE” “1” ) )

:column_names (“title” “fname” “lname”

“post_code”

“house_number”) :nfound 1 } }

one list for each hit

15

pgdb_serverDM/FR

Customer Database

Meta-attributes Meta-attributes

Actual attributes

fname lname post_code . . .

first name surname post code . . .

Database Server

properties file

via Hub

16

Multilingual Response Generator

• Implemented with separate servers in English, French and German

• Uses an extended data model: maps meta-names to actual words in the appropriate language

• Constructs responses according to our abstract representation of system utterances

17

Response Generator: Sample input, output frames

{c French_Generator.Generate OR FrenchDialogueOutput :output_frame {c output

:type “request” :repetition 0 :attributes ( “name”

“post_code” )

:modifiers ( “” “old” ) } }

{c French_Generator.Generate :output_string “Pouvez-vous me donner votre nom et votre ancien code postal, s’il vous plait?” }

from Dialogue Manager

to Display Server

18

Response Generator: Schema for representing system utterances

:type :repetition :attributes :modifiers

A: Amitiés, how can I help, [French], [German]?“prompt” 0A: Can I take your name and your old post code, please?“request” 0 (“name” “post_code”)(“” “old”)A: Thank you. I need to ask you some questions so that I can update

your address in our records.“respond” (“change_address”)A: How can I help?“prompt” 1A: I’m sorry, what can I do for you?“prompt”2A: Please hold while I transfer you to a Customer Service

Representative.“respond” (“hold”)

19

Dialogue Manager

Response Generator

German Generator

properties fileFrench Generator

meta-names

English namesFrench namesGerman names

Language Generator Servers

properties file

communication via the Hub

20

Response Generator: Meta-names and Engl/Fr/De equivalents

name name nom/m Namen/ffname first name prénom/m Vornamen/mlname last name nom/m Nachnamen/mhouse_number house number nombre de maison/f

Hausnummer/faddress address adresse/f Adresse/fpost_codepost code code postal/m Postleitzahl/faccount_number account number numéro de compte/m

Kontonummer/fbirthdatedate of birth date de naissance/f Geburtstag/mold old ancien/ancienne altenew new nouveau/nouvelle neue