Dialogue Annotation at SUNY

Hilda Hardy

Amitiés Annotation WorkshopLIMSI-CNRS, France

July 8-10, 2002

2

What did we annotate?

Transcriptions of dialogues from 3 folders in the englishPlainTextDialogues.zip archive (dated March 28, 2002):

– 22mar02, 20 of 60 files (Uzo Enyinna)– 22mar02, 21 of 60 files (Hilda Hardy)– 26feb02, 30 of 64 files (Tom Palen) – 26mar02, 36 of 36 files (Hilda Hardy)

3

What approach did we use?

We annotated the Semantic and Functional layers of the dialogues (not the Stylistic layer), using XDMLTool version 2.0.

4

5

Why annotate Semantic and Functional Aspects of Real Call-Center Dialogues?• Get ideas and data for designing a better, more

human-like Dialogue Manager– Find out structures of dialogues

• Basic patterns of Dialogue Acts• Basic flow of the dialogue• What is essential in accomplishing the task?• What is parenthetical?

– Find out how information is exchanged• How is information broken down?• How is information ordered in a dialogue?

– Discover relationships between functional and semantic layers

6

Why annotate Semantic and Functional Aspects? Cont’d

– What are the most common questions customers ask agents?

– What facts do agents need to learn from customers? What facts do they give customers? We need supplementary information from the call centers to handle these aspects.

• Improve language generation– How do agents converse with customers? Look at

phrasing, length of turns

• Decide what can be automated

• Other: a well-annotated corpus is a valuable contribution to the research community

7

Semantic Annotation: Customer AccessFrames

Customer transactions: every dialogue has a purpose: at least one topic or task that the customer needs to have answered or satisfied.

Task Information

AcctBalanceStatement

Insurance

MakePayment

ChangeAddress

CancelDebit

CloseAccount

LostCard Fees

8

Semantic Annotation: Agent AccessFramesAgent transactions: tasks initiated by the agent in the process of satisfying the customer’s needs

GreetCaller

VerifyId

“Good afternoon, . . . How can I help?”

Information required for security checks: name, old post code, date of birth, usual method of payment, etc.

Closure “Is there anything else? . . . Goodbye”

9

Semantic Annotation: Attributes, Values and Modifiers

Most frames are associated with details that may be organized into key-value pairs. Examples:

AccessFrame Attribute Value Modifier

Name Paul Agent

PostCode AB1 2CD New

HouseNumber 10 New

GreetCaller

ChangeAddress

10

Semantic Annotation: General Shape of a Call Center Dialogue

A typical call center dialogue progresses with this general order of frames:

GreetCaller

ChangeAddress

VerifyId

Info-Insurance MakePaymentor or

Closure

11

A: Good afternoon customer services Paul speakingC: Hello I want to inform you of my change of address pleaseA: Certainly what name is it please?C: Name? It's Mrs Pauline SmithA+C: Right Mrs Hunter I'm just keying in the account number for your bear with me a momentUh hmmA: (pause)Could you confirm the, your previous postcode and telephone number please?C: Yeah the previous one was AB1 2CD01111 111111A: Thank you, you're speaking to Paul, good afternoonC: Uh hmmA: Right so it's Paul and PaulineC: (laughs)A: The new address, is it a house number Mrs Smith?C: Yeah it's number 3 A: And the postcode there is?C: W for Whiskey X for X-ray two ohA: Yes

Semantic Annotation: example

GreetCaller

ChangeAddress

VerifyId

GreetCaller

ChangeAddress

12

Semantic Annotation: example, cont’dC: Three Y for Yankee Z for ZuluA: And is that The Oaks?C: That's rightA: The Hall?C: That's itA: Do you have a new telephone number?C: Yes I do it's erm 01111A: YesC: 111A: YesC: 111A+C: And this is your Principles card isn't it?YeahA+C: Have you got any other shop cards that you need to changeNo, no I haven't noA: FineA+C: Is there anything else I can do for you?OkayC: That's all thank you very muchA: Thanks for your callC: Thank you byeA: Bye

(END OF CALL)

ChangeAddress

Closure

13

Semantic Annotation with XDMLTool

Semantic categories

14

Semantic Annotation with XDMLTool

• Semantic annotation is domain-dependent– But the categories AccessFrame, Attribute, Value,

and Modifier were designed to be domain-independent.

– For a new domain, use new lists of tags.

• Helpful to begin with a set of labels developed during preliminary annotation

• Necessary to allow the annotator to add to the lists as new information appears in the data

15

Semantic Annotation: Suggestions

• Compare annotators’ lists and merge labels that are clearly the same– ChangeAddress and UpdateAddress– VerifyId and VerifyCallerID– Card and NameofCard

• Discuss how to handle cases that are not so obvious: What level of detail is best?– Address = StreetAddr, Area/Town, City, County?

• Try to finalize lists as far as possible, leaving room for new labels

16

Functional Annotation with XDMLTool

Functional categories

17

Functional Annotation: General Approach

• Annotate dialogue act functions (dialogue-related speech acts, or DAs)

• Use DAMSL (Dialog Act Markup in Several Layers), tailored to reflect call center dialogue features– Both the categories and the lists are domain-

independent– Minor modifications to DAMSL are necessary– Annotation conventions must be discussed

18

DAMSL Amitiés/XDML (proposed)

Communicative Status (Features)Self-talk used occasionally

Uninterpretable not used (seems better suited

to transcription)

Abandoned used occasionally

Non-verbal (keyboard

noise, music, dog barking)

Third-party-talk

(side conversations)

19

DAMSL Amitiés/XDML (proposed)

Information-LevelTask annotated by default

Task-management not used (rare; agent reacts

to the customer)

Communication-mgt used as defined

Other-level Out-of-topic

(jokes, non-sequiturs, small talk, and any comments

or remarks that have no direct bearing on accomplishing

the task)

20

DAMSL Amitiés/XDML (proposed)

Forward Looking Function Statement

Assert used for simple assertions,

such as answers

Explanation (agent explains

policies, customer gives reasons)

Recap (agent summarizes, wraps

up, or recapitulates the task)

Reassert used as defined

Other-statement not used

21

DAMSL Amitiés/XDML (proposed)

Influence-on-listenerAction-directive used as defined

Open-option used as defined (when a course of

action is suggested with no obligation on the listener)

Info-request used only to request information,

not to elicit a yes/no response

Confirmation-request (needs a

yes/no response)

22

DAMSL Amitiés/XDML (proposed)

Influence-on-speakerOffer used as defined

Commit used as defined

ConventionalOpening used as defined

Closing used as defined

Explicit-performative

Exclamation Expression (thank you, okay,

Other-forward-function no problem, smashing, sorry)

23

DAMSL Amitiés/XDML (proposed)

Backward Looking Function Agreement

Accept

Accept-part

Maybe used as defined

Hold

Reject

Reject-part

I_don’t_know

24

DAMSL Amitiés/XDML (proposed)

UnderstandingSignal-non-understanding Non-understandingSignal-understanding

Acknowledge used as defined*Repeat-rephrase used as definedCompletion used as defined

Correct-misspeaking Correction

Answer used as defined*Same definition as “Backchannel”.NOTE: we recommend removing “Confirm” and

“Disconfirm” (same as Accept and Reject)

25

Annotation: Other Issues

• When should an annotator split (segment) a turn?– Semantic data must be annotated separately:

C: Post code AB1 2CD, phone 01111 111111

– One turn may have different functions:

C: Hi there erm I’ve lost my card and also I’ve changed address

– An overlapping turn has different speakers:

A+C: It’s Top Shop account card

Top Shop thanks

26

Annotation: Other Issues

• Leave the turn intact when it begins with an acknowledgment. Info-level is “Task” and not “Communication-mgt.”

A+C: Top Shop thanks

A: Okay what’s your surname?

A: Ok and your address, the old one that is?

27

Automatic tag selectionsSeveral of the DAMSL tags are contained in others or require other selections to be made. For example, "answers by definition will always be asserts." (Draft of DAMSL, p. 23). For added convenience, XDMLTool v. 2.0 makes the following additional selections by default. If exceptions arise, the user may change the automatic selections manually.

When annotator selects... ... XDMLTool selects...

any Backward Communicative Function Previous utterance number as "Response-to" value Conventional->Opening Info-level->Communication-mgt Conventional->Closing Info-level->Communication-mgt Understanding->Acknowledge Info-level->Communication-mgt Understanding->Backchannel Info-level->Communication-mgt Understanding->Non-understanding Info-level->Communication-mgt Answer Statement->Assert Agreement->Accept Answer, Statement->Assert Agreement->Accept-part Answer, Statement->Assert Agreement->Reject Answer, Statement->Assert Agreement->Reject-part Answer, Statement->Assert