Multi-modal Reference Resolution in Situated Dialogue by Integrating Linguistic and Extra-Linguistic Clues

Ryu IidaMasaaki YasuharaTakenobu TokunagaTokyo Institute of Technology

IJCNLP 2011 (Nov 9 2011)

2

Research background

Typical coreference/anaphora resolution Researchers have tackled problems provided by

MUC, ACE and CoNLL shared tasks (a.k.a. OntoNote)

Mainly focused on linguistic aspect of reference function

Multi-modal research community(Byron, 2005; Prasov and Chai, 2008; Prasov and Chai, 2010; Schütte et al., 2010, Iida et al. 2010) Essential for human-computer interaction Identifying referents of referring expressions in a

static scene or a situated world, taking extra-linguistic clues into account

3

Multi-modal reference resolution

move the triangle to the left

Rotate the triangle at top right 60 degrees clockwise

All right… done it..

O.K.

dialogue history

…piece 1: move (X:230,Y:150)piece 7: move (X:311, Y:510)piece 3: rotate 60°

action history

eye-gaze

4

Aim

Integrate several types of multi-modal information into a machine learning-based reference resolution model

Investigate which kinds of clues are effective on multi-modal reference resolution

5

Multi-modal problem setting:related work

3D virtual world (Byron 2005, Stonia et al. 2008) e.g. Participants controlled an avatar in a virtual

world for exploring hidden treasures Frequently occurring scene updates Referring expressions will be relatively skewed to exophoric cases

Static scene (Dale 1992) Centrality and size of each object in computer

display is fixed through dialogues Change of visual salience of objects is not observed

6

Evaluation data set creation

REX-J corpus (Spanger et al. 2010) Dialogues and transcripts of collaborative

work (solving Tangram puzzles) by two Japanese participants

Designed the puzzle solving task to require the frequent use of both anaphoric and exophoric referring expressions

7

solver operator

Setting of collecting data

not availabl

eshield screen

working area working area

goal shape

8

Collecting eye gaze data

Recruited 18 Japanese graduate students split them into 9 pairs All pairs knew each other previously and were of

the same sex and approximately the same age Introduced to solve 4 different Tangram puzzles

Use the Tobii T60 Eye Tracker, sampling at 60 Hz for recording users’ eye gaze with 0.5 degrees in accuracy 5 dialogues in which the tracking results

contained more than 40% errors were removed

9

Annotating referring expressions Conducted using a multimedia

annotation tool, ELAN Annotator manually detects a referring

expression and then selects its referent out of the possible puzzle pieces shown on the computer display

Total number of annotated referring expressions:1,462 instances in 27 dialogues 1,192 instances in solver’s utterances

(81.5%) 270 instances in operator’s utterances

(18.5%)

10

Multi-modal reference resolution Base model

Ranking candidate referents is important for better accuracy (Iida et al. 2003, Yang et al. 2003, Denis & Baldridge 2008)

Apply Ranking SVM algorithm (Joachims, 2002) Learn a weight vector to rank candidates for a

given partial ranking of each referent Training instances

To define the partial ranking of candidate referents, simply rank referents referred to by a given referring expression as first place and any other referents as second place

11

Feature set

1. Linguistic features: Ling (Iida et al. 2010):10 features

Capture the linguistic salience of each referent based on the discourse history

2. Task-specific features: TaskSp (Iida et al. 2010):12 features

Consider the visual salience based on the recent movements of mouse cursor and recent pieces manipulated by the operator

3. Eye-gaze features: Gaze (proposed):14 features

12

Eye gaze as clues of reference function

Eye gaze Saccades: quick, simultaneous movements

of both eyes in the same direction Eye-fixations: maintaining of the visual gaze

on a single location Direction of eye gaze directly reflects the

focus of attention (Richardson et al., 2007) Used the eye fixations as clues for

identifying the pieces focused on Separating saccades and eye fixations:

Dispersion-threshold identification (Salvucci and Anderson, 2001)

13

Eye gaze features

time

“First you need to move the smallest triangle to the left”

a

b

cd

e

f

g

fixating on piece_b

t-T

T = 1500msec (Prasov and Chai 2010) t t’

fixating on piece_a

how frequently orhow long the speaker fixates on each piece

14

Empirical evaluation

Compared models with different combinations of the three types of features Conducted 5-fold cross-validation

Proposed model with model separation (Iida et al. 2010) the referential behaviour of pronouns is

completely different from non-pronounsSeparately create two reference resolution models; pronoun model: identifies a referent of a given

pronoun non-pronoun model: identifies a referent of all

other expressions (e.g. NP)

15

Results of (non-)pronouns

model pronoun non-pronoun

Ling 56.0 65.4

Gaze 56.7 48.0

TaskSp 79.2 21.1

Ling+Gaze 66.5 75.7

Ling+TaskSp 79.0 67.1

TaskSp+Gaze 78.0 48.4

Ling+TaskSp+Gaze 78.7 76.0

16

Overall results

model accuracy

Ling 61.8

Gaze 51.2

TaskSp 42.8

Ling+Gaze 72.3

Ling+TaskSp 71.5

TaskSp+Gaze 59.5

Ling+TaskSp+Gaze 77.0

17

Investigation of the significance of features

Calculate the weight of features according to the following formula

set of the support vectors in a ranker weight of the

support vector x

function that returns 1 if f occurs in x

18

Weights of features in each model

pronoun model non-pronoun model

rank feature weight feature weight

1 TaskSp1 0.4744 Ling6 0.6149

2 TaskSp3 0.2684 Gaze10 0.1566

3 Ling1 0.2298 Gaze9 0.1566

4 TaskSp7 0.1929 Gaze7 0.1255

5 TaskSp9 0.1605 Gaze11 0.1225

6 Gaze10 0.1547 Gaze14 0.1134

7 Gaze9 0.1547 Gaze13 0.1134

8 Ling6 0.1442 Gaze12 0.1026

9 Gaze7 0.1267 Ling2 0.1014

10 Ling2 0.1164 Gaze1 0.0750

TaskSp1: mouse cursor was over a piece at the beginning of uttering a referring expression

TaskSp3: time distance is less than or equal to 10 sec after the mouse cursor was over a piece

19

Weights of features in each model

pronoun model non-pronoun model

rank feature weight feature weight

1 TaskSp1 0.4744 Ling6 0.6149

2 TaskSp3 0.2684 Gaze10 0.1566

3 Ling1 0.2298 Gaze9 0.1566

4 TaskSp7 0.1929 Gaze7 0.1255

5 TaskSp9 0.1605 Gaze11 0.1225

6 Gaze10 0.1547 Gaze14 0.1134

7 Gaze9 0.1547 Gaze13 0.1134

8 Ling6 0.1442 Gaze12 0.1026

9 Gaze7 0.1267 Ling2 0.1014

10 Ling2 0.1164 Gaze1 0.0750

Ling6: shape attributes of a piece are compatible with the attributes of a referring expression

Gaze10: there exists the fixation time of a piece in the time period [t − T , t]

Gaze 9: the fixation time of a piece in the time period [t − T , t] is longest out of all pieces

20

Summary

Investigated the impact of multi-modal information on reference resolution in Japanese situated dialogues

The results demonstrate The referents of pronouns rely on the visual

focus of attention such as is indicated by moving the mouse cursor

Non-pronouns are strongly related to eye fixations on its referent

Integrating these two types of multi-modal information into linguistic information contributes to increasing accuracy of reference resolution

21

Future work

Need further data collection All objects in Tangram puzzle (i.e. puzzle

pieces) have nearly the same size Rejecting the factor that a relatively larger

object occupying the computer display has higher prominence over smaller objects

Zero-anaphors in utterances need to be annotated frequent use of them in Japanese