Watching the eyes when talking about size: An investigation of message formulation and utterance planning

Sarah Brown-Schmidt, Michael K. Tanenhaus

Presentation by Olatz MendiolaLing 5555November 17, 2010

Introduction

How do speakers plan and update messages that are translated into utterances during language production?

How are message formulation and utterance planning coordinated?

Speakers often update messages on the fly based on new information. Message formulation and utterance planning are

mixed in time and must communicate with one another at a relatively fine temporal grain.

Introduction

Levelt & Maassen (1981) Latency difference evidence that the unit of advance planning includes each

of the lemmas in the subject noun phrase.

Meyer (1996) Dutch speakers perform a naming task with auditory distractors. Naming latencies were delayed by a distractor that was phonologically

related to the first noun in NP but not the second word in the NP. Distractors semantically related to either 1st or 2nd noun decreased

latencies.

Schriefers &Teruel (1999) Presented phonological distractors to German speakers in a naming task. Phonological distractors related to 1st syllable of 1st word facilitated naming

times, no effect found for distractors related to 2nd word and distractors related to 2nd syllable of 1st word had a weak effect.

In some cases, articulation may begin before the 1st phonological word is entirely planned.

Only 1st word is prepared before speech onset

Introduction

The degree to which speakers pre-plan a complex phrase is highly variable. Amount of advance planning varies depending on the task and type of

utterance Planning is partially under control of speaker.

Little work has addressed how messages are prepared

Griffin and Bock (2000) Participants described simple events (woman shooting a man…) The sequence of eye movements reflected the order of constituents in the

utterance. Speakers looked at picture objects about 800ms to 1s before naming them. This eye-voice lag is similar to the time it takes to initiate naming an object in

isolation, suggesting that the eye-voice delay reflects word preparation.

Difficult?

Introduction

Timing of eye movements could provide insight into interplay between message formulation and utterance planning.

Work investigating gaze-speech link may typically conflate message formulation with utterance planning.

How disentangle message formulation form utterance planning?

creating situations in which the speaker, while in the process of planning or producing an utterance, encounters new information that requires revising the message.

If eye movements can be used to infer when the speaker 1st encounters that information, then the timing between the uptake of the new information and the form of the utterance might shed light on the interface between message production and utterance planning.

Brown-Schmidt

Tanenhaus (2006)

Present Study. Experiment 1

18 pairs of participants Seated at different computers. 5 feet separation No eye contact Subject X Eye tracker + headset

microphone Subject Y Headset microphone

Listener’s perspective Speaker’s perspective

Listener’s perspective Speaker’s perspective

Present Study

Hypothesis 1 Fixation to the contrast will predict use or non-use of a size

adjective. Higher rates of modification when contrast has been fixated.

Hypothesis 2 There is a systematic relationship btw timing of 1st look to

the contrast and the form of the utterance. Hypothesis 3

Timing of looks to the contrast and use of a pre-nominal vs. post-noun repair for descriptions of complex and simple shapes should provide insight into the size of message planning units, and how repairs to these units affect utterance planning.

Results

Manipulations successful in eliciting expected forms:

1. Size adj. typically used only when contrast present in display.

2. Pre-nominal adj. used for Simple Shapes.

Adj. in post-nominal PP used for Complex Shapes.

Results

1. Use of size adjectives

Contrast : size adj. on 98% of trials No contrast: size adj. on 27% of trials

Contrast/Simple: size adj. on 99% of trials Contrast/Complex: size adj. on 97% of trials NoContrast/Simple: size adj. on 23% of trials NoContrast/Complex: size adj. on 31 % of trials

Statistically Significant

Not significant

Results

2. Utterance form

Contrast / Size adj. / Simple shape pre-nominal 89% of trials

Contrast / Size adj. /Complex shape post-nominal PP 73% of trials

Size adj. also occurred in post-noun repairs“The triangle, uh the small one”“The square with triangles, uh small ones”

11% of trials for simple shapes 16% of trials for complex shapes

Results.

Hypothesis 1 Fixation to the contrast will predict use or non-use of a size adjective.

Higher rates of modification when contrast has been fixated.

Size contrast / fixated Size adj. on 99% of trials

Size contrast / not fixated Size adj. on 68% of trials

Results.

Time of the first fixation to the contrast relative to onset of utterance.

Form of referring expression is related to timing btw 1st look to size contrast and onset of utterance.

ANOVA: Effect of adjective position

Earlier fixation for pre-nominal adj.

Effect of complexity. Earlier fixation for references to

simple shapes Interaction btw complexity and

adj. position was not significant

New message elements can be added to a planned

referential description duringor immediately before production

Supports Hypothesis 2UNEXPECTED

Results

Simple shapes: Negative relationship between first fixations and adj. onset time.

Complex shape: marginal effect

When need for a size adjective is noticed after speech onset, preparation of the repair is delayed by preparation of the original utterance.

Lag between 1st fixation to contrast and adjective onset Complex: 1220ms Simple: 440ms

Results

Simple shape Prenominal

1546ms after display Post repair

1061ms after display

Complex shape Prenominal

1434ms after display Post repair

1390ms after display

Conclusions

Fixation on size contrast predicts use of size adj.

Timing of fixations predicts the form of the utterance.

When preparing a modified referring expression, messages are incrementally prepared and passed onto utterance planning in units smaller than the size needed for an entire referring expression.

Towards experiment 2

2 unexpected results 600ms lag between fixation to the contrast and onset of

utterance. 200ms faster than literature reports. Explanation? A small number of shapes were repeatedly used

on target trials, which may reduce the demands of lexical encoding

Size adj. were used on more than 25% of trials when there was no contrast. Explanation? 25% of trials had targets with size contrast

which might make inclusion of size in the description somewhat formulaic

Experiment 2 Increasing target type Decreasing proportion of target trials

Experiment 2

Results

Speakers rarely used a size adj. when there was no contrast in display = 1 %

Speakers used a size adj. 72% of the references on trials with contrast

Proportions consistent with other studies Of the size contrast trials where adj. was

used: 62% pre-nominal 37% post-noun repairs

Results Analysis was restricted to trials were there was a size contrast on same side

of the river as target. Fluent and disfluent trials analyzed separately.

Fluent trials Looks to contrast predicted whether the size of target would be

mentioned (Hyp 1) = Exp 1

86% of fluent trials during which speakers looked at contrast, the referring expression included a size modifier.

When there was no fixation to the contrast, modifier was only used in 19 % of trials

First fixation to the contrast was systematically related to the form of the utterance. (Hyp 2) = Exp 1

The mean 1st fixation to contrast for pre-nominal modifiers was887ms before onset of referring expression.

The mean 1st fixation to contrast with post-noun repairs was 14ms before onset.

Results

Disfluent trials When speakers fixated the contrast adj was used

on 91%. When contrast was not fixated ajd was used 34%. First fixation to contrast systematically related to

form of utterance. First fixations were on average 528ms before onset of

referring expression for utterance with pre-nominal modifiers.

First fixations were on average +819ms for post-noun repair utterances

Results.

Earliest fixations are associated with fluent prenominal utterance

Latest fixations are associated with a post-noun repair. Consistent with H3: suggests speaker was able to use

disfluency to buy time to include size in the noun phrase as a prenominla adj. rather than as a postnoun repair.

=Exp 1. Relationship btw time speakers 1st fixated the size contrast and the timing of the size adj. for postnoun repairs.

Results

Time to begin speaking Average time between the highlighting of target and NP onset <

2s ≠ Exp1. Utterance onset time was not significantly affected by the

form of the utterance For initially fluent utterances, speakers started pre-nominally

modified expressions 345 ms faster than expressions with post-noun repairs (not reliable)

For disfluent utterances pre-nominally modified expressions were only 120ms faster than utterances with postnoun repairs

Speakers uttered disfluent pre-nominally modified expressions on average 66ms faster than fluent pre-nominally modified utterances (not reliable)

General Discussion

3 results 1. Speakers typically used size adj. in a referential

expression only when there was a similar object in the referential domain that differed in size

2. Use of size adj was more likely if speaker had fixated on the size contrast.

3.Timing of first fixation to contrast was related to the form of referential expression Earlier – prenominal adj Later – postnoun repair

General Discussion

Preliminary support for hypotheses about interplay between message formulation and utterance planning.

When an error is detected in the utterance plan after speaking has begun, initiation of a repair may be delayed by preparation of original utterance.

Messages that will be mapped onto referring expressions can be constructed and passed onto utterance planning incrementally.

Eskerrik asko!