Pictures and spoken descriptions elicit similar eye movements during mental imagery, both in light and

in complete darkness

Roger Johansson, Jana Holsanova, Kenneth HolmqvistDepartment of Cognitive Science, Lund

University, Sweden

Eye mo

vement

s as a

windo

w to t

he min

d

Jörg B

runste

in

Theoretical background

• Mental images:

Do they exist?

And if yes, how do they work?

• Relevance: Where do people look when reflecting, remembering the last display etc.?

Hypotheses

• Similar EM when

seeing pictures,

remembering pictures,

and retelling verbal descriptions.

(a) EM indicate locations of objects

(b) Equally strong for retelling and remembering

(c) In light and in complete darkness

Experiments

1. Listen and retell in light2. Look and retell in light3. Listen and retell in complete darkness4. Look (in light) and retell in complete

darkness

• Data: EM during encoding (see/listen) and retrieval (describe/retell)

Methods

• 12 + 12 + 28 + 28 participants (50% (fe)male)

• SMI iView X at 50 Hz (bicycle helmet)– Glasses and lenses no problem, but

mascara

• White board 657 x 960 mm (about 36 x 38 inches) in 150 cm distance (about 59 inches)

Welcome to our lab!

Calibration

Phase 2

• Look: “You will soon see a picture. We want you to study the picture as thoroughly as possible. While you study the picture we will measure you pupil size.” (30 sec.)

• Listen: “Imagine a two-dimensional picture. There is…” (2 min. 6 sec.)

Look!

Retell!

Interview

1. What do you think the objective of this study was?

2. Rate the vividness of your visualization during the description phase on a scale ranging from 1(not very vivid) to 5 (extremely vivid).

3. Access whether you usually think in pictures or words.

Data

Experiments 1 - 4• Experiment 1: Listen and retell in light

(2 min + 1-2 min)• Experiment 2: Look and describe in

light (30 sec + 1-2 min)• Experiment 3: Listen and retell in

complete darkness (2 min + 1-2 min)• Experiment 4: Look (in light) and

describe (in complete darkness) (30 sec + 1-2 min)

Data analysis• Temporal reference:

• Holsanova (2001): eye-voice latencies typically between 2 and 4 sec.

– listening: looking after hearing (2.1 sec)

– retelling: looking before or after telling (0.29 sec.)

5 sec. before and after onset

Data analysis• Spatial reference:

• Global correspondence: looking to correct position relative to complete scene (direction + distance) (up, down, left, right; full/half distance; stand still)

• Local correspondence: looking into correct direction

• No correspondence: neither location nor direction within time window

Results

Exp. Local Global

Listen (L) 64.3 54.8

Retell (L) 74.9 55.2

Describe (L) 74.8 54.4

Listen (D) 60.8 27.6

Retell (D) 64.9 35.1

Describe (D) 69.0 40.5

Discussion: Content

• Eye movements reflect positions of objects

• Retelling = describing from memory

• Functional role of eye movements for mental images as spatial indexes (working memory, simulated vision, utterance planning)

Discussion: Methods

• Complex pictures and images• Spatial arrangements instead directions only• Relative ROIs instead of absolute ones for

scaling effects• Mental images instead of visual percepts• Tracking in light and in complete darkness• Minor: don’t say that they track directions but

calibrate participants