Visually-induced Visually-induced auditory spatial adaptation auditory spatial adaptation
in monkeys and humansin monkeys and humans
Norbert Kopčo, I-Fan Lin, Barbara Shinn-Cunningham, Jennifer Groh
Center for Cognitive Neuroscience, Duke UniversityHearing Research Center, Boston University
Technical University, Košice, Slovakia
2Nov 6, 2007 SFN 07 San Diego
Introduction
Vision affects auditory spatial perception (e.g. the ventriloquist effect)
Visually-induced shifts in sound localization can persist in the absence of visual stimuli (e.g. barn owl prism adaptation studies)
3Nov 6, 2007 SFN 07 San Diego
QUESTION
How does vision calibrate sound perception in primates?
- monkeys and humans
Unlike barn owls, monkeys and humans make eye movements. With every eye movement, the relationship between visual space and auditory space changes.
4Nov 6, 2007 SFN 07 San Diego
Visual and auditory spatial information are different!
VISION:
Retina provides “map” of object locations
Locations shift when eyes move
Frame of reference is “eye-centered”
5Nov 6, 2007 SFN 07 San Diego
Visual and auditory spatial information are different!
AUDITORY:Sound location calculated from interaural timing and level differences
Cue values do NOT shift when eyes move
Frame of reference is “head-centered”
6Nov 6, 2007 SFN 07 San Diego
Goals
Does visual-calibration of auditory space occur in eye-centered, head-centered, or a hybrid coordinate system?
Are humans and monkeys similar?
7Nov 6, 2007 SFN 07 San Diego
Experimental Setup
Audiovisual display as viewed by the subject
Horizontal location (degrees)
Ver
tical
loca
tion
(deg
rees
)
9 speakers in front of listener (~1 m distance), separated by 7.5° (humans) or 6° (monkeys)
Light-emitting diodes (LEDs)at three center speakers:- aligned with speakers, or- displaced to the left or to the right (by 5°-humans, 6°-monkeys)
2 LEDs below speaker array used as fixation points (FP)
Stimuli:Auditory stimulus:300-ms broadband noise burstAudio-Visual stimulus:Same noise with synchronously
lid LED.
8Nov 6, 2007 SFN 07 San Diego
-40 -30 -20 -10 0 10 20 30 40-40
-30
-20
-10
0
10
20
30
40
stimuli (degree)
resp
onse
(de
gree
)
trained A-only fp
no shift
shift to the same direction as AVshift to the opposite direction as AV
Experiment: Data Presentation Format
Sample Stimulus-Response Raw Data Plot
Actual Stimulus Location (°)
Per
ceiv
ed /
Res
pond
ed L
ocat
ion
(°)
Sample Plot Showing Bias in Responses (Response – Actual Location)
Actual Stimulus Location (°)B
ias
in R
espo
nses
(°) Rightward bias
Leftward bias
Rightw
ard
bias
Leftw
ard
bias
9Nov 6, 2007 SFN 07 San Diego
Experiment: Hypothesis
1. Induce Ventriloquist Effect for AV stimuli presented in central sub-region of space, keeping fixation point on the right ( +8°)
2. Test that adaptation results in the effect being present also for Auditory-only stimuli (same FP).
3. Induced shift will decrease outside the trained sub-region (for the same FP).
4. Move FP to the left. Test effect on Auditory-only stimuli:- No change Head-centered.- Response pattern moves with eyes Eye-centered.- Combined representations.
Stimulus Location (°)
Bia
s (°
)
Audiovisual display Expected Responses
FPLEDs
Speakers
10Nov 6, 2007 SFN 07 San Diego
Experiment: Procedure
Audiovisual display as viewed by the subject
Horizontal location (degrees)
Ver
tical
loca
tion
(deg
rees
)
One trial consists of:
1. Fixation point (FP) appears.
2. Subject fixates FP.
3. Target stimulus is presented (Audio-Visual or Auditory-only).
4. Subject saccades to perceived location of stimulus (humans instructed to always saccade to sound).
5. Monkeys only: Reward for responding within a criterion window (+- 10° from speaker).
11Nov 6, 2007 SFN 07 San Diego
Experiment: ProcedureExperiment divided into 1-hour blocks (12 for humans, 16 for monkeys; 7 humans, 2 monkeys).Within a block three types of trials, randomly interleaved:
Three types of AV stimuli (AV stimulus type kept constant within a block. Data also collected with AV FP on left. These data mirror-flipped to simplify presentation. For monkeys, also AV-aligned stimuli at +-30° to enforce non-linearity):
AV stimuli: 50 %
FPLEDs
Speakers
A-only,trained FP: 25% A-only,shifted FP: 25%
AV stimuli aligned Hypometric shift Hypermetric shift
12Nov 6, 2007 SFN 07 San Diego
Results: Humans
Stimulus Location (°)
Audiovisual display Expected Responses
FPLEDs
SpeakersHuman Behavior
Data collapsed acrossdirection of inducedshiftAV responses:- as expected
Trained FP A-onlyresponses:- Shift induced in trained sub-region- Generalization to untrained regions (asymmetrical)
Shifted FP A-onlyresponses:- Shift reduced in center regionHead-centered repre-sentation, modulated by eye position
Mean+SE
13Nov 6, 2007 SFN 07 San Diego
Results: Humans vs. MonkeysAudiovisual display
FPLEDs
SpeakersHuman Behavior Monkey data
(only hypometric)
AV responses:- as expected
Trained FP A-onlyresponses:- Shift in trained sub- region weaker- Generalization to untrained regions stronger (asymmetry opposite to humans)
Shifted FP A-onlyresponses:- Shifted with eyes
Representation moreeye-centered
Mean+SE
Monkey Behavior
Mean
6
6-24 0 24
-24 0 24
-24 0 24
Mean+individualsMean+SE
Mean+SE Mean+individuals
14Nov 6, 2007 SFN 07 San Diego
SummaryGood news – the main results are consistent across species:
Locally induced ventriloquist effect results in short-term adaptation, causing shifts in responses to A-only stimuli from trained sub-region.The pattern of induced shift is modified as the eyes move.
Bad news – there is a lot of differences between species:
Humans MonkeysRepresentation head-centered, eye-modulated eye-centered
Generalizationto untrained sub-regions more on the side away from FP opposite
Difference betweenhyper- and hypometricshifts no yes
Representation whenshift induced on side (data not shown) head-centered, no eye modulation eye-centered
15Nov 6, 2007 SFN 07 San Diego
DiscussionThe differences between species can be caused by:
1. differences in neural representation and/or learned behaviorE.g., monkeys often respond by double saccades
2. differences in the stage along the pathway at which the calibration occurs(Insert figure of ear->CN->SOC->IC->MGB->Aud. Ctx->Parietal Ctx->SC->oculomotor)E.g., in humans, the head-centered shift was induced fast, while the eye-centered modulation was much slower different time scales may imply adaptation at different stages
3. procedural differences (e.g., monkeys work for water)
Future work:
Humans: examine temporal and spatial factors influencing the eye-centered modulation.
Monkeys: attempt to induce stronger effect using larger A-V separation.