11/14/2011

1

G89.2223 Perception

Sensory Cue Combination

Laurence T. Maloney

Some examples of cues to depthor shape

Retinal projection depends on size and distance

Monocular depth cues Epstein (1965) familiar size experiment

How far away is the coin?

Cast Shadows Shading and contour

11/14/2011

2

Texture

1. Density2. Foreshortening

3. Size

Linear perspective

Binocular depth cues Vergence Angle As One Binocular Source

Vergence Angle As One Binocular Source

Vergence Angle As One Binocular Source

11/14/2011

3

Wheatstone stereoscope (c. 1838)

Sir Charles Wheatstone

Dual mirror stereoscope

Uncrossed disparity

Zero retinal disparity

Crossed disparity

Disparity

zero disparity

uncrossed disparity

crossed disparity

Fixation point

Cues in conflict

What cues?

11/14/2011

4

Ames room Ames room

Ames room

Cue Combination

Rock & Victor (1964)

Visually and haptically specified shapes differed.What shape is perceived?

View object through distorting lens while exploring object haptically

Irv Rock

Rock & Victor (1964)Experimental Design

11/14/2011

5

Rock & Victor (1964)Results

1.90 0.98 1.85

13.4 23.1 14.1 mm

14.1 20.5 14.5 mm

Rock & Victor (1964)Results

1.90 0.98 1.85

13.4 23.1 14.1 mm

14.1 20.5 14.5 mm

Visual Capture

How should we combine cues?

V

HS

S

haptic size estimate

visual size estimate

random variables

Modeling Cue Combination

,

,

H

V V

HS Gaussian s

S Gaussian s

true location

s

VS HS

s

,

,

H

V V

HS Gaussian s

S Gaussian s

standard deviation

s

VS HS

s

11/14/2011

6

location-scale families

s

VS

,V VS Gaussian s

V

VE S s

10 ,1

10 ,2

H

V

S Gaussian cm cm

S Gaussian cm cm

s

VS HS $10 if you are within1 cm of s

Which cue?

Chances of winning?

s

VS HS

Can we do better by combining cues?

1V HS wS w S

weighted linear combination

s

VS HS

1

(1 )V HE S wE S w E S

ws w s s

s

VS HS

1

(1 )V HE S wE S w E S

ws w s s

Gaussian ,S Gaussian s w

What is SD?

s

VS HS

22

2 2 2 2

1

(1 )

V H

V H

Var S w Var S w Var S

w w

a parabola in w

11/14/2011

7

2 2 2 2(1 )V HVar S w w

0 1w

2V

2H

2 2 2 2(1 )V HVar S w w

0 1w

2V

2H

could it be?

s

VS HS

2 2

2

2 2

2 2(1 ) ! 0V H

H

V H

Var Sw w

w

w

0 1w

2V

2H

s

VS HS

2

2 2H

V H

w

some examples ….

2 2 2 2(1 )V HVar S w w

0 1w

2V

2H

optimal cue combination

effective cue combination

Rock & Victor (1964)

Visually and haptically specified shapes differed.What shape is perceived?

View object through distorting lens while exploring object haptically

Irv Rock

Why visual capture?

11/14/2011

8

Visual/Haptic Setup

Visual Capture ?

Why should vision be the “gold standard”all other modalities are compared to?

SVH wV SV wH SH

Variance

Weights

wV H

2

V

2 H

2

1

VH

2

1

V

2

1

H

2

Visual Capture ?

Why should vision be the “gold standard”all other modalities are compared to?

SVH wV SV wH SH

Variance

Weights

wV H

2

V

2 H

2

1

VH

2

1

V

2

1

H

2

Visual Capture ?

Why should vision be the “gold standard”all other modalities are compared to?

SVH wV SV wH SH

Variance

Weights

wV H

2

V

2 H

2

1

VH

2

1

V

2

1

H

2

Visual Capture ?

Why should vision be the “gold standard”all other modalities are compared to?

SVH wV SV wH SH

Variance

Weights

wV H

2

V

2 H

2

1

VH

2

1

V

2

1

H

2

11/14/2011

9

Experimental Outline

1) determine (& manipulate) within-modality variances• discrimination thresholds (2-IFC, constant stimuli)

2) make predictions for combined performance • using MLE model to predict weights & combined variance.

3) measure combined performance & compare to prediction• similar to within-modality 2-IFC discrimination task (get PSE and thresholds)

Standard Comparison

no feedback!

2‐IFC Task

Visual-HapticVisual-alone Haptic-alone

Three Conditions

Predictions

Determining Within‐Modality Variance Determining Within‐Modality Variance

Threshold

Determining Within‐Modality Variance

Threshold

Determining Within‐Modality Variance

Threshold

11/14/2011

10

Determining Within‐Modality Variance

Threshold

Determining Within‐Modality Variance

Threshold

From Variance to Threshold

visual-haptic varianceestimators weights

Predicted weights for combined performance from within-modal data

Predicted combined threshold from within-modal data

wV H

2

V

2 H

2

visual-haptic thresholdestimators weights

wV JNDH

2

JNDV2 JNDH

2

1

JNDVH2

1

JNDV2

1

JNDH2

1

VH

2

1

V

2

1

H

2

JNDi 2 i JNDi 2 i

Visual‐Haptic Discrimination

Visual‐Haptic Discrimination Visual‐Haptic Discrimination

11/14/2011

11

Visual‐Haptic Discrimination Visual‐Haptic Discrimination

Empirical Thresholds and Weights

“visual capture”

“haptic capture”

Weights & PSEs

Individual Differences

LAS

LAS

RSB

1

RSB

MOEE

mp

iric

al V

isu

al W

eig

ht

MOE

HTE

HTE

JWW

JWWKML

KML

0% noise

133% noise

Predicted Visual Weight

0.80.60.40.20

1

0.8

0.6

0.4

0.2

Conclusions

Combination reduces variance.

Linear weighting scheme for visual-haptic perception.

Explains behavior like “visual capture” or visual dominance.i.e, vision is given a weight of ~ 1.0 if the variance of the visual estimate is less then the variances of the other modalities.