Cross-modal perception of motion-based visual-haptic stimuli

Ian Oakley & Sile O’Modhrain

Palpable Machines Research Group

http://www.mle.ie/palpable

ian@mle.iesile@media.mit.edu

Media Lab EuropeSugar House Lane

BellevueDublinIreland

Overview

• Background – Touch in Broadcast Media

• The Touch TV Project• Role of cross modal perception in

TTV• Experimental Study - Pilot• Conclusions and Future Work

Touch in Broadcast Media

• Adding haptic information to a broadcast may enhance viewing

• Motivation Consumer demand for richer media

• Led to advances in audio and video technologies• Touch may provide greater richness• Latest advances not enhanced viewing experience

e.g. digital and interactive TV

Evidence supporting this assertion• Touch in virtual environments linked to increased

ratings of presence and co-presence• Adoption of touch in gaming and simulator systems

The Touch TV Project

• Ongoing work investigating integration of touch into broadcast content Currently, focused on a practical and

theoretical exploration of the design space available for this work

Involves a number of research threads ranging from authoring of media to psychological study

Cross-Modal Perception in TTV• Touch TV inherently cross-modal

Involves construction of immersive and involving media with graphics, audio and touch components

Techniques for combining graphics and audio established in this context

• E.g. visual capture

How to add touch to this format is an open question

• How do we add haptic information to an audio-visual stream ensuring that it will be interpreted as we intend?

Experimental Study

• Interested in combination of visual and haptic stimuli representing movement in complex graphical scenes Relatively complex stimuli

• Initial questions: Can haptic cues be associated with visual

movement? What is the effect of visual distractors? What is the effect of visual target position? What kind of haptic cues should we use?

• Attempted to address these questions in a pilot study

Experimental Study

• In each trial, the visual display consisted of a number of bouncing balls controlled by a simple mathematical model

• Each ball had a different spring constant – different “bounciness” and was subject to gravity

• Force displayed on a PHANToM haptic interface either represented the velocity of one of the displayed balls, or of an unseen ball

• Participant’s task was to determine whether or not what they could feel related to one of the displayed balls

Experimental Study

• Varied: Number of distractors (5 possibilities) Target spring constant (6 possibilities) Target position (5 possibilities) Haptic-Visual match (2 possibilities)

• Total of 300 trials presented in random order• Practice consisted of a random selection of 50 trials• 10 Participants

All employees at Media Lab Europe 6 male, 4 female, all right-handed, none reported tactile or

visual impairment, range of familiarity with haptic devices (2.5 on a scale from 0-5)

• Participants wore noise cancelling headphones• Were able to rest between each trial

Experimental Study

• Gathered trial time, error rate• Trial time for visual presence/absence of

haptic stimuli:

• T-test revealed no significant differences

4.128 3.993

00.5

11.5

22.5

33.5

44.5

Mea

n

Tri

al T

ime

(sec

onds)

Presence Absent

Visual presence/ absence of haptic stimuli

Experimental Study

• Further analysis restricted to trials in which haptic ball was visually present

• Used ANOVA, post-hoc t-tests with Bonferroni CI adjustments

• Did not include trials in which subject made errors in temporal analysis However, relatively large error rate Collapsed position factor for temporal analysis

• Used a 5 (distractors) x 6 (spring values)• No interactions found (F=0.399, p=0.991)

• Error data 5 (distractors) x 6 (spring values) x 5 (positions) One interaction: Distractors by Spring Value (F=1.974, p<0.01)

Results

• Trial time for differing numbers of visual distractors

• ANOVA: F=8.235, p<0.001

00.5

11.5

22.5

33.5

44.5

Tri

al T

ime

(sec

onds)

0 1 2 3 4

Number of Distractors

No of Distracto

rs

0 1 2 3 4

0 X Not Sig Not Sig P<0.01 P<0.01

1 X x Not Sig P<0.05 P<0.01

2 X X X Not Sig P<0.05

3 X X X X Not Sig

4 X X x X X

Results of post-hoc t-tests

Results

• Trial time for differing spring values• ANOVA: F=83.881, p<0.001

0

1

2

3

4

5

6

Tri

al T

ime

(sec

onds)

0.15 0.2 0.25 0.3 0.35 0.4

Spring valuesLarger figures correspond to

"bouncier" balls

Spring Values

0.15 0.2 0.25 0.3 0.35 0.4

0.15 X Not Sig P<=0.01

P<=0.01

P<=0.01

P<=0.01

0.2 X X P<=0.01

P<=0.01

P<=0.01

P<=0.01

0.25 X X X P<=0.01

P<=0.01

P<=0.01

0.3 X X X X P<=0.01

P<=0.01

0.35 X X X X X Not Sig

0.4 X X X X X X

Results of post-hoc t-tests

Results

• Error rate for differing numbers of distractors

• ANOVA: F=20.614, p<0.001

0

5

10

15

20

25

30

Per

centa

ge

of Tri

als

Answ

ered

Inco

rrec

tly

0 1 2 3 4

No of DistractorsResults of post-hoc t-tests

No of Distracto

rs

0 1 2 3 4

0 X P<0.05 P<0.01 P<0.01 P<0.01

1 X x Not Sig P<0.05 P<0.01

2 X X X Not Sig P<0.01

3 X X X X Not Sig

4 X X x X X

Results

• Error rate for differing spring values

• ANOVA: F=4.861, p<0.001• Post-hoc t-tests revealed 0.15 spring value led

to significantly higher error rate than 0.4 spring value

0

5

10

15

20

25

Per

centa

ge

of

Tri

als

Answ

ered

In

corr

ectl

y

0.15 0.2 0.25 0.3 0.35 0.4

Spring valuesLarger figures correspond to

"bouncier" balls

Results

• Error rate for target positions• ANOVA: F=0.399, p<0.05• Post-hoc t-tests did not attain significance

0

5

10

15

20

25

Per

centa

ge

of Tri

als

Answ

ered

Inco

rrec

tly

-45 -25 0 25 45

Screen position (centre at 0)

Interpretation

• Subjects merged visual-haptic motion stimuli Error rate always less than chance

• Increase in trial time and error rate with increased numbers of distractors indicates serial search Haptic target did not “pop out” as in parallel search

• Equality of present and absent results suggests subjects performed an exhaustive search

• Possible speed/accuracy trade off Errors decreased, time increased with larger spring values

• Eccentricity effect in position error data suggests subjects focused on the centre of the screen

Conclusions and Future Work• Study successfully informed basic

questions relevant for our TTV domain Relevant to our authoring process

• Many questions remain Contrast same scenario with visual/audio and

visual/audio/haptic stimuli Role of visually distinct objects Is mapping force to motion the best solution?

• Basic scaling research on force presentation in this domain

…

Cross-modal perception of motion based visual-haptic stimuli

Ian Oakley & Sile O’Modhrain

Palpable Machines Research Group

http://www.mle.ie/palpable

sile@media.mit.eduian@mle.ie

Media Lab EuropeSugar House Lane

BellevueDublinIreland

EuroHaptics 2003 6th-9th July www.mle.ie/palpable/

eurohaptics2003