HAPTIC CINEMATOGRAPHY : AUTHORING HAPTIC ...fdanieau.free.fr/pubs/hs16_small.pdf9Principles 1. Staging 2. Continuity 3. Realism 4. Perception 5. Anticipation 6. Exaggeration 7. Synchronism

HAPTIC CINEMATOGRAPHY :AUTHORING HAPTIC FEEDBACKFOR MOVIESCreative Haptic Expression : Trends and Tools

April 8th , 2016

Fabien DANIEAUTechnicolor R&[email protected]

mailto:[email protected]

Content Introduction H-Studio Haptic Cinematography Principles for Haptic Cinematography Conclusion References

Content

1. Introduction

2. H-Studio

3. Haptic Cinematography

4. Principles for Haptic Cinematography

5. Conclusion

6. References

2

INTRODUCTION


Audiovisual Experience

◮ Evolution of the audiovisual experience◮ Improving the Quality of Experience◮ Focused on images and sound◮ Next step : sense of touch ? (Haptics)

4


Haptics and Audiovisuals

◮ Haptics already used in VR◮ Early results on haptics for

AV◮ Haptics

may enrich AV experience[O’Modhrain and Oakley, 2003]

◮ New medium to expresscontent[Magnenat and Bonanni, 2006]

◮ New eld of study :Haptic-AudioVisuals[El Saddik et al., 2011]

Virtual Reality Setup [Cirio et al., 2011]

4D Cinema Setup

ChallengeHow to augment audiovisuals with haptic feedback ?

5


Haptics and Audiovisuals

◮ Haptics already used in VR◮ Early results on haptics for

AV◮ Haptics

may enrich AV experience[O’Modhrain and Oakley, 2003]

◮ New medium to expresscontent[Magnenat and Bonanni, 2006]

◮ New eld of study :Haptic-AudioVisuals[El Saddik et al., 2011]

Virtual Reality Setup [Cirio et al., 2011]

4D Cinema Setup

ChallengeHow to augment audiovisuals with haptic feedback ?

5


Work ow

◮ Based on architecture for video streaming◮ Three steps

◮ Production◮ Distribution◮ Rendering

Audiovisual

media

Haptic

effects

Capturing

Synthesizing

Automatic

extractionNetwork

Screen &

Speakers

Haptic

devices

Audiovisual

renderer

Haptic

renderer

Automatic

extraction

E

N

C

O

D

E

R

E

N

C

O

D

E

R

D

E

C

O

D

E

R

D

E

C

O

D

E

R

[Danieau et al., 2013]

6

H-STUDIO


Overview of the Authoring Tool

8


Overview of the Authoring tool

◮ Edition of motion e ects◮ Capture of motion data◮ Manual edition thanks to force-feedback device◮ Automatic extraction of vibration e ects◮ Preview of haptic e ects

9


Overview of the Authoring tool

◮ Edition of motion e ects◮ Capture of motion data◮ Manual edition thanks to force-feedback device◮ Automatic extraction of vibration e ects◮ Preview of haptic effects

9


Acquisition

◮ Capture device◮ Camera◮ IMU : linear acceleration

◮ Content

10


Preview of Haptic E ects

◮ Signal processing◮ Synchro with AV (haptic clap)◮ Low pass ltering◮ Gravity removal

◮ User feedback◮ Realistic movement

11


H-Studio 2

◮ Ported on Android device◮ Use embedded sensors

Edition with touchscreenFull body stimulationMultiple haptic e ects (vibration, temperature, motion, etc.)

[Mollet et al., 2015]

12


H-Studio 2

◮ Ported on Android device◮ Use embedded sensors◮ Edition with touchscreen

Full body stimulationMultiple haptic e ects (vibration, temperature, motion, etc.)

[Mollet et al., 2015] 12


H-Studio 2

◮ Ported on Android device◮ Use embedded sensors◮ Edition with touchscreen◮ Full body stimulation

Multiple haptic e ects (vibration, temperature, motion, etc.)



H-Studio 2

◮ Ported on Android device◮ Use embedded sensors◮ Edition with touchscreen◮ Full body stimulation◮ Multiple haptic e ects (vibration, temperature, motion, etc.)


HAPTIC CINEMATOGRAPHY


Haptic Cinematography

◮ How to associate haptic feedback to movies ?

Mainly used with action movies

Physical events : explosions, car chases, gun shots, etc.4D cinemas

Drama, comedy, romantic movies ?

Non-diegetic e ectsMove beyond enhancement of physical events only

14



◮ How to associate haptic feedback to movies ?◮ Mainly used with action movies

◮ Physical events : explosions, car chases, gun shots, etc.◮ 4D cinemas

Drama, comedy, romantic movies ?


14





◮ Drama, comedy, romantic movies ?


14





◮ Drama, comedy, romantic movies ?


14


Diegesis : de nition

◮ Non-diegetic e ect◮ E ect whose source is neither visible on the screen nor has

been implied to be present in the action◮ Example : non-diegetic sound

◮ Narrator’s commentary◮ Mood music

15


Taxonomy of Haptic E ects

Haptic

Effects

Diegetic

Global

Local

Non-

Diegetic

Camera

Editing

Music

Context


FocusStudy of cinematographic camera e ects

16


Taxonomy of Haptic E ects

Haptic

Effects

Diegetic

Global

Local

Non-

Diegetic

Camera

Editing

Music

Context


FocusStudy of cinematographic camera e ects

16


Cinematographic Camera E ects

◮ Typical cameras e ects◮ Arcing◮ Traveling◮ Tilting◮ Zoom-in◮ Vertigo◮ Crane shot◮ Dutch angle

Two types of haptic feedback

Cinematic direct mappingSemantic haptic metaphor

Proof-of-Concept

7 video sequencesHapSeat

17



◮ Typical cameras e ects◮ Two types of haptic feedback

◮ Cinematic→direct mapping◮ Semantic→haptic metaphor

Proof-of-Concept

7 video sequencesHapSeat

17



◮ Typical cameras e ects

◮ Two types of haptic feedback◮ Cinematic→direct mapping◮ Semantic→haptic metaphor

◮ Proof-of-Concept

◮ 7 video sequences

◮ HapSeat

17


Cinematic Model

◮ Follows camera movement◮ Example with Arcing, Tilting and Zoom-in

18


Semantic Model

◮ Based on the semantic ofthe Camera E ect

◮ Crane shot→Flying away

◮ Dutch angle→Instability

◮ Arcing→Intensi cation

◮ Traveling→Crab walk

◮ Tilting→Inferiority

◮ Zoom-in→Walk forward

◮ Vertigo→Vertigo

◮ Designed with H-Studio19


User Study

◮ Objective : study of the in uence of the haptic e ects basedon camera motions on the QoE

◮ Hypothesis : New haptic e ects increases QoE◮ Variables

◮ 7 video sequences◮ 4 haptic conditions (cinematic, semantic, random, none)

◮ Measure◮ Pairwise comparison (78 couples)◮ Score computed for each model

◮ 38 participants (x̄ = 36.3, σx = 10.4)

20


User Study

◮ Haptic e ects enhance QoE

◮ Cinematic > Semantic≈ None > Random

◮ Semantic > None◮ Vertigo◮ Arcing◮ Tilting

◮ Discussion◮ Metaphors well

understood◮ Dynamics of haptic

feedback should besimilar to dynamics AVcontent

0

0,2

0,4

0,6

0,8

1

QoE for all sequences

Cinematic Random None Semantic

Score

Crane Shot Dutch Angle Traveling Zoom Vertigo Arcing Tilting0

0,2

0,4

0,6

0,8

1

QoE per sequence

Cinematic Semantic Random None

Sco

re

21


Application

22

PRINCIPLES FOR HAPTICCINEMATOGRAPHY


Inspiration

◮ Guidelines for designing haptic e ects ?◮ Inspired from Disney’s 12 principles of animation

◮ realistic animations◮ law of physics, emotional timing, character appeal, etc.

[Thomas and Johnston, 1995] 24


Principles for haptic cinematography ?

◮ Lot of tests, demonstrations, discussion with professionals◮ Work with student in cinematography◮ H-Studio◮ Blender Foundation Movies

9 Principles for the Haptic Cinematography [Guillotel et al., 2016]

25





25





25


9 Principles

1. Staging2. Continuity3. Realism4. Perception5. Anticipation6. Exaggeration7. Synchronism8. Kinetic Interpolation9. Diegesis

26


9 Principles

1. Staging

2. Continuity3. Realism4. Perception5. Anticipation6. Exaggeration7. Synchronism8. Kinetic Interpolation9. Diegesis

26


9 Principles

1. Staging2. Continuity

3. Realism4. Perception5. Anticipation6. Exaggeration7. Synchronism8. Kinetic Interpolation9. Diegesis

26


9 Principles

1. Staging2. Continuity3. Realism

4. Perception5. Anticipation6. Exaggeration7. Synchronism8. Kinetic Interpolation9. Diegesis

26


9 Principles

1. Staging2. Continuity3. Realism4. Perception

5. Anticipation6. Exaggeration7. Synchronism8. Kinetic Interpolation9. Diegesis

26


9 Principles

1. Staging2. Continuity3. Realism4. Perception5. Anticipation

6. Exaggeration7. Synchronism8. Kinetic Interpolation9. Diegesis

26


9 Principles

1. Staging2. Continuity3. Realism4. Perception5. Anticipation6. Exaggeration

7. Synchronism8. Kinetic Interpolation9. Diegesis

26


9 Principles

1. Staging2. Continuity3. Realism4. Perception5. Anticipation6. Exaggeration7. Synchronism

8. Kinetic Interpolation9. Diegesis

26


9 Principles

1. Staging2. Continuity3. Realism4. Perception5. Anticipation6. Exaggeration7. Synchronism8. Kinetic Interpolation

9. Diegesis26


9 Principles

1. Staging2. Continuity3. Realism4. Perception5. Anticipation6. Exaggeration7. Synchronism8. Kinetic Interpolation9. Diegesis

26

CONCLUSION


Conclusion

◮ Haptics→strong potential for enhancing AV◮ Authoring tool : H-Studio

◮ Capture◮ Manual edition◮ Preview with force-feedback device◮ Full body

◮ Haptic Cinematography◮ Taxonomy of haptic e ects◮ 9 principles

28


Thanks ! Questions ?

29

REFERENCES


References I

Cirio, G., Marchal, M., Hillaire, S., and Lecuyer, A. (2011).Six degrees-of-freedom haptic interaction with uids.Visualization and Computer Graphics, IEEE Transactions on, 17(11) :1714–1727.Danieau, F., Fleureau, J., Guillotel, P., Mollet, N., Christie, M., and Lecuyer, A.(2014).Toward haptic cinematography : enhancing movie experiences with camera-basedhaptic e ects.MultiMedia, IEEE, 21(2) :11–21.Danieau, F., Lécuyer, A., Guillotel, P., Fleureau, J., Mollet, N., and Christie, M.(2013).Enhancing audiovisual experience with haptic feedback : a survey on HAV.volume 6, pages 193–205.El Saddik, A., Orozco, M., Eid, M., and Cha, J. (2011).Haptic technologies - bringing Touch to multimedia.Springer.Guillotel, P., Danieau, F., Fleureau, J., and Rouxel, I. (2016).Introducing basic principles of haptic cinematography editing.Eurographics Workshop on Intelligent Cinematography and Editing.

31


References II

Magnenat, N. and Bonanni, U. (2006).Haptics in virtual reality and multimedia.IEEE Multimedia, 13(3) :6–11.Mollet, N., Dillon, T., Danieau, F., and Le Clerc, F. (2015).A work ow for next generation immersive content.IBC.O’Modhrain, S. and Oakley, I. (2003).Touch TV : adding feeling to broadcast media.In European Conference on Interactive Television : from Viewers to Actors.Thomas, F. and Johnston, O. (1995).The Illusion of Life : Disney Animation.Hyperion, Disney edition.

32

Documents

HAPTIC CINEMATOGRAPHY : AUTHORING HAPTIC ...fdanieau.free.fr/pubs/hs16_small.pdf9Principles 1. Staging 2. Continuity 3. Realism 4. Perception 5. Anticipation 6. Exaggeration 7. Synchronism