Haptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality Environments

Konstantinos Moustakas, Assistant ProfessorKonstantinos Moustakas, Assistant Professor

Electrical and Computer Engineering Department, University of Electrical and Computer Engineering Department, University of PatrasPatras

•• WhatWhat isis haptichaptic interaction?interaction?•• HapticHaptic psychophysicspsychophysics outlineoutline•• HapticHaptic interactioninteraction essentialessential processingprocessing::

Outline Haptic interactionHaptic interaction

OutlineOutline

•• HapticHaptic interactioninteraction essentialessential processingprocessing::–– CollisionCollision detectiondetection–– HapticHaptic renderingrendering

•• ApplicationsApplications•• FutureFuture researchresearch directionsdirections

• Wikipedia, Haptics: Haptics refers to the sense oftouch (from Greek άπτω = "I fasten onto, I touch).

• Merriam-Webster, Haptic: relating to or based onthe sense of touch

Introduction Haptic interactionHaptic interaction

HapticsHaptics

the sense of touch• Haptics is the science of applying touch

(tactile) sensation and control to interactionwith computer applications.

Introduction Haptic interactionHaptic interaction

HapticsHaptics

• Tactile information– The responses of

the receptors of the skin in contact with an objectan object

• Kinesthetic information– Net forces applied

to limbs

Introduction Haptic interactionHaptic interaction

Haptic interaction Haptic interaction -- ImportanceImportance

• The sense of touch is underestimated!• Loss of the sense of touch can cause impairments that

cannot be compensated by sight– Hand dexterity– Haptic capabilities, grasping, etc.– Walking– Walking– Limb position perception– Eating– Speaking

• In virtual environments the loss of the sense of touch can impair performance and interaction efficiency

HapticHaptic interactioninteraction -- ImportanceImportance

Introduction Haptic interactionHaptic interaction

Strike a match (with vision)

Normal case

7.5 sec

Anesthesia

29 sec

Haptic Psychophysics Haptic interactionHaptic interaction

Different types of tactile sensorsDifferent types of tactile sensors

Haptic Psychophysics Haptic interactionHaptic interaction

Surface encoding Surface encoding ––deformation mattersdeformation matters

Haptic Psychophysics Haptic interactionHaptic interaction

Spatial resolutionSpatial resolution

Haptic Psychophysics Haptic interactionHaptic interaction

Temporal resolutionTemporal resolution

~1 kHz constraint

while in graphics~25 Hz constraint

Haptic Psychophysics Haptic interactionHaptic interaction

• Tactile– Surface texture– Orientation– Shape

What do we sense?What do we sense?

• Haptic– Mass/Weight– Force– Stiffness– Shape

– Size– Pressure– Curvature– Motion

– Stiffness– Viscosity

• Thermal– Cooling– Warming

Haptic Rendering Haptic interactionHaptic interaction

HapticsHaptics & Environment& Environment

Haptic Devices Haptic interactionHaptic interaction

• Haptic devices:– Tactile devices– Force feedback devices

• Exoskeletons• One point of interaction devices

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devicesHaptic devices

Haptic Devices Haptic interactionHaptic interaction

Haptic devices VS Human SensorsHaptic devices VS Human Sensors

Haptic devicesHuman sensors

Many sensors One sensor (x N)

Direct tactile sensation Only through motion

Deformation N/A

• Haptic devices act as a filter to the human tactile/haptic sensors.

• How would you see a visual scene with such a filter?

Haptic Devices Haptic interactionHaptic interaction

Haptic devices VS Human SensorsHaptic devices VS Human Sensors

Haptic Rendering Haptic interactionHaptic interaction

Haptic vs Graphics rendering processesHaptic vs Graphics rendering processes

ViewViewTransformTransform

LightingLighting ProjectionProjection TexturingTexturingScene Scene

TraversalTraversal

GeometryGeometry RasterizerRasterizerApplicationApplication DisplayDisplay

ForceForce TactileTactileTraversalTraversal DisplayDisplay

ForceForceCalculationCalculation

Force Force SmoothingSmoothing

ForceForceMappingMapping

HapticHapticTexturingTexturing

Collision Collision DetectionDetection

Collision detection Haptic interactionHaptic interaction

• Collision is one of natures powerful tools:– Object interaction

• Walking• Machines are based on collisions

Collision detectionCollision detection

• Machines are based on collisions

– Sound source

• In the real world physical laws and matter take-over

• In the virtual world???

Collision detection Haptic interactionHaptic interaction

Collision detectionCollision detection

In no case a trivial issue!!!

Collision detection Haptic interactionHaptic interaction

Bounding volume hierarchiesBounding volume hierarchies

Collision detection Haptic interactionHaptic interaction

Bounding volume hierarchies (video)Bounding volume hierarchies (video)

Collision detection Haptic interactionHaptic interaction

Force feedbackForce feedback

A. Vogiannou, K. Moustakas, D. Tzovaras and M.G. Strintzis, “Enhancing Bounding Volumes using Support Plane Mappings for Collision Detection”, Eurographics Computer Graphics Forum, vol. 29, no. 5, pp. 1595-1604, August 2010.

Haptic Rendering Haptic interactionHaptic interaction

Haptic rendering basicsHaptic rendering basics

Haptic Rendering Haptic interactionHaptic interaction

Haptic rendering: Stiffness vs StabilityHaptic rendering: Stiffness vs Stability

Force Calculation Force Calculation ––Virtual wallVirtual wall

Insufficient stiffnessInsufficient stiffnessV V <<<<<<<< 00

Virtual wall Virtual wall Moving into the wallMoving into the wall

F F

timetime

K K wallwall •• ∆∆ x + B vx + B vF =F =

SolutionsSolutionsHigh frequencyHigh frequency

Virtual couplingVirtual couplingGod object God object –– virtual proxyvirtual proxy

Haptic Rendering Haptic interactionHaptic interaction

Force discontinuitiesForce discontinuities

Haptic Rendering Haptic interactionHaptic interaction

Force shadingForce shading

Haptic Rendering Haptic interactionHaptic interaction

• Height maps• Friction/Damping fields• Stochastic textures

Haptic texturingHaptic texturing

• Virtual textures/bumps

The force field haptic rendering method

Haptic rendering Haptic interactionHaptic interaction

method

K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia , vol. 14, no. 1, pp. 62-72, January 2007.

Haptic rendering Haptic interactionHaptic interaction

From the distance field to the force fieldFrom the distance field to the force field

-di -di

di

Distance field

Force field

K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia , vol. 14, no. 1, pp. 62-72, January 2007.

Haptic rendering Haptic interactionHaptic interaction

Haptic renderingHaptic rendering• The force feedback is calculated for a given

point through:

• whereD(x) andF(x) are the values and the

∈ 3x R

• whereD(x) andF(x) are the values and the vector of the distance and force field respectively

-di -di

di

Distance field Force field

SQ-Map: Efficient layered collision detection and haptic renderingdetection and haptic rendering

K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93, January 2007.

Haptic rendering Haptic interactionHaptic interaction

• Motivation: New method fast as the distance fields, without high memory requirements

• SQ-Map steps– Preprocessing

SQSQ--Map overviewMap overview

– Preprocessing• Object approximate segmentation• Superquadric modeling• Distance map generation

– Run-time processing• Layered collision detection (3 levels of processing)

K. Moustakas, D. Tzovaras and M.G. Strintzis, “SQ-Map: Efficient Layered Collision Detection and Haptic Rendering”, IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 1, pp. 80 - 93, January 2007.

Haptic rendering Haptic interactionHaptic interaction

SQSQ--Map distance mapsMap distance maps

Haptic rendering Haptic interactionHaptic interaction

Collision detection: Layer 1Collision detection: Layer 1

( )

2

1

12 2

22 2

1 2 3

, ,x y z

F x y za a a

εε

εε ε

= + +

> 1

Haptic rendering Haptic interactionHaptic interaction

Collision detection: Layer 1Collision detection: Layer 1

( )

2

1

12 2

22 2

1 2 3

, ,x y z

F x y za a a

εε

εε ε

= + +

< 1

Haptic rendering Haptic interactionHaptic interaction

Collision detection: Layer 2Collision detection: Layer 2

Haptic rendering Haptic interactionHaptic interaction

Collision detection: Layer 2Collision detection: Layer 2

dDSQ

P

If d>DSQ collision is detected

Haptic rendering Haptic interactionHaptic interaction

• Similar processing to layer 2 with two add-ons– Possibility to perform distance map

subdivision so as to increase accuracy

Collision detection: Layer 3Collision detection: Layer 3

subdivision so as to increase accuracy– The Layer 3 distance map is used that

projects also mesh vertices onto the superquadric surfaces

Haptic rendering Haptic interactionHaptic interaction

• Lemma: – The distance map and the superquadric can

provide an exact representation of the 3D model iff function fC that maps all points of the

Accuracy analysisAccuracy analysis

model iff function fC that maps all points of the object’s surface onto the superquadric is injective.

– Proof in the text

Haptic rendering Haptic interactionHaptic interaction

Accuracy analysisAccuracy analysis

Concave region of typeQQ2

Concave region of type Q1

Haptic rendering Haptic interactionHaptic interaction

Haptic renderingHaptic rendering

Force magnitude estimation

Haptic rendering Haptic interactionHaptic interaction

Haptic renderingHaptic rendering

Force direction estimation

Can be analytically calculated

Haptic rendering Haptic interactionHaptic interaction

Haptic renderingHaptic rendering

Advantages:

• Rapid estimation of the reaction force

• Possibility to analytically process and handle the • Possibility to analytically process and handle the force feedback

Friction:

Haptic texture:

Gaussian noise

Haptic rendering Haptic interactionHaptic interaction

ComparisonsComparisons

Haptic rendering Haptic interactionHaptic interaction

Qualitative resultsQualitative results

OBB SQ-Map, Layer 1

SQ-Map, Layer 2 SQ-Map, Layer 3

Haptic rendering Haptic interactionHaptic interaction

Qualitative resultsQualitative results

Applications Haptic interactionHaptic interaction

• Surgical simulation

• Telemanipulation, robot-assisted surgery

Application examplesApplication examples

• Computer aided design

• Entertainment• Haptic Visualization• Applications for the

disabled• …

Applications Haptic interactionHaptic interaction

Haptic mapsHaptic maps

Haptic map with embedded semantic iformation

K. Moustakas, G. Nikolakis, K. Kostopoulos, D. Tzovaras and M.G. Strintzis, “Haptic Rendering of Visual Data for the Visually Impaired”, IEEE Multimedia , vol. 14, no. 1, pp. 62-72, January 2007.

Applications Haptic interactionHaptic interaction

Haptic browserHaptic browser

Applications Haptic interactionHaptic interaction

Virtual reality cane simulationVirtual reality cane simulation

D. Tzovaras et.al., “Design and implementation of haptic virtual environments for the training of the visually impaired, IEEE Neural Systems and Rehab. Eng., 2004

Applications Haptic interactionHaptic interaction

Mixed reality cane simulationMixed reality cane simulation

Virtual environment

Virtual

Magnetic sensor 1

CyberGrasp

D. Tzovaras, K. Moustakas, G. Nikolakis and M.G. Strintzis, "Interactive Mixed Reality White Cane Simulation for the Training of the Blind and the Visually Impaired", Springer Journal on Personal and Ubiquitous Computing, vol.13, no.1, pp.51-58, January 2009.

Virtual cane

Cane replica

CyberGrasp

Magnetic sensor 2

VR game for the hearing and the visually impaired

Applications Haptic interactionHaptic interaction

visually impaired

K. Moustakas, L. Dybkjaer, O. Aran, D. Tzovaras and N.O. Bernsen, "Communication Between Blind and Hearing Impaired People Through a Multimodal Interactive Game", IEEE Multimedia , accepted for publication.

Applications Haptic interactionHaptic interaction

The ideal caseThe ideal case

Applications Haptic interactionHaptic interaction

ArchitectureArchitecture

Applications Haptic interactionHaptic interaction

Game stepsGame steps

Applications Haptic interactionHaptic interaction

Game videoGame video

Future work Haptic interactionHaptic interaction

• Haptic visualization– Use the haptic channel to complement vision

in information visualization applications

• Haptic interaction with videos

Future directionsFuture directions

• Haptic interaction with videos– Highly dynamic

• Content• Interaction

– Partial input of the environment, ill-posed

• Towards a theory of haptic rendering…

Haptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality EnvironmentsHaptic Interaction in Virtual and Mixed Reality Environments

Thank you for your

Konstantinos Moustakas, Assistant Professor Konstantinos Moustakas, Assistant Professor moustakas@ece.upatras.grmoustakas@ece.upatras.gr

Electrical and Computer Engineering Department, University of Electrical and Computer Engineering Department, University of PatrasPatras

Thank you for your attention