Upload
rolf-john-rose
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Enabling enactive interaction in virtualized experiences
Stefano Tubaro and Augusto SartiDEI – Politecnico di Milano, Italy
Nome relatore
How do we enable virtualized enaction with How do we enable virtualized enaction with incomplete sensory stimulation?incomplete sensory stimulation?
Gaining enactive knowledge from a mediated experience
Enactive Knowledge:information gained through perception-action interaction
• usually refers to an environment or a process (task)• inherently multimodal as it requires the
coordination of the various senses• knowledge acquired in a mediated form
is usually in symbolic or iconic form• enactive knowledge is based on the
active use of our senses• direct and intuitive (based on experiece)• often acquired without being aware of it
• Research on enaction is mostly focused on developing enactive multimodal interfaces
Nome relatore
Realistic or plausible?
• Aside from videogames, where plausibility plays
a dominant role, enactive knowledge acquisition
is usually aimed at learning about reality.
This is why mediated enaction usually relies on
realism and sensorial transparency, which makes
its applicability rather limited
• More promising (for massive use) is to accept virtualized enaction to be
based on reduced sensory perception and develop
• methodologies that allow the user to develop reliable
mappings between real and virtual enactive knowledge
• applications that are able to guess whether such mappings are
correctly established in a closed-loop fashion
Nome relatore
An example: the virtual dressing room
The e-market of articles of clothing is slumping because it offers no virtual try-on at home
The impact of applications such as virtual mirrors has been modest because of high cost; modestrealism; and lack of multimodal interaction
Research is needed for developing virtual try-on applications based on physically and perceptually plausible interactional rendering with reduced sensory perception (audio-visual), and limited realism, as long as we know how to• render what is perceptually relevant for a virtual experience that
can be instinctly remapped onto real life by the user • correctly map haptic interaction onto audio-visual interaction
(sensory substitution through cross-modal processing)• correctly map real interaction onto virtual interaction exploiting the
user’s experiential knowledge
Nome relatore
An example: the virtual dressing room
The mappings between experiential memory and virtual experience need
be established in a closed-loop fashion by an intelligent HMI, which is
able to read metacommunication cues carried by posture, gestures, gait,
facial expression, voice prosody, etc.
The HMI must become enactiveThe HMI must become enactive and learn on the fly whether the and learn on the fly whether the
virtual experience is being virtual experience is being effectively remapped onto the effectively remapped onto the
real worldreal world
Nome relatore
How do we enable enactive virtualized interpersonal How do we enable enactive virtualized interpersonal interaction with incomplete sensory perception? interaction with incomplete sensory perception?
Next step: enactive interpersonal interaction in virtual spaces
Interpersonal interaction is based on a closed-loop exchange of• symbolic or iconic information: Semantic information exchanged through
speech, images, intentional gestures, display of clothes or specific objects, ...
• meta-communication cues: body language (face expression, gestures, body posture, gait, etc.), voice prosody ad expression, tactile cues ...
Interactional meta-communication is• purely enactive• inherently multimodal• instinctive (reduced awareness)
Nome relatore
Issues in virtualized interactional experiences
Two options for enactive interaction• Interaction conveyed through signals
• metainformation to be captured, transferred and rendered as transparently and completely as possible
• e.g. video and audio 3D• problems: lack of effective and inexpensive
haptic sensors/display• Interaction mediated by models
• e.g. 3D body models (avatars), interactional sounds, etc.
• needs: compensate for incomplete and imperfect sensing and displaying technology with intelligent sensors, affective and cross-modal processing, perceptual multimodal displays, etc.
Again, Again, the HMI must become enactivethe HMI must become enactive and be aware of and be aware of the impact of the virtual experience onto the usersthe impact of the virtual experience onto the users
Nome relatore
In perspective
• Natural human-machine interaction • Multimodal interaction at multiple
levels of abstracion: from physical to emotional
• Enactive machines• Machines that learn from the
interaction with humans• Machines that learn from multimodal
observations of human-to-human interaction
Point of convergence between• Research on intelligent sensing
• computer vision, audio and acoustic analysis and processing, pattern recognition, multimodal fusion, ...
• Research on multimodal sensors and displays• Research on behavioural psychology
(social learning theory), and sociology