Hyper Communication: Toward the Creation

of New Media for the Information Network

Ryohei Nakatsu

ATR Media Integration & Communications

Research Laboratories

Contents

Hyper Communication

Approach toward Hyper Communication

Technologies for Hyper Communication

Examples of Hyper Communication System

Hyper Communication

Telephone, Teleconferencing

Multimedia Communications

Future Media

Movies

Video Game

TVNovel, Poem

Toward the Realization of Future Media

New Trend in Various Kinds of Media

Communications /Entertainment

in Cyberspace

Telecommunications (Communications)

Movies (Entertainment) Video Game (Amusement)

Internet

Roll Playing GameCG Movies, Digital Movies

in Cyberspace

Communications Filling the Gap of Distance, Time, and Culture

Natural Communications with Computers

Hyper Communication

• ECreation of Cyberspace (Cyberspace Generation)

• EWarp into Cyberspace (Metamorphosis)

(Agent Generation)

(Nonverbal Communications)

(Communication Support)

• EOperation of Materials in Cyberspace (Cyberspace Handling)

• ECommunication with Characters in Cyberspace

Technologies for Hyper Communication

Art and Technology

Artists

Engineers-Designers

Artists-Engineers

Collaboration between Engineers

and Artists

Engineers

Engineers

Artists

Communications Channels

Verbal Channel

Nonverbal Channel

Communication Model

Communication

(Language)

Interaction

Reaction

A I

Interactive Art

Traditional Art

Interactive Art

Artists

Audiences

AudiencesArtists

Artworks

Artworks

Artistic Approach v.s. Engineering Approach

Art

Technology

based on 'kansei' (sensitivity) and

intuition of artists

based on analysis of human behaviors and

utilization of image/speech processing

technologies

Engineering Approach vs.

Artistic Approach

An Example of

an Artistic Approach

•i"MIC" by Naoko Tosa•j

Surprise

Happy

An Example of

an Engineering Approach

Cyberspace Generation

Realistic Environment Generation

Hyper-Realistic Environment Generation

Merging of Real and CG Images

Real Scene Image CG Scene Image

Interactive Environment

• EEnvironments are fixed

• EHumans select environments

depending on communications.

• EEnvironments are flexible

• EEnvironments vary depending

on communications

Interactive

Environment

Communication

Environment

(Laurent Mignonneau & Christa Sommerer for ATR MIC Lab, 1995)

Concept of Interactive Environment

Interaction in Interactive Environment

Environment

Interaction Interaction

Interaction

Environment : Mirror of our mind, Our 'doppel'

Communication and Environment

Image Communication System

Camera

Microphone Array

Screen

Projector

Speaker

Computer

Metamorphosis Technology

Virtual KABUKI

Facial Expression

Detection/

Reproduction

Virtual KABUKI

Reproduction of

"Realistic" Facial Expressions

Mapping on KABUKI Actor's Face

("Metamorphosis")

Flowchart for Reconstruction

of Facial Images

Original Image

Detection of

Facial Expression

Estimation of

Facial Expression

Deformation of

3D WFMTexture Mapping

Reconstruction of Facial Image

3D WFMColor

Texture

Modeling

ReconstructionDetection

Reproduction of

Facial Expressions

Mapping on

a KABUKI Actor's Face

Mapping on

a KABUKI Actor's Face

Detection of Body Motions

Thermographical Image Detection Result

•@

Metamorphosis

into a KABUKI Actor

Manipulation in Cyberspace

Glove device Intuitive manipulation

Difficulty in fine manipulation

Force feedback PDDM

Cooperative object manipulation

Hand gesture recognition

P D D M

(Palmtop Display for Dextrous Manipulation)

Structure:Small LCD

•{

Force Display

with

Torque Controlled

Ultrasonic Motor

Control Phases:Observing Phase

Handling Phase

Manipulation by PDDM

Cooperative Object Manipulation

Multi-hand Manipulation

Multi-user Manipulation

Precise object manipulation

Extentional slave hand

Integration of operations by multiple users

Improvement of presence with cooperative manipulation

Multi-user Manipulation

Realtime Recognition of

Hand Gestures

Rotation Angle Detection Finger Bending Detection

Virtual Object Handling

by Hand Gesture Recognition

Generation of

Autonomous Agent

Automatic Generation of

Walking Amimations

Animation generation by animators

Automatic generation of animations

Multi-layer Architecture

for Agent Behaviors

Generation Tool for

Walking Animations

Automatic Generation of

Animations

Nonverbal Communications

Emotional Agents "MIC"

Computer Agents

Emotional Agents

Logical Processing

Verbal Communications

Emotions Processing

Nonverbal Communications

Emotions Recognition

Emotions

Joy, Anger, Surprise, Sadness, Disgust, Teasing, Fear, Normal

Speaker-independent, Content-independent

Emotions Recognition

Algorithm : Neural Network

Multi-speaker, Phoneme-balanced Speech Data

An Example of

MIC's Reactions

Block-diagram of

Emotional Agent

Voice Input

Emotion

to

Reaction

Conversion

Animated

Output Face

Output

Sound

Emotion

Recognition

Block-diagram for

Emotions Recognition

Speech feature calculation

Extraction of speech period

Speech feature extraction

Training of neural networkEmotion recognition

using neural network

Mapping on emotion plane

Generation of reaction

Generation of output speech

Speech input

Recognized emotion

Speech Processing

Emotion Recognition

Reaction Generation

Configuration of

Emotion Recognition Part

Decision logic

Speech feature parameters

Output

Sub-neural

network

(neutral)

Sub-neural

network

(sadness)

Sub-neural

network

(anger)

Training Data

Words: 100 phoneme-balanced words

Speakers: five males and five females

Utterances: 800 for each speaker

Training and Recognition

Recognition Experiment

Recognition rate : about 60%