Conversational Informatics (E), October 4, 2017

1. Introduction—Conversational Informatics—

Toyoaki NishidaKyoto University

Copyright © 2017, Toyoaki Nishida, Atsushi Nakazawa, Yoshimasa Ohmoto, Yasser Mohammad, At ,Inc. All Rights Reserved.

Why do we study conversation?

AI People

Data

Service

Common ground

Conversation as continuous update of the common ground

Future life supported by digital companion

Role of digital companion

• Protect boss• Help boss• Monitor boss’s health• Entertain boss• Negotiate on behalf of boss• Make a diary for boss• Learn from boss

Social lifeDigitalCompanion

Technology‐enhanced social platform 

DigitalCompanion

DigitalCompanion

Augmentation

Understanding

Conversation

Conversation in the wild

In the wild

In the lab setting

History of conversational systems development

[Nishida, T., Nakazawa, A., Ohmoto, Y., Mohammad, Y. Conversational Informatics–A Data‐Intensive Approach with Emphasis on Nonverbal Communication, Springer 2014]

t1990 2000 201019801970

USER: “Pick up a big red block”SHRDLU: “OK”

simulated robot arm

SHRDLU: Natural language understanding (1971)

ALIVE: Life-like agents (1995)

PEEDY: Embodied speech dialogue system (1997)

ELIZA:: natural language dialogue(1966)

H: Men are all alike.E: IN WHAT WAYH: They're always bugging us about something or other.E: CAN YOU THINK OF A SPECIFIC EXAMPLEH: Well, my boyfriend made me come here.E: YOUR BOYFRIEND MADE YOU COME HERE

Façade: Interactive drama (2004)Text‐only

Multimedia

Synthetic character

Q: Why did the hostess give John a menu?A: So John could ask the waiter for the meal.Q: Why did John leave the waiter a large tip?A: Probably John remembered the waiter

served the lobster to him quickly.

SAM: Story understanding system (1975)

PUT-THAT-THERE: Multimodal dialogue system (1980)

REA: multimodal dialogue system (1999)

The Knowledge Navigator: AI assistant (1987)

Story

Multimodal

History of conversational systems development

t1990 2000 201019801970

Natural  language dialogue systems

Speech dialogue systems

Multi‐modal dialogue systems

Embodied Conversational Agents / Intelligent Virtual Human

Story Understanding systems

Conversational Systems

Transactional systems

Interactional systems

Affective Computing

Cognitive systems

Natural  language question answering systems

The Knowledge Navigator

Challenge: A robot that can participate in conversation

Long‐term goal

Not just conversational but also empathic

Eye gaze

Hand gesture PostureFacial expression

AskingNegotiating

Proposing

ConvivialitySocial networksTrust

Conversation is a complex business

[A marketplace in Chiang Mai; courtesy of Sutasinee Thovuttikul]

Application

Platform Evaluation

Content production Model building

Analysis

Theory

Measurement

Conversational interactions

Conversational Informatics

[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]

Building conversational

systems 

Understanding conversation

Common ground

Physical configuration

Imaginary scene shared by participants

Communal background including the cultural and biological aspects

Common ground

Synthetic evidential study

Synthetic evidential study (SES) combines dramatic role play and group discussion to help people spin stories by bringing together partial thoughts and evidence.

Componentize

Reuse

SES session Interpretation archive

Structured collection of {story, background, critique}Agent Play

Dramatic role play

Group discussions

[Nishida et al 2015]

At the beginning of the 18th century, a feudal lord named Asano Takumi-no-kami Naganori was in charge of a reception for envoys from the Imperial Court in Kyoto. Another feudal lord, Kira Kozuke-no-suke Yoshinaka, was appointed to instruct Asano in the ceremonies. On the day of the reception, while Kira was talking with Yoriteru Kajikawa, a lesser official, at “Matsu no Roka” (“Hallway of Pine Trees”) in Edo Castle, Asano came up to them screaming “This is for revenge!!” and slashed Kira twice with a short sword. Soon after the incident, Kajikawa restrained Asano, who was then imprisoned. The reason for the attack was not known, though it was widely believed that Kira had somehow humiliated Asano. Ultimately Asano was sentenced to commit seppuku, a ritual suicide, but Kira went without punishment.

Hallway of Pine Trees (from Chushingura)

Kira Kozuke-no-suke Yoshinaka

Asano Takumi-no-kami NaganoriYoriteru Kajikawa

Why was it possible?

How did it happen?

What did each think?

Dramatic Role Play

Group play capture

Agent play

Discussion phase

T. Ookaki, M. Abe, M. Yoshino, Y. Ohmoto and T. Nishida. Synthetic Evidential Study for Deepening Inside Their Heart. IEA/AIE 2015.

Asano

Kira

Kajikawa

Third person view First person view

Discussions

Conversational Informatics Group

Synthetic Evidential Study:An integrated framework for common ground building in conversational informatics

Dramatic Role Play Agent play

Immersive Collaborative Interaction Environment Interactive DomeSmart conversation space

First person perception

Group motion capture

Third person view First person view for A

First person view for B

First person view for CGroup discussions

Estimating mental status based on cognitive model

Common Ground

Artificial Intelligence People

Presuppositions for conversation that each participant is supposed to share about surroundings, activities, perceptions, emotions, plans, interests, etc.

Communicative Intelligence for Bridging People and AI

Approach 1: Smart environment for conversation

Smart conversation space that encompasses participants and referents of conversation.‐> Engaged conversational interactions‐> More insights about the common ground

Augmentationby MR (VR—AR)

Daily living space

[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]

ICIE – immersive collaborative interaction environment

Immersive interaction with ICIE

https://www.youtube.com/watch?v=V‐9SKpcMrzk

[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]

Telepresence by connecting ICIE with a networked robot

Feedback generationMotion mappingUser motion sensingHead recognitionGesture recognitionFace modelHuman body model

WOZ operating environment

WOZ operatorTele-operated robot

The conversation place

[Nishida-Nakazawa-Ohmoto-Mohammad 2014]

Inducing intentional stance toward agent players

Q: How can we induce an intentional stance toward NPCs?H: Demonstrating strategy change.

[Suyama‐Ohmoto‐Nishida 2015]

Player #1 in ICIE #1

Red hat

Player #2 in ICIE #2

Green hat

Player #3 (agent)

Blue hat

Interactive Dome

Appearance Architecture Projection

Inside viewhttps://www.youtube.com/watch?v=wxkZ9armrI8

Analyze the behaviors of participants by integrating audio-visual and physiological .measurement

Approach 2: Understanding by measurement

IMADE: Interaction Measurement, Analysis, and Design Environment

3D conversation capture—over the shoulder view

https://www.youtube.com/watch?v=J08vG8wnrnw[Yano 2012]

Corneal Imaging CameraScene cameraEye camera

• Lightweight and versatile system • Appl.: Google Glass like HMD, unconstrained setups

Corneal Image Feature MatchingProblem:Local feature correspondence + RANSAC does not work due to large noise in eye images

Approach:1. Formulate problem as registration of 3D 

spherical light maps of eye and scene image2. Single point algorithm for robust alignment

Non‐intrusive Eye Gaze Tracking (EGT) by corneal imaging

Eye images with GRP

Scene images with PoG

↑ Aligned results (from eye images)

Peripheral vision map overlaid to scene imagePeripheral vision map in eye image

Gaze Reflection Point Mapping

Application 1: Non‐intrusive and uncalibrated PoG estimation

Application 2: Peripheral vision estimation

Gaze trajectory in static scene image

Inducing intentional stance toward agent players

[Takeda‐Matsuda‐Ohmoto‐Nishida 2015]

Deliberating but not reacting

Deliberating and reacting

Doing nothing Reacting but not deliberating

SCR

LF/HF

+

+

‐

‐

Hide from the chaser

Not concentrated

Hide in the place the chaser checked previously.

Simply moving around

Multi-dimensional model of estimating internal state of human

Concentration

Level of proficiency

Approach 3: Learning by imitation—Generic framework

Measurement Corpus Generalization Dialoguepatterns

[Nishida et al 2014]

Endow robots with an ability of autonomously imitating human behaviors.

Interactions from observation—General framework

Causes

Causes

Causes

1a

2a

3a

t

t

t[Nishida et al 2014]

Problem formulation:Find approximately repeated subsequences in a longer time series.

(1) Motif Discovery—Finding Patterns of Interaction

[Nishida et al 2014]

Future

Change angle

GH

Past Futuret

;...; 1H t seq t n seq t 1 ;...;G t seq t seq t n

1

1

1

ˆ

f

f

Ti i i

l

i ii

l

ii

s t t t

csx

c

t

TtVtStUtH )()()()( Find optimal lP

ggT uutGtG )()(Find optimal lF

11and,)( jjjFg

ii liut

fTll

Tll

i litUUtUUt ,)()()(

)()()()()(ˆ)(~ tttttxtx PFPF

Learning by imitation

Robust Singular Spectrum Transform

[Mohammad 2009]

[Mohammad 2009]

Learning by demonstration

[Mohammad 2016]

Human‐AI Communication

Goal: Communication envisioning and its application

Communication envisioning environmentHelps the user unveil and express tacit information on‐the‐fly.

Facilitate interpretation

Educational purposes

Real‐time assistance

Research issues:

AI‐enhanced analysis Automated estimation of mental process Implementation and evaluation of the platform

• Building conversation scenes• Automated generation of 

communicative behaviors• Emotion and intention• Presence of agents• Evaluation methodology

Conversation as a continuous update to the common ground

• Modeling the common ground• Contrasting real and virtual

Agenda

CreditsWill be awarded based on a report on subjects given at the class.  Due date (January 31st, 2018)

Agenda (planned)

1. October 4: Introduction   by Nishida2. October 11: Cognitive Interaction Design  by Ohmoto3. October 18: Methodologies for Conversational System Development (1) by Nishida4. October 25: Methodologies for Conversational System Development (2) by Nishida5. November 1: Affective Computing by Nishida6. November 8: Theory of Mind by Nishida7. November 15: Smart Conversation Space by Ohmoto8. November 22: Measurement, Analysis and Modeling by Ohmoto 9. November 29: Learning by Imitation ‐ 1 by Nishida10. December 6: Learning by Imitation ‐ 2 by Nishida11. December 13: Aspects of Conversation ‐ 1 by Nishida12. December 20: Aspects of Conversation ‐ 2 by Nishida13. December 27: Aspects of Conversation ‐ 3 by Nishida14. January 10: Synergy by Nishida

Course materials available from: http://www.ii.ist.i.kyoto‐u.ac.jp/?page_id=5881