Conversational Informatics (E), October 3, 2018

1. Introduction—Conversational Informatics—

Toyoaki NishidaKyoto University

Copyright © 2018, Toyoaki Nishida, Atsushi Nakazawa, Yoshimasa Ohmoto, Yasser Mohammad, At ,Inc. All Rights Reserved.

Why do we study conversation?

AI People

Data

Service

Common ground

Conversation as continuous update of the common ground

Conversations in different settings

Laboratories (controlled) Wild (uncontrolled)

Augmented (unrestricted)

Main Focus: Conversation

Shot by Ms. Sutasinee Thovuttikul with permission at a market in Chiang Mai, Thailand

Eye gaze

Hand gesture PostureFacial expression

AskingNegotiating

Proposing

Conviviality

Social networksTrust

Conversation is a complex business

[A marketplace in Chiang Mai; courtesy of Sutasinee Thovuttikul]

Challenge: A robot that can participate in conversation

Long-term goal

Not just conversational but also empathic

History of conversational systems development

t1990 2000 201019801970

Natural language dialogue systems

Speech dialogue systems

Multi-modal dialogue systems

Embodied Conversational Agents / Intelligent Virtual Human

Story Understanding systems

Conversational Systems

Transactional systems

Interactional systems

Affective Computing

Cognitive systems

Natural language question answering systems

The Knowledge Navigator

Our research group

SPOC (2005)

SituatedKeyword-based

EgoChat (2000)

CoMeMo Community (1998) POC TV(2001)

Physical robots

Vicky (2008)

Sustainable Knowledge Globe (2006)

Has addressed numerous aspects

[Nishida 1998][Nishida 1998]

CoMeMo Community

[Kubota 2000]

Using alterego agents

[Kubota 2000]

EgoChat

Knowledge cardKnowledge card

Knowledge cardKnowledge card

Knowledge card server

Knowledge card

presentation by ECA

Knowledge card editor

Knowledge channel control policy

landscapes

[Nishida DNIS 2003]

Knowledge Card Circulation

EgoChat -- demo

[Kubota+ 2003]

[Nishida 2002]

POC: Public Opinion Channel [Nishida 1999]

POC TV (KDDI FTTH Trial)

Capturing conversation quanta using auxiliary devices

P

A

B

button press

utterance 1 utterance 2reaction to the present

reaction to the past

proaction to the future

the timing of button press

and conversation quantum

Analysis

[Kubota and Saito 2005]

In-vehicle conversation circulation system

[Okamura+ 2007]

conversation assistance and capture

by recognizing nonverbal behaviors (e.g., pointing)

driving simulatorautomobile with video recorder capture and reuse conversation in automobile

- personal experience and social interaction -

detect pointing, display names and previous conversations

Capture

Driving experiences data base

Interactive presentation

Driving experience recording system

Scenes+GPS info

[Nishida 2008]

User behavior

Intelligent Driving Simulator

The purpose: to help people share and evolve the driving experiences

data base by allowing them to conversationally annotate driving scenes.

Vickey: In-vehicle conversation circulation system

The Sustainable Knowledge Globe (SKG)

[Kubota+ 2004]

[Kubota+ 2004]

The Sustainable Knowledge Globe (SKG)

[Nakano 2005]

SPOC

[Terada+ 2002]

Autonomous mobile chair

Conversational artifacts shows how synthetic

characters or intelligent robots use eye gaze, gestures

and other non-verbal communicators to interact.

Conversational contents looks at developing

techniques for acquiring, editing, distributing and utilising

the contents that are produced and consumed in

conversation.

Conversation environment design explains

techniques for creating intelligent virtual environments

and for representing individuals within a virtual

environment

Conversation measurement, analysis and modelling

demonstrate how conversational

behaviour can be measured and

analyzed.

Buy your copy here

or online at

www.Wiley.com

An Engineering Approach

http://as.wiley.com/WileyCDA/WileyTitle/productCd-0470026995,descCd-tableOfContents.html

Our research group addressed

SPOC (2005)

SituatedKeyword-based

EgoChat (2000)

CoMeMo Community (1998) POC TV(2001)

Gesture-based

Vicky (2008)

Sustainable Knowledge Globe (2006)

Has addressed numerous aspects

Not realistic!

But limited

Approach 1: Making environment playful

Smart conversation space that encompasses

participants and referents of conversation.

-> Engaged conversational interactions

-> More insights about the common ground

Augmentation

by MR (VR—AR)

Daily living space

[Nishida-Nakazawa-Ohmoto-Mohammad 2014]

[Nitschke 2013]https://www.youtube.com/watch?v=V-9SKpcMrzk

Conversations in an artificially created environment

ICIE: Immersive Collaborative Interaction Environment

Appearance Architecture Projection

Inside view

https://www.youtube.com/watch?v=wxkZ9armrI8

Interactive Dome

DEAL: Platform for ICIE

[Ohmoto 2011]

Learn from Playful interactions

Shared interaction environment cohabited by people and agents

[Suyama, Ohmoto 2015]

Player #1 in ICIE #1

Red hat

Player #2 in ICIE #2

Green hat

Player #3 (agent)

Blue hat

Analyze the behaviors of

participants by integrating audio-

visual and

physiological .measurement

Approach 2: Understanding by measurement and analysis

Approach 2: Understanding by measurement and analysis

Measurement in an open spaceMeasuring conversation in a fully equipped space

Conversation in a specific spatial arrangement Measurement in an augmented space

Corneal Imaging CameraScene cameraEye camera

• Lightweight and versatile system

• Appl.: Google Glass like HMD, unconstrained setups

Corneal Image Feature Matching

Problem:

Local feature correspondence + RANSAC does

not work due to large noise in eye images

Approach:

1. Formulate problem as registration of 3D spherical light maps of eye and scene image

2. Single point algorithm for robust alignment

Non-intrusive Eye Gaze Tracking (EGT) by corneal imaging

Intentions are dynamic and tacit

[Nishida-Nakazawa-Ohmoto-Mohammad 2014]

Facilitative agent that can track dynamic and tacit intentions

Degrees of concentration

Level of proficiency

Tacit social signals and mental process beneath surface

Approach 3: Learning by imitation

Endow robots with an ability

of autonomously imitating

human behaviors.

Approach 3: Learning by imitation—Generic framework

Measurement Corpus Generalization Dialogue

patterns

[Nishida et al 2014]

Endow robots with an ability

of autonomously imitating

human behaviors.

Interactions from observation—General framework

Causes

Causes

Causes

1a

2a

3a

t

t

t

[Nishida et al 2014]

Problem formulation:Find approximately repeated subsequences in a longer time series.

(1) Motif Discovery—Finding Patterns of Interaction

[Nishida et al 2014]

Future

Change angle

GH

Past Futuret

( ) ( ) ( );...; 1H t seq t n seq t= − −

( ) ( ) ( )1 ;...;G t seq t seq t n= + +

( ) ( ) ( )

( ) 1

1

1

ˆ

f

f

T

i i i

l

i i

i

l

i

i

s t t t

cs

x

c

t

=

=

= −

=

TtVtStUtH )()()()( =

Find optimal lPggT uutGtG =)()(

Find optimal lF

11and,)( +−= jjjF

g

ii liut

fT

ll

T

lli li

tUU

tUUt = ,

)(

)()(

)()()()()(ˆ)(~ tttttxtx PFPF −−=

Learning by imitation

Robust Singular Spectrum Transform

[Mohammad 2009]

[Mohammad 2009]

Learning by demonstration

[Mohammad 2016]

Application

Platform Evaluation

Content production

Model building

Analysis

Theory

Measurement

Conversational interactions

Conversational Informatics

[Nishida-Nakazawa-Ohmoto-Mohammad 2014]

Building conversational

systems

Understanding conversation

Limitations

Our conversational agents do not produce

a strong sense of presence.

We lack the common ground!

Common ground

Physical configuration

Imaginary scene shared by participants

Communal background including the cultural and biological aspects

Common ground

Bargaining scenario (successful)

C: Oh, OK. How much is this one?

S: This one? It worth nothing, you can have it [for free].

C: Haha, well, please...

S: The price is 500 tomans.

C: Oh! 500 tomans? It is very expensive? Do you have anything cheaper?

S: This is from a very good brand, we have the no-name ones, you can see

up there, and they are cheaper.

C: But the colors of those do not match my table. I would love to buy this if

you give me a very good discount. You know, I am about to marry,

marriage itself costs a lot, so it would be nice if you would do that.

S: It is your marriage, you should treat us! … Because it is a happy event I

can give it for 450.

C: 450 is still a little bit expensive. Why not we make a deal for 400?

S: 400 is too low, but since it is my first sell of the day, and you are getting

married, and everything looks very great I can give it for 430, but you

know, I haven’t sold it less than 450 so far.

C: You have been so nice so far, why not getting rid of that 30 and make it

round for 400. I am sure you are making lots of purchases today with

other customers and you can compensate for that. And I’m going to be a

frequent customer from now on, I will come to your shop again.

S: We would love to have you again, but the price is somehow at its

margin. But, I can guarantee it for one year for you, and I can give it to

you for 420, final! This is very good brand, you would never regret it, you

would love it, you can have my word that it works for you perfectly and

for one year if anything happens to that you can bring it to us and we can

fix it for you for free. Ok, I’ll write 420 for you.

C: Ok, thank you, you did me a good favor. Thanks a lot.

S: Oh please. We enjoy you being here and please come back soon.

C: Yeah I will. I’m actually going to do more purchases from now on from

your shop. I had a good experience.

S: Sure! Have a good time.

C: Hi. How are you?

S: Hi. Welcome to this shop.

C: Thank you. How much are these handicrafts here?

S: Oh, these are called Khaatam .The big ones are 120, [middle

sized ones for] 80 tomans and [small ones are] 50 tomans.

C: Um, this is Termeh, right?

S: Yes!

C: What about the Termeh?

S: Actually we have different types of Termeh. We have the red

ones there starts from 50 and the brown ones start from 150

and these special ones starts from 300.

C: You know what? I am about to marry, and I am looking for

something very nice to decorate my table. Its color is brown.

So, what do you suggest?

S: Oh! Congratulations for your marriage!

C: Thank you.

S: We usually go for the red ones for brown tables, but because

you are a great customer and you have good taste, probably

you would go for this one, this is very beautiful and it is from a

very famous designer.

Ostensible offer

[Mirzaei+ 2017]

How much

is this one?

I expect him to

understand that

I want to know

the price.

Immediate quote

is not good.

Successful Scenario: Extract Analysis

I am asking him to

give me the

information on the

price so that I can

decide whether to

buy it or not.

She is asking

how much I

expect her to

pay for this item.

She does not expect

me to quote the price

immediately. It is

against convention.

[Mirzaei+ 2017]

Have it

for free

Look at his body

language.

He shows his

sincerity.

Successful Scenario: Extract Analysis

Two arms extended forward with palms up

He is being kind

and polite but his

offer is not real.

He expects me to

decline the offer to

balance the equity.

She will be happy

by my ostensible

offer and she will

ostensibly decline.

I show her my

sincerity. I’m making

a non-real offer.

[Mirzaei+ 2017]

Haha

Successful Scenario: Extract Analysis

She enjoyed it

and she knows

the offer is not

real.

I expect him to

be relieved by

receiving my

declination.

I understand you are

being very nice and I

acknowledge your

politeness.

She acknowledged

my offer. She is nice

and the atmosphere

of the conversation

is good.

She expects me to

continue with the

convention (i.e.,

going with 2nd offer)

[Mirzaei+ 2017]

Synthetic evidential study

Synthetic evidential study (SES) combines dramatic role play and group discussion to help people spin stories by bringing together partial thoughts and evidence.

Componentize

Reuse

SES session Interpretation archive

Structured collection of {story, background, critique}Agent Play

Dramatic role play

Group discussions

[Nishida et al 2015]

At the beginning of the 18th century, a feudal lord named Asano Takumi-no-kami

Naganori was in charge of a reception for envoys from the Imperial Court in Kyoto.

Another feudal lord, Kira Kozuke-no-suke Yoshinaka, was appointed to instruct

Asano in the ceremonies. On the day of the reception, while Kira was talking with

Yoriteru Kajikawa, a lesser official, at “Matsu no Roka” (“Hallway of Pine Trees”) in

Edo Castle, Asano came up to them screaming “This is for revenge!!” and slashed

Kira twice with a short sword. Soon after the incident, Kajikawa restrained Asano,

who was then imprisoned. The reason for the attack was not known, though it was

widely believed that Kira had somehow humiliated Asano. Ultimately Asano was

sentenced to commit seppuku, a ritual suicide, but Kira went without punishment.

Hallway of Pine Trees (from Chushingura)

Kira Kozuke-no-suke Yoshinaka

Asano Takumi-no-kami Naganori

Yoriteru Kajikawa

Why was it possible?

How did it happen?

What did each think?

Dramatic Role Play

Group play capture

Agent play

Discussion phase

T. Ookaki, M. Abe, M. Yoshino, Y. Ohmoto and T. Nishida. Synthetic Evidential Study for Deepening

Inside Their Heart. IEA/AIE 2015.

Asano

Kira

Kajikawa

Third person view First person view

Discussions

1. Common ground for society and AI

2. Conversation as a contiguous update of common ground.

3. Conversation envisioning as visualization of conversation and common ground

4. Conversation envisioner as a platform for conversation envisioning

Conversation as

contiguous update

of common ground

Analysis

Training

Assistance

Conversation envisioner

Vision — Conversation Envisioner

TSEiA: The Story Envisioning Agent

[Qiang Zhang. TSEiA: The Story Envisioning Agent, HCII 2018, Student Design Competition (to be presented)]

TSEiA: The Story Envisioning Agent

[Qiang Zhang. TSEiA: The Story Envisioning Agent, HCII 2018, Student Design Competition (to be presented)]

FACSvatar: An Asynchronous Open Source Modular Framework From Face to FACS Based Avatar Animation

Flow of data in framework

Facial expression data can be used with any character[van der Struijk+ 2018] Stef van der Struijk, Hung-Hsuan Huang, Maryam Sadat Mirzaei, Toyoaki Nishida. FACSvatar: An Open Source Modular Framework for Real-Time FACS based Facial Animation, IVA 2018 (accepted)

Role-play Envision

L2 Learners New L2 Learners Teacher

AI Actor AI Assistance

Conversation Envisioning for CALL Purposes

CALL: Computer-Assisted Language Learning

[Mirzaei+ EutoCALL 2018]

Agenda

CreditsWill be awarded based on a report on subjects given at the class. Due date (January 31st, 2019)

Agenda (planned)

1. October 3: Introduction by Nishida2. October 10: History of Conversational Systems by Nishida 3. October 17: Methodologies for Conversational System Development (1) by Nishida4. October 24: Methodologies for Conversational System Development (2) by Nishida5. October 31: Methodologies for Conversational System Development (3) by Nishida

(November 7: no class)6. November 14: Cognitive Interaction Design by Ohmoto7. November 21: Smart Conversation Space by Ohmoto

(November 28: no class)8. December 5: Measurement, Analysis and Modeling by Ohmoto9. December 12: Affective Computing, Theory of Mind by Nishida10. December 19: Learning by Imitation by Nishida 11. December 26: Aspects of Conversation – 1 by Nishida12. January 9: Aspects of Conversation – 2 by Nishida13. January 16: Function of Nonverbal Behaviors in Conversation by Nishida14. January 23: Synergy by Nishida

Course materials available from: http://www.ii.ist.i.kyoto-u.ac.jp/?page_id=5881