Intelligent Robot ArchitectureIntelligent Robot Architecture (1-(1-3)3) Background of research

Research objectives

By recognizing and analyzing user’s utterances and actions, an intelligent robot must be able to capture the intention of the user to make correct responses In a multimodal dialog environment, a human and a robot interact by using combinations of spoken language and various gestures, even facial expressions. Therefore, a multimodal interface technique is important for developing an intelligent robot

Perception

Vision(gesture)

speech

Knowledge Bases

WorldModel

ObjectModel

UserModel

Domain Task Model(Plans/Goals)

…Model

Multimodal Instance grounding

Dialogue Model

DialogueHistory

DiscourseKnowledge

Recognition of user’s intention & Goal Planner (generation of robot’s

intention)

Task and Knowledge Manager

TaskAgenda

Deliberative Layer

Multimodal Interaction Manager

(dialogue + expression + gesture) generation

Perception

Vision(gesture)

speech

Knowledge Bases

WorldModel

ObjectModel

UserModel

Domain Task Model(Plans/Goals)

…Model

Multimodal Instance grounding

Dialogue Model

DialogueHistory

DiscourseKnowledge

Recognition of user’s intention & Goal Planner (generation of robot’s

intention)

Task and Knowledge Manager

TaskAgenda

Deliberative Layer

Multimodal Interaction Manager

(dialogue + expression + gesture) generation

Multimodal interaction manager model - Combine and analyze the speech and gesture - Recognition to capture the intention of the user - Decide the robot’s intention according to the user’s intention. i.e., decide what to do and what to say.

Intelligent user modeling - Define generic user models and develop a reinforcement learning for a friendly evolving interaction.

Research contents

Intelligent Robot ArchitectureIntelligent Robot Architecture (1-(1-3)3)

Developing a multimodal interaction model - Developing a Multimodal anaphora processing method - Defining the representation of user’s intentions and robot’s domain actions in a multimodal interaction environment - Developing a prototype system of a multimodal interaction manager

Observation

ObservableObject

Agent

TimeInstance

ObservingAgent

ObservingAgent

ObservationTime

ObservationTime

Drinking Action

Drink

TableNotebook

ObservedObject

ObservedObject

DrinkingTime

DrinkingTime

DrinkingAgent

DrinkingAgent

DrinkedDrink

DrinkedDrink

솔의눈

space At

At

Cup

사람 Robot

Vision

Hold

Hold

On

On

X,Y,ZAbsolute Cord

Absolute Cord

Volume ModelVolume

Volume

거실

Dialogue ModelDialogue Model

User’s IntentionUser’s IntentionRecognizerRecognizer

Robot’s TaskRobot’s TaskPlannerPlanner

DiscourseDiscourseHistoryHistory

DiscourseDiscourseKnowledgeKnowledge

Multimodal Interaction ManagerMultimodal Interaction Manager Real-world & Object OntologyReal-world & Object OntologySpeech & Gesture RecognitionSpeech & Gesture Recognition

MultimodalMultimodalInstanceInstance

GroundingGrounding

RecognizedRecognizedSpeech & GestureSpeech & Gesture

AnaphoraAnaphora

InstanceInstance

컵 101

Research contents

Intelligent Robot ArchitectureIntelligent Robot Architecture (1-(1-3)3)

Plan Recipe Repository

Instance of Variable(Domain Discourse)

User Intention Recognizer

Database

Transformer

Recipe ParsingInput Parsing

Inference Engine

SearchEngine

GenerationEngine

Instance of Variable(Domain Discourse)

Multi Modal Instance

grounding module

User modeling Module

Postpositional Word Selector

Sentence Templete Repository

Case Frame Repository

User Profile Repository

Recognition model of user’s goal and intention - Developing to analyze user’s intention using a plan inference technique and a dialogue manager - Developing a sentence generation engine

Inference technique based on user models - Defining a ontology to include a user profile. i.e., preference, life pattern

Jungyun Seo

Dept. of Computer Science &

Interdisciplinary Prog. of Integrated Biotechnology, Sogang Univ.

seojy@sogang.ac.kr, http://nlp.sogang.ac.kr

Research Institutes : Sogang University,

Kangwon National University, Dongseo University

Researchers : 18(Univ. 18, Industry 0)

Project Leader