gUI: Specifying Complete User gUI: Specifying Complete User InteractionInteraction

Soft computing LaboratoryYonsei University

October 25, 2004

2

ContentsContents• Computer human user interfaces – Synthesis• Value of empathy in synthesis• Computer human user interfaces – Analysis• gUI research objectives• The role of the anthropomorphic interface metaphor• gUI data specification issues• Implementation issues• Results of the prototype system• Conclusion

3

Computer Human User Interfaces – SynthesisComputer Human User Interfaces – Synthesis

• Humane CHIs (Computer Human Interfaces)– Embodied character agents (ECAs), Embodied

conversational agents, virtual conversational characters, virtual humans, talking heads

• Basic criteria for a standard markup language– Completeness, simplicity, consistency, intuitiveness,

abstraction, usability, standardization, evaluation

• VHML– An open specification available at the VHML web site– Two applications

MetaFace, an ECAThe Mentor systems, a dialogue management system

4

Value of Empathy in SynthesisValue of Empathy in Synthesis• ECA with emotions or a personality: Increase the feeling of

familiarity

• Application areas with common anthropomorphic metaphors– Universities– E-commerce– Web guides– Traveling– Interactive games– Knowledge interactive companions

5

Computer Human User Interfaces – AnalysisComputer Human User Interfaces – Analysis

• “Aware”: The user does not know how to perform the task

• Gestalt user interfaces (gUIs)– “gestalt”: An organized whole in which each part affects

every other part– The mind organizes events and situations as a pattern “big

picture”– The entities

Similarity, proximity, closure, continuity, membership

– Input: A combined “stimulus” of text, button clicks, and analyzed facial, vocal, and emotional gestures

– Recognition: Translate the stimulus into the intent

6

Value of Empathy in AnalysisValue of Empathy in Analysis• “more complete” encapsulation of human interaction

– The empathic ability of an interface

– The extra functionality acquired

– An intuitive interface

• Overlooked empathic state of the virtual character– Attend: Considering the thoughts and feelings of the user

– Engage: Aligning actions, thoughts, and feelings with those of the user

– Value: Expressing the value of the user’s interaction

– Encourage: Expression encouragement for further interaction

– Parting: Suspending dialogue while user performs other tasks

– Available: Allowing the user to interrupt for interaction

gUIs

7

gUI Research ObjectivesgUI Research Objectives• Formal specification, development, implementation,

evaluation, and standardization of a markup language is necessary to provide a stable, consistent base for both industrial use and future research into multi-modal Human-Computer interaction

• MPEG-4: Solutions to facial and body animation at the low level

• Require a unified standard architecture plus language to control/record the higher level human ECA interaction

8

The Role of the Anthropomorphic Interface The Role of the Anthropomorphic Interface Metaphor (1)Metaphor (1)

• Anthropomorphic metaphor– To specify complete user interaction with an ECA– To use the analysis and synthesis of emotion and gesture

• Desktop metaphor– The most predominant metaphor used in computer systems– The metaphor’s likeness to our perception of the world

• “illusions”– Reduce cognitive overhead when users interact with an

anthropomorphic interface– gUI applications become an extension of real life

Waste paper basket on the Macintosh desktopThe humanity of talking heads

9

The Role of the Anthropomorphic Interface The Role of the Anthropomorphic Interface Metaphor (2)Metaphor (2)

• Anthropomorphic web applications– Bryan and Gerchman

Browsing: The leisurely navigation along an arbitrary path Searching: Goal oriented finding specific information

– A knowledgable ECA (MetaFace) Knows that you are getting frustrated in web searching Offers assistance as a friend

• Transcription– Representation, transformation, and storage of analyzed emotions

and gestures

• Real-time interaction and data representation– Efficient storage and communication of the data– Intelligent television application: handle real-time emotions and

gestures to control the function of the TV

10

gUI Data Specification Issues (1)gUI Data Specification Issues (1)• “Affect recognition”

– Proper gestalt understanding of the user requirements– “Non-verbal” behavior and “Non-verbal” communications

web site (http://www3.usal.es/~nonverbal/paper.htm)

• Temporal window– Differentiate short mixed emotions such as surprise before

anger or fear– Detect more lengthy emotions such as gradual happiness as

“understanding comes to someone”

• Specification of the gUI input data format– At the lowest level: Efficient binary format– At the higher levels: a more structured format aiding in

abstraction and cognitive understanding

11

gUI Data Specification Issues (2)gUI Data Specification Issues (2)

12

gUI Data Specification Issues (3)gUI Data Specification Issues (3)• Dynamic context transitions

• Time-stamping of emotion and gestures: Synchronizing the resulting XML document and providing context

• MPEG-4 face and body animation– Efficient communication standard– Emotion and gesture representation for real-time

applications

• Some problems in VHML data representation– Lack of a consistent set of emotions, and the ability to add

more– Lack of ability to show change in different sets of emotions

or just a single emotion– Lack of additional attributes for emotion besides intensity

13

gUI Data Specification Issues (4)gUI Data Specification Issues (4)• A more realistic and extensible semantic structure

• Gesture input

14

gUI Data Specification Issues (5)gUI Data Specification Issues (5)• Issues on specifying gestures

– A huge diversity of gestures for the human body to try and identify a base set for all analysis and synthesis applications

– Own associated information on each gesture: No standard parameters for all gestures

– Culturally dependent gestures

• A less naïve format

15

Implementation Issues (1)Implementation Issues (1)• MetaFace framework

– An example ECA used to test the underlying theories– Based on MPEG-4 and VHML– The ability to vocalize and visually display emotion and

gesture– Dialogue management tool language (DMTL)

• Evaluate tag: Inference the context

16

Implementation Issues (2)Implementation Issues (2)

17

ConclusionConclusion• Challenges

– The overlap of text input, gesture, and emotion as one coherent stimulus

– Standardization of emotion and gesture representation

• Complete user interaction with gestalt user interface– Use anthropomorphic metaphors– Reduce the conceptual load– Mentor and MetaFace research project