Deliverable 6 - Gaips at Inesc-Idgaips.inesc-id.pt/safira/publications/D-SAFIRA-WP6-D6.1.pdf · The SAFIRA application uses profile ownership, Quality of Service adaptation to individualised

Document Id: D-SAFIRA-WP6-D6.1

Commission Ref.: D6.1

Classification: project&commission

Distribution: project&commission

Date: March, 31st, 2002

IST Project IST-1999-11683

SAFIRA Supporting Affective Interactions for Real-time Applications

Deliverable 6.1

First Specification and Prototype of the Demonstrators

Yasmine Arafa, ICSTM

Marco Vala, INESC-ID

Carlos Martinho, INESC-ID

Rui Prada , INESC-ID

André Vala, INESC-ID

Petar Goulev, ICSTM

Ana Paiva, INESC-ID

Moisés Piedade, INESC-ID

Phoebe Sengers, GMD

Ricardo Chaves, INESC-ID

Marco Costa, INESC-ID

Daniel Sobral, INESC-ID

Abstract

This document constitutes the first deliverable of workpackage 6 of the IST Project SAFIRA. This deliverable will present the specification of the three demonstrators: James the Butler; FantasyA and the Influencing Machine, and a description on how some of the components from the toolkit are being used within these demonstrators.

Keywords

Affective Interactions in Personal Assistants, E-commerce, Syntetic Characters and 3D games.

Project Manager:

Ana Paiva

INESC-ID Rua Alves Redol 9, 6º Esq. P-1000-029 Lisboa Portugal

Tel.: +351 21 3100219 Fax: +351 21 3145843 Email: [email protected]

Project Homepage:

http://gaips.inesc.pt/safira

Project Email:

[email protected]

Deliverable D6.1

Specification and First prototype of the Demonstrators

Deliverable Number:

Version:

Contractual Date of Delivery to CEC:

Actual Date of Delivery to CEC:

Editor:

Deliverable Type:

Deliverable Nature:

Workpackages Contributing:

Partners Contributing:

D-SAFIRA-WP6-D6.1

1.0

March 31st, 2002

WP6

P-Public

R-Report

WP6

INESC-ID, GMD, ICSTM

This document is a Deliverable of the SAFIRA project (“Supporting Affective Interactions for Real-time Applications”), a project performed within the IST programme (project identifier IST-1999-11683).

The SAFIRA project is performed by a consortium consisting of the following partner organisations:

INESC-ID – Instituto de Engenharia de Sistemas e Computadores (P, Coordinating Partner)

ADETTI – Associação para o Desenvolvimento das Telecomunicações e Técnicas de Informática (P)

DFKI – Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (D)

FhG– Fraunhofer-Gesellschaft (Previously GMD) (D)

ICSTM – Imperial College of Science, Technology and Medicine (UK)

ÖFAI – Austrian Research Institute for Artificial Intelligence (A)

SICS – Swedish Institute of Computer Science (SE)

IST-1999-11683:D6.1 31 March 2002

D-SAFIRA-WP6-D6.1 © IST Project IST-1999-11683 SAFIRA 5

Executive Summary

This Deliverable (D-SAFIRA-WP6-D6.2) is the first deliverable from WP6 (Demonstrators) of the IST-sponsored SAFIRA project. It contains a detailed description of the three demonstrators of the Safira Project. These demonstrators do not intend to provide final products as such which can be commercialised and distributed. Rather, the aim is to allow for the demonstration of the use of the toolkit and its components to create affective interactions. To do that we have chosen three quite innovative application areas that are: e-commerce with James the Butler; entertainment-computer games with FantasyA; and electronic art with the Influencing Machine. We believe that this set of demonstrators is a good example of how our toolkit and our research can be used in a future for concrete and even commercial applications.

The objectives of WP6, as per the Technical Annex, p 39, were the following:

(1) to prove experimentally the validity of the conducted research, and

(2) to illustrate the use of the framework and toolkit in innovative and practical examples, where emotion and personality based autonomous agents play a fundamental role.

We have planned in this workpackage the development of three demonstrators: James the Butler; FantasyA and the Influencing Machine. For each one a study of the theories and types of emotions that may occur in these scenarios was done and three initial prototypes were constructed. Such prototypes use some of the components developed in WP3, WP4 and WP5.

Finally, note that the work on the Influencing Machine was significantly cut (from 6MM to 1M) after the first annual review of the project, and therefore the results here shown reflect mainly the aggregation of the components developed in the other workpackages.

On the whole, this work contributes substantially to the overall SAFIRA project objectives, in providing innovative components which not only embed aspects of affective computing and affective interactions, but also extends this area into the challenges of this decade with the advent of personalisation and adaptation to the users.

The research and results here described are primarily based on collaborative work between the partners of this workpackage as well as all consortium members.

The research and development has been completed in a uniformly high quality. It has generated significant and substantial research directions.

The following chapters are organised to reflect the work done in WP6 and thus are divided as follows:

IST-1999-11683:D6.1 31 March 2002


Chapter 1 acts as a general introduction explaining the three demonstrators and their role in the Safira project.

Chapter 2 provides a description of James the Butler .

Chapter 3 provides a description of FantasyA.

Chapter 4 provides a description of the Influencing Machine.

Chapter 5 provides a plan of work for the next 6 Months.

IST-1999-11683:D6.1 31 March 2002


1. The Demonstrators of Safira

The work package 6 deals with the design and development of simple applications that utilise affective input, reasoning and expression to obtain emotion-based interactions.

In general, this work package has a dual goal: (1) to prove experimentally the validity of the conducted research. (2) to illustrate the use of the framework and toolkit in innovative and practical examples, where emotion and personality based autonomous agents play a fundamental role.

In this deliverable we will describe the initial development of the 3 demonstrators: a personal assistant for e-commerce applications (James the Butler); an augmented realities 3D Fantasy world (FantasyA) and the Influencing Machine.

For each one of the demonstrators, there was an initial study of the theories and types of emotions that occur in the studies scenarios as well as the components used for the agent’s architectures. The aim is to cover and analyse the impact of affective interactions in a wide range of target users, covering adults and children.

Given that this work package consists of three tasks that are independent from each other, this deliverable will provide the descriptions of each of these tasks, focusing on the design and initial development of each one of the demonstrators.

IST-1999-11683:D6.1 31 March 2002


IST-1999-11683:D6.1 31 March 2002


2. James the Butler: Personal Sales Agent in an E-Commerce Domain James, the Butler operates in the context of a full screen interface representing an on-line Winestore. James is a Personal Sales Assistant (PSAs) that employs intelligence and adaptive reasoning methods to provide active, collaborative assistance to a customer of an online store. The product domain is wines along with an associated knowledge-base of wine tasting expertise, grape knowledge, quality of wines vineyards and regions, and the various product recommendation skills of an wine merchant

The objective of this demonstrator is to provide and evaluate the use of an autonomous PSA agent with the skills necessary to exhibit lifelike qualities manifested in textual, verbal and visual behaviour and action. The PSA may also take on different functional roles like a Sales Assistant agent, Help agent, Recommendation agent, among others (but is limited to the role of a Personal Sales Assistant.

Product images are presented using a variant of the “Rapid Serial Visual Presentation” method (RSVP) developed by Spence and others [deBruijn & Spence, 2000] and [Spence, 1999]. We utilise the RSVP presentation technique to allow a large range of products to be displayed in a small screen area. It appears to be particularly appropriate when a retailer wishes to present a substantial stock range for immediate viewing, as in a shop environment. RSVP represents an effective, and distinctive, alternative to the more conventional and common-place “catalogue pages” approach to online site design.

A main interaction that concerns the this demonstrator is the business-to-consumer (b2c) interaction involved in retail, in which loyalty is created by human-human interaction and the personal touch. As such, a question this demonstrator addresses is how to reproduce this kind of service online? Everyone remembers personal service or a know-legible shop assistant, and such intelligence and individuality (one-to-one treatment) implies that the multimedia interface could benefit from some kind of similar personality and sensitivity. This is realised by the implementation of agents with a visual animated lifelike character and a distinct personality. With the intention for the customer to be in control: that is the agent is highly intelligent, highly competent character that is essentially a servant. The character adds the personal touch to the interface, but moreover increases in competence over time, thus increasing its intrinsic value to a single customer (i.e. its owner). The concept is reinforced by an Interface Agent with a visual representation of a butler, ‘James’.

James is an agent that amplifies or modifies the motivational state of an agent and its perceived bodily state. It has the ability to perceive and produce the visual (animated expressions), verbal and non-verbal signals and regulate the flow of information between service agents, the interface agent and the user. These capabilities enable James to engage in complex interactions with customers via natural social communication rather than complex command languages, or direct manipulations.

IST-1999-11683:D6.1 31 March 2002


Thomas and Johnson in the Illusion of Life [Thomas & Johnson, ‘81] discuss how an animated character’s personality consists of characteristic attitudes and actions that users learn to associate with that character, as revealed during their interaction with the system. Based on this, James’s development of it’s characters’ personality is formulated by establishing attitudes and behaviours users will come to expect from the character [Field, ‘94] and which relate to the domain being presented. Hence the reason behind choosing James the butler with a personality that is commonly associated with butlers.

Personality is just one form of creating, likeable characters. Personality and Affective states are represented externally through manifestations and changes in animated character movement and gestures, intonation of conversations whether verbal or textual; and internally in their reactive, proactive and adaptive behaviour.

2.1 Design

The way this demonstrator is used and presented has been determined by examining existing techniques in Customer-Relationship Management (CRM). By understanding the value of information, interaction, personalisation, privacy, (mutual) respect and re-use in creating customer loyalty in conventional retailing [Vargas, M], we can see how this value can be transferred to electronic commerce. The base of the work presented here is a result of the ESPRIT MAPPA project (EP28831) [Arafa, et al.,1999].

The MAPPA project analysed properties of information, interaction, personalisation, privacy, respect and re-use, considering what the issues involved in realising the value of each property. The realisation of the properties of Information, persistence and personalisation has been addressed in the MAPPA project, upon which the architecture and domain data are based. This deliverable is concerned with the value of interaction.

The interaction that concerns us here if the business-to-consumer (b2c) interaction involved in retail, namely Loyalty. We refer to loyalty, here, as being the perception of loyalty by which customers are encouraged to return to an electronic-based retail system and/or the store. Potentially, continuous customer return may lead to increased sales. The SAFIRA application uses profile ownership, Quality of Service adaptation to individualised criteria and requirements, visual animated Personalised Sales Assistants, adaptive interaction and presentation, loyalty points and bonus schemes to engender the proposed loyalty. Here, loyalty is created by human-human interaction and the ‘personal touch’. The question then is how to reproduce that kind of service online in e-commerce? Everyone remembers personal service or a knowledgeable shop assistant, and such intelligence and individuality (one-to-one treatment) implies that the multimedia interface could benefit from some kind of similar personality and sensitivity.

However, this benefit will only be realised if it is predicated on good HCI (Human-Computer Interaction). This implies:

IST-1999-11683:D6.1 31 March 2002


- avoid adaptive interfaces, which are unstable and increase uncertainty in the user (but configurable interfaces with optional features are required to satisfy a wide range of users)

- avoid paperclip technology, which is intrusive and takes initiative away from the user (but just as WIMP interfaces were an improvement over command-line interfaces because of the ease of recognition over recall, we want to intelligent interfaces which can remind users of functionality)

- avoid excessive anthropomorphisation, which can raise undue expectations in the user (but human elements are found to be reassuring when dealing with un-user-friendly new digital technology).

The value of interaction can be realised if it remembered that the interaction should create value, not exchange value. In particular, a loyalty scheme is not a discount scheme. Too often, existing “loyalty” card based schemes result only in an exchange of value: some meagre “reward” points in exchange for purchase information which is used in data mining. This exchange does not engender loyalty: most people are prepared to exchange information about their shopping habits and/or preferences in return for a discount but shop elsewhere if the opportunity arises.

To create value in electronic commerce, it is necessary, we argue, to give the customer a stake in the system, and then to see that stake grow or become better in some way. In particular, we want to exploit people’s attachments to objects, as identified by Reeves and Nass [Reeves & Nass, 1996], and also the increased trend towards the software ownership model, whereby users no longer own their hardware and take out a licence to run someone else's software, but instead own their software and have a licence to run it one someone else's software.

The value of interaction is realised in SAFIRA by the implementation of agents with personality, which is stored on the client’s computer. This intention is for the user to be in control: that is the agent is highly intelligent, highly competent character that is essentially a slave. The character adds the personal touch to the interface, but more importantly (as will be seen in subsequent sections) increases in competence over time, thus increasing its intrinsic value to a single user (i.e. its owner). This also exploits the user’s attachment to objects: s/he owns the smart card and its contents (witness to the strength of this attachment - to a gadget and an electronic entity that is ‘grown’ - can be seen by the crazes for Pokemon [Pokemon]).

Persistence of state and function is another design consideration for the Winestore demonstrator. Persistence of context, to provide an extension to personalised information that can maintain the customer’s current interests and supports off- line information and service gathering. This persistence means that a user has an agent identified and working for them, and continuing to work for them whether or not they are actually online. Existing and new interface modalities can ensure that the user may make contact with, or be contacted by, their agent by voice, email, or other means.

IST-1999-11683:D6.1 31 March 2002


2.2 James the Butler

The role of James the Butler is to act as mediators between the human and the application (a wine store) and to be capable of personalising the interface by monitoring and sensing user actions, behaviour, product interests, and preferences. James is in effect an Interface Agent (IA), the functionality is realised on two levels: the service level and the interface level. The IA is, hence, considered a service agent that must communicate and negotiate with other agents in a multi-agent system to determine which and how services are to be provided. As all software agents are distinguishably characterised by the services they provide, the IA is principally characterised as a user-oriented agent. It is expected to facilitate and provide mechanisms that help enhance an application’s efficiency and usability from both interface and functionality perspectives:

- simplifying and clarifying the user interface, the information presented and the services available,

- providing means for explicit, more efficient user interaction

- providing assistance on understanding the application as whole in terms of:

• how it functions

• the services provided

• personalized focus of services to individual users

- providing support for user tasks, plans, and intentions

- presenting information more effectively by personalizing and tailoring the interface

- interacting and communicating with the user to provide personalized services.

James exhibits aspects of the characteristics described hereafter, so as to assimilate, accumulate and maintain the necessary data within a user profile to provide personalised and context sensitive assistance and recommendation in a clear simplified manner:

Accessible

James is constantly on standby ready to offer assistance when appropriate whether or not it is asked for, and will draw the attention of the user to its existence. At the presentation level James presents itself, its intentions and achievements to the user. The nature or manner of this visualization need not necessarily be manifested in human-like or animated figures the main concern is to draw the user’s attention to the services and achievements of the interface agent. Animated figures and personified manifestations can be considered as an added feature to the interface agent but are not a principle or arguably not an essential characteristic.

IST-1999-11683:D6.1 31 March 2002


The user has control over when to resort to assistance and when not. The user may consult the assistant for help or may delegate routine tasks when required. This approach to agent visibility provides a framework for passing control between the user and the interface agent allowing the user to become more confidant with the interface agent’s abilities and can then build trust in the services, help and information being provided.

Adaptive

The interface agent provides it’s proactive assis tant to a specific user by observing and monitoring his/her actions, interests and frequent interactions within a particular application and according to its role. Incrementally, the interface agent learns about user preferences and adapts the interface accordingly. It also uses its observations and learning to become better acquainted and familiar with frequent requests and tasks gradually gaining more competence so as to suggest better or alternative ways to perform these tasks, perform such task autonomously and only update the user of status, results and/or problems.

Autonomous

Autonomy is a key characteristic of James although it is one that the user is usually unaware of; nonetheless, it is one that contributes substantially to the James‘s proactive behavior. It facilitates for providing a higher level of assistance by operating independently and/or concurrently with the user [Wooldridge & Jennings, ‘95], even though this may not always be of value to the user. By constantly operating in the background the interface agent (James) is able to sense and respond to activities in the immediate environment, carry out routine, complex or time consuming tasks on a users behalf only prompting the user when there are results, complications, or when queries or clarifications are required.

Proactive

James is integrated with application services with minimum overload on a user’s focus of attention. It needs to be constantly active in the background observing and learning from the environment, user actions, and will naturally act on demand to perform a particular task as instructed, react as a result of problems or irregularities in the system informing the user when appropriate, and be proactive.

Persistence

Persistence of context, to provide an extension to personalised information that maintains the user’s current interests. Also, to support off- line information and service gathering and to develop the functionality to continue to work “off- line”, acting proactively or attempting to complete a transaction. This persistence means that each user has an agent identified and working for them, which also continues to work for them whether or not they are actually on-line. Existing and new interface modalities can

IST-1999-11683:D6.1 31 March 2002


ensure that the user may make contact with, or be contacted by, their agent by voice, email, or other means.

Persistence is necessary for many of the agent’s most important traits:

- Reliability: when a user requests a service the agents need to persist in doing that until it is complete whether successfully or unsuccessfully.

- Personalisation: agents can adapt to customer needs and preferences by updating the user profile with a service base and affect state history log.

- Added social traits: agents can remember who the users are and their characteristics, thus presenting a consistent persona.

- Learning: agents can modify their behaviour according to experience.

- Mobility: the same agent (in terms of knowledge, behaviour and appearance) should appear regardless of location or host computer.

In general, the agents in this demonstrator persist by keeping record of their patterns of behaviour and interaction, affective state in context with various situation, personality traits and customer preferences. James maintains its persona, customer awareness and behaviour following system failures or modifications.

The general benefit of applying the Personal Assistant metaphor lies in the above characteristics, which collectively may enable the James to enhance an applications efficiency and effectiveness, reduce the over all work required in using the application, and provide a more extensible design for the interface developer.

2.2.1 Demonstrator’s Main Functionality

The user interface represents the sole point of contact between Agent and Customer (User), and a variety of techniques, delineated in this section, are employed to extract valid personalisation information. Some considerable care must be taken if the criteria previously discussed are to be met. Our approach is to combine active, elective and passive information gathering methods.

2.2.1.1 Gathering Customer Preferences

Active methods include electronic form filling. We request as little personal data as possible in this way, as this can be seen as intrusive by some users (decreasing the sense of privacy) and can delay access to the main business oriented activities. Some active information gathering is inevitable, such as name and delivery addresses. In the case of both wine and video sales the customers age band must be ascertained in order to meet legal obligations, fortunately, over 18 years old the actual value is irrelevant. We also request a (self-assessment) of computer experience to tailor the amount of introductory help presented on first use.

IST-1999-11683:D6.1 31 March 2002


In general, we prefer not to present the customer with an extensive “preferences” form, but rather to gather that information incrementally by elective means. Each product selection dialog presents two extra buttons, Like and Dislike. Pressing either of these buttons informs James that the customer prefers (or not) the current selection indicated. This information then forms part of the persistent knowledge James holds about the customer.

Finally, we may infer preferences from the actions of the customer and the various choices they make. This is at the same time more direct but carries a higher risk than the elective technique. As this profile is built over time, we consider that the investment a customer makes in informing James as to his or her specific likes and dislikes contributes to the sense of loyalty between customer and Agent system. It is clear that these mechanisms must be applied judiciously; the Agent should not appear to act slavishly as a consequence of each preference. It must also allow for exceptions, such as when a customer selects a gift for someone with different tastes, and for the changing tastes of individuals. To address this point the affective state of the customer within context of situation, event, service, time, action, system response and overall system environment is also taken into consideration while maintaining the customer profile. User affective states are inferred through the interactions with James and the consequent services request, as well as through the use of Affectiveware as discussed in the section on Gathering Customer Affective States.

Types of Multimodal interactions devices used

In the James demonstrator implementation we provide services in the following modalities, voice, text, voice and text, animated character with voice. The system facilitates enabling, disabling and combining these modalities to aid the evaluation process. In each case the information presented, the services provided and the recommendations given are all the same (i.e. that applicable to the wine, but derived from the same database source) and the only variations is they channel in which the output is conveyed.

1. Text only output

Here help and recommendations provided by the system are presented to the user in text boxes that are displayed on request. These boxes are in the form of little “Post- it” note boxes that appear in a constant position on the right hand corner of the display. The user can close (remove) the text boxes (Post-its) by right clicking on the box.

2. Voice output

The same help and recommendations provided in the text boxes are prompted to the user through audio channels. The user has to listen to instructions; product descriptions or recommendations to get the information requested but can interrupt the voice message at any time.

3. Voice and text only

IST-1999-11683:D6.1 31 March 2002


Here both the modalities described above are combined. The help and recommendations are presented in text boxes displayed and are read to the user using text-to-speech synthesis The user can listen to instructions, product descriptions or recommendations and/or can read the information text displayed in the Post- it box at the same time. The advantage here is that the user’s attention is drawn to the box displayed and has the option of either listening to or reading along with the synthesised voice

4. Character with voice output

Here James is embodied in a synthetic lifelike character animation that moves about the screen and can point to objects on the screen providing help, recommendations and product descriptions through audio channels.

We use affect primarily for its influential implications on behaviour, and specifically because it allows the implementation of a social channel of interaction that is fundamental to the effectiveness and believability of a conversation. For this channel to be truly affective, the James character must not only facilitate the channel, but must appropriately respond to cues from the user and the system environment and further, be able to produce affective responses that reinforces the intended message rather than confuse the situation [Bates, ‘94]. An example in this demonstrator would be that the James will express happiness when he has successfully accomplished a task requested by the user, or excitement if he has found special offers that match the user’s preferences or requirements.

2.2.1.2 Gathering Customer Affective States

James uses a serial interface to measure customer physiological parameters. The connection between psychological and physiological response is used to investigate and define the suitable rules to provide the appropriate indicators that will determine James affective dimension of behaviour when responding each customer. We use a two channel light weight and portable affectiveware. The first channel will measure blood volume pulse (BVP) and the second - skin conductance (SC). Both of them are easier to be placed over the user body and do not require specialist knowledge to apply and use. Because of their design, BVP and SC sensors are safe, and do not cause bodily harm in case of misuse. In our beta version the frequency of measurement collection is 10 Hz (integration time of 0.1 sec.).

Increase of BVP is a universal measurement for the physiological changes that give an indicator of a change of the customer’s affective state. Any changes are registered at relatively small time intervals of 10-20 seconds according to current implementations of such affectiveware at MIT [Picard, ‘97]. A disadvantage is that BVP measurements can not distinguish positive or negative responses. This is determined through the context of the situation at hand. However, BVP gives enough indication that there is some sort of change that is then filtered more precisely.

Skin conductance response gives an indication that the customer may not feel comfortable for some reason or another that may or may not be attributed to the

IST-1999-11683:D6.1 31 March 2002


interface or interaction. The response may be in situations like irritation caused by the screen content, frustration from services provided or the fact that a requested product is not in stock, and so on.

It must be stated, however, that these physiological changes may be caused by changes in physical environment around the customer, such as phone calls, presence of other humans in the room, radio news, etc and are not related to the application. Nonetheless, James uses the collected data to formulate associations with events occurring and uses them to produce more appropriate reactions. If James is unable to make these associations he will prompt the customer with an enquiry.

The data collected and the associations formulated are maintained in the user profile. The software written for James will provide database access to all measurements of the beginning of the session and event generation if there is significant change in one of the channels.

2.2.1.3 Maintaining Customer Profiles

In the James the Butler demonstrator profiles are used by agents to take on certain aspects of the role of retailers and customers. The idea is that a customer has a buying history which can be used to form the basis of a profile of that customer’s preferences for certain products and services. The customer is encouraged to view and change this profile so that it can better reflect the customer’s preference. An agent uses this profile along with domain knowledge about the products available in the system. This means a consumer can ask very general questions about the products and the agent will filter them according to the consumer’s profile and its domain knowledge. It can then search the product databases for specific products that match the filtered request.

This can be made more realistic if other agents in the system have profiles that enable them to work on behalf of the vendor. The vendor can have information of a particular customer’s purchasing preferences from looking at previous purchases the customer has made. The vendor agents can then make suggestions to the consumer about offers and deals tailored to its perception of what s/he wants. Naturally, as the customer buys more the vendor agent will have more data to reason about the perceived preferences and so can tailor its offers better to meet customer needs.

Given this scenario it is not hard to see that agents can then negotiate on a customer’s behalf. A customer indicates that they are interested in a type of product. A search of the system according to the personal profile returns to the customer a certain number of choices. However a vendor agent might be able to make a number of offers or deals to the customer trying to get the customer to buy products it wants to sell but are at the same time relevant to the customer. At this point the vendor and agent and the customer agent could negotiate using the customer’s preference on how much he or she are prepared to spend and the vendors preferences for increasing gross margin and selling products that need pushing. A compromise is reached and the customer is given a number of choices.

IST-1999-11683:D6.1 31 March 2002


2.2.2 Toolkit Components Used

The demonstrator is developed using the SAFIRA Toolkit editor. The main components used are the CRS (Central Registry Service) that provides component interaction APIs and messaging services, the CML (Character Mark-up Language generation and scripting component) for the behaviour description and animation scripting language of James’s expressive behaviour and animation, the ASM (Affective Speech Module) to generate James’s affective spoken utterances and the Appraisal component for the reasoning about and the generation of the appropriate emotional signals that are used to influence the planning and generation of James’s affective behaviour.

Internally the Winestore data and knowledge-bases have independent modules connected to the CRS (Central Registry Service). These modules are written in Java and are connected with databases via JDBC. They can proceed with specific queries by connecting with the information stored.

The product database uses the CRS to return full information about the wine specified by its number. The client database is able to provide demographic information about the client, such as their age, gender and nationality. It is also able to record their prior behaviour on the site.

The web-based interface for the Winestore does not actually use any of the affective components produced in SAFIRA. James the personal sales agent communicates directly with the Internet browser using the CRS. This will improve the speed of the interaction between customers and James.

2.2.3 The Demonstrators Internal Architecture

The James the Butler demonstrator is implemented as an agent, which uses an expert system to manage its internal states and uses the components defined by the SAFIRA toolkit to for its affective behaviour. The agent maps the semantic of FIPA performatives to basic JESS (JAVA Expert System Shell) functions. More complicated commands are communicated to the Database Agent in the content of an FIPA ACL and executed by the JESS engine. The agent uses profiles to dictate its basic operation style and the kind of knowledge it works on. For example, on start-up in the system the James loads a Knowledge base which has domain facts and heuristics about wines. When a user logs into the system after validation James loads that persons profile, the agent then uses these two knowledge bases to personalise general questions from the user to find the appropriate product that best fits the users request and profile. This basic functionality can also be made to act as a Vendor Agent acting on behalf of the used retailer. It can do this by loading the appropriate profile for a vendor and then it will behave accordingly. Vendor profiles contain information about products held at the vendor store: inventory, prices, discounts and special offers. Its functionality is biased to the vendor and it is the role of the agent is to filter information from available vendors based on user budget specifications and preferences and present the customer with a list of one or more alternatives.

IST-1999-11683:D6.1 31 March 2002


2.2.3.1 The Big picture

The SAFIRA approach aims at providing a framework that will support not only the traditional view on what it means to access a service-providing system, but also the support new ways of utilising the services offered. As society is becoming more and more networked, powerful devices are becoming smaller, and small devices are becoming more powerful, we can now envisage what technology will enable in the near future.

The intention is that users can initiate interactions with SAFIRA services from their home system, leave these running when they go off- line (i.e. disconnect), and resume the interaction the next time they connect to SAFIRA. The next connection may be from an in-store fixed system (e.g. kiosk) or mobile device.

The logical architecture of the system is illustrated in the following figure.

Fig. 2.1- James the Butler Logical Architecture

It is assumed that each active device in the overall system – Winestore central, Client device, kiosk, etc. – supports up to three processes. Firstly, WAP (Winestore Agent Platform), which supports personal agents; secondly, an Affective Engine (AE); and thirdly, some presentation system to suit the display characteristics of the actual device. Winestore central system needs only to support a WAP: any other device (i.e. with interaction requirements) must support all three.

WAP is required to support mobile agents, the AE to co-ordinate display of personality, emotional state and character, and the presentation system to display all multimedia objects (including character) according to device constraints and capabilities.

IST-1999-11683:D6.1 31 March 2002


The intention is that agents ‘live’ (reside) on a user-owned device. By issuing a GO command an agent moves from its resident WAP to the WAP on some Access Point (this can be either the home device or the in-store kiosk). User interactions with the agent are then through the presentation system of the device moved to.

2.2.3.2 Agent Architecture & Functionality

In the system that we investigate agents are software components that act on behalf of a user or a group. Internally an agent consists of a head component and a body component [Mamdani et al 1999]. The head component is a generic, domain independent component that allows the agent to control the communication of the agent with the user and other agents including the agent’s interactions with the environment. The head also contains the agent’s current state, which when the agent is active, it contains domain independent knowledge and the progress of all interactions that the agent is currently engaged in.

The body component is a domain dependent component that consists of a set of roles. Roles are components describing the agent’s capabilities and a set of rules describing what an agent can or canno t do while playing that role within a specific environment. From the viewpoint of software development a role contains a private state and groups units of capability for an agent, the tasks that the agent can perform in the system. Private information within a role state can be shared between the different roles in the body by using the state of the agent specified in the head as a kind of shared memory.

A role can be complex in that it can be composed and defined in terms of sub-roles. For this purpose, we distinguish between atomic roles and compound roles. Atomic roles are the simplest possible roles and may contain only the capabilities of the agent in the form of tasks. Compound roles on the other hands contain tasks and additional structure in the form of sub-roles.

The single agent architecture is used as a component itself to build a multi-agent system systematically and possibly dynamically. We take a synthetic approach where users and their associated user agents form complex organisations representing the community system. This distinguishes between the presentation and the environment. The presentation layer abstracts away from the mechanisms used to visualise the environment, while the environment contains the entities and their interactions including components representing agents. The entities involved - apart from the agents – are a set of user spaces holding data structures owned by the user.

One important characteristic of the architecture is the way agent actions are represented in the architecture. At this preliminary stage, the representation of an agent’s action consists of two main parts: • Instantiation part – holds an instance describing the action, for example:

set name => “John” surname => “Smith”; • Character part – holds a series of meta-actions showing the behaviour of the agent

for a specific action, for example:

IST-1999-11683:D6.1 31 March 2002


before action=>set and param=>name do look=>serious;

after action=>set and param=>surname do look=>happy.

The distinction between action instantiation and character allows us to specify separately the action from the character, while at the same time associate each action with a specific character. It is then a domain specific decision to select which agents exhibit character. If we decide that an agent should not exhibit character, the character part for each action will be empty for that agent.

This section discusses the preliminary work on integrating affect models in Multi-agent systems. The aim is to provide interface agents with personality, emotional responsiveness, and affective user modelling.

2.3 The Winestore Architecture

Each of the agents represented in the figure above are briefly described below.

Fig 2.2 Architecture of the Winestore

2.3.1.1 RSVP: The Rapid Serial Visual Presentation (RSVP)

In the RSVP Presentation System display method, many "products" are shown in a display window This area serves as an on-screen metaphor for a shelf of products, as might be encountered in any shop or store. It is an alternative to the “catalogue pages”

IST-1999-11683:D6.1 31 March 2002


approach familiar to all those that shop “online” regularly. RSVP largely overcomes two familiar problems with the catalogue approach. First, the customer may browse a large number of products quickly in a single page. This avoids the familiar problem of navigating many pages to locate a particular item. Second, the contents of the “shelf” may be quickly configured to display particular products of interest to the customer..

A customer interacts with the RSVP display in three ways. First the customer may “riffle” through items on the “shelf” using a pointing device (mouse pointer, or with the finger on a touch screen), as if picking the item from the shelf and inspecting it. As the pointer moves over the visible portion of each product image the touched product image moves away from the others. It is thus made more visible and is highlighted. The effect is both distinctive and effective. Second, the customer may move back-and-forth “along” the shelf, as if they were moving to another part of the shelf. This is accomplished by “left” and “right” buttons. Equally, a slider bar could be used to traverse the length of the “shelf”. Third, the customer may indicate particular interest in a product, or request a purchase, by selecting it (for instance, with a mouse click/double click, or by tapping a touch screen).

Fig 2.3- The RSVP display area

Each action by the customer is messaged to the James, which can intervene at any point, or just observe the customer’s activities. The Presentation System is accessed using a series of methods, communicated to the Presentation System using the CRS. Unsolicited events, such as actions by the customer, are reported via the Proxy Agent to the Personal Agent with well-defined ACL compatible messages.

The Presentation System implements and exploits a version of the RSVP idea of Spence, and others [Lam and Spence, 1997], [Wittenberg, et al., 1998], [Spence, 1999] and [deBruijn and Spence, 2000]. These researchers have been primarily interested in understanding how computer users may most rapidly and effectively identify image files within a large number of similar images. Our primary concern is to enable a customer to rapidly and effectively browse a potentially large number of (possibly

IST-1999-11683:D6.1 31 March 2002


rather similar) products in a single display, and to select a product to obtain further information, and finally to make a purchase of any item(s) required. Additionally the RSVP display acts as a presentation area over which the Presentation/Personal Agent may roam, providing extra assistance to the customer or make a sales recommendations.

2.3.1.2 Proxy Agent

The proxy agent is a wrapper for the RSVP, which can also communicate in ACL with the other agents.

When it is started it registers with the Directory Facilitator (DF) and with the Agent Management System (AMS) and then waits for messages from the Personal Agent transforming them in C++ procedure calls on the RSVP. Therefore the Personal Agent is able to communicate with any Presentation Engine by providing additional wrapper information to understand and call methods in a replacement Presentation Engine.

2.3.1.3 The Database Agent

The overall architecture of the Winestore knowledge and data-bases is shown in the figure below. There are two independent databases. Both of them are based on the MySQL server.

The “products” database contains information about the items which are on sale via the internet store. It includes records of the wine bottles, like their country of origin, the type of grape and their value. It also contains specific information on how any particular wine should be stored and consumed.

The “clients” database contains the information about the customers using this server. It basically forms the user profile. The customer records are made secure by requiring customers to provide a username and password to access the site. The persistent aspect for personalised individual custom sessions are collected and maintained in this profile.

Both databases are connected with James and web server using ODBC. The whole system is installed on a single computer. The webserver, products database and the part of the client database that contains demographic information constitute a fully operational system which can work alone..

The main functionality of the database module is to store the information and to make it accessible to the other modules in the whole system. Its standard response to queries is via ODBC. Additionally it can accept queries through CRS. The more specific task which this module can undertake is to generate a list of wines that can be recommended to the client. The criteria for this selection can be defined by the clients themselves or generated by James.

IST-1999-11683:D6.1 31 March 2002


2.3.1.4 The Winestore Web Interface

The interface has a five frame structure. On the left panel is a selector which allows the client to choose wines according to their prefe rence for a certain grape, region, colour, value and so on. On the right panel is the shopping basket. Between these two panels is the main frame which represents a realistic view of a wine cellar with one or more shelves containing numerous wine bottles for the client to select. When a particular bottle is selected, it becomes enlarged so that the client can read the label.

On the top right hand side is a banner providing advertising space. On the top left is the Winestore logo. This logo contains active Java applets, and is used to establish connection with the Sales Agents and the client database

The screen objects are designed for easy use by the client. The background of the left and right hand panels is a rustic wood panelling. When they move the mouse onto a button, it becomes highlighted.

The selection buttons offer the client the opportunity to set the parameters of their wine selection, for example, the grape they would like to view.

The shopping basket automatically calculates the total price of all selected bottles. The buttons provide the client with the opportunity to add or exclude bottles from their basket.

2.3.1.5 Affectiveware for Affective Feedback

It is highly desirable to take account of the emotional state of the client using the system, tapping their emotions as they interact with the Winestore system. Using additional hardware such as affectiveware will allow the physiological response of the user to be captured. This physiological input can then be used to deduce the estimated emotional state of the client. It is anticipated that the efficiency and authenticity of the whole system will be improved by this affectiveware module.

2.3.1.5.1 Architecture

There are two computers participating in the proposed architecture. One executes the CRS module. The other runs the registration software, which records the physiological responses of the client’s body.

Both the modules drawn with dashed lines (video camera and registration software) will not be included in final release of the system. They will be used to develop ‘rules of influence’ which will define how the Sales Agent responds to the client’s estimated emotions. They may be used during the evaluation.

IST-1999-11683:D6.1 31 March 2002


The major functionality of this module is to give an opportunity for the system to respond according to the client’s emotional reactions. This is achieved by analysing the electrical signals captured by affective observation of the user. In this system two measures are used, blood volume pulse(BVP) and skin conductance(SC). BVP gives information that the emotional state of the user has changed. SC indicates the direction of this change, either positive or negative.

IST-1999-11683:D6.1 31 March 2002


3. FantasyA

FantasyA is a computer game where users play the role of an apprentice wizard who is challenged to find the leader of his/her clan in the land of FantasyA. The game has an introductory phase, where the four 1 clans, the magic stones, and the duels are explained. Then the player is placed in the land of FantasyA where he will engage in exploration, duels and cooperation in order to find his leader of the clan.

The main goals of FantasyA are:

- To test some of the components developed in Safira (associated with the Safira Toolkit), in particular

o the affective input through a physical interface; and

o the affective body expression

- to design a computer game that uses “emotions” and emotional expression as a way to evolve in the game.

This section provides a description of the game in four different parts: (1) the concept, the setting and development of the game itself; (2) description of the emotional agents in FantasyA; (3) the technical architecture built for the game and the game engine; and (4) the components used from the Safira toolkit.

3.1 FantasyA: the concept

FantasyA is a world guided by the Dreams of seven mighty Gods who, although imprisoned in the godly stones, the KemY’s, use the Dreams to impose their will over the inhabitants of the Land. Through the Dreams, each God chose His IdenjaYeh, His first child and, for Ages, the seven IdenjaYeh have been the uncontested rulers over the land of FantasyA. Each IdenjaYeh preaches a different Path, the Path of his God, the Path of one Element. And around the seven Paths are the seven Clans founded.

Guided by the Gods, the IdenjaYeh built the Covenants, the strongholds of the Clans, symbol of Learning and Power. There, the Magi of each Clan, after being called to the Covenant through the Dreams, are trained in the art of AlkemHYe, the art of Elemental control. But one day, the

1 There are seven clans in the original concept. However, this version of the game will only consider four clans.

IST-1999-11683:D6.1 31 March 2002


dreams suddenly ended... and FantasyA was left to its own devices. The IdenjaYeh, predicting a time of Chaos and war across the land, left for the KestYe, a quest to recover their Dreams. Long and cold winters passed, until one day... one IdenjaYeh returned.

He was accompanied by a Menjeh, a man that cannot Dream. The IdenjaYeh ordered the Elders of his Clan to train the Menjeh in the ways of his Element. His final words were: “Send him to me as soon as the Training is over”. Nothing else he added before leaving silently the Covenant... One year passed and the chosen one finally finished the training. After the Initiation Ceremony, the Menjeh leaves the Covenant to search for the Iden-jaYeh.

You are this Menjeh.

After this initial introduction and before starting the game, the player must choose the character (s) he will be controlling throughout the game. This is done by choosing the clan of the character and its date of birth. The choice of the clan will directly determine the initial equipment the character will find in the room (s) he will begin in, and also act as cues to the inherent magical aptitudes of the character. Each clan has a set of particular strengthneses that characterize all the members of that clan. Then the player starts the game entering the world of FantasyA.

3.1.1 The Clans of FantasyA and the choice of the avatar by the player

There are 4 clans in FantasyA , each one identified with an “element” of the land of FantasyA.

1. The clan of Air, associated with magic gem Saphire (Safira) is the realm of sensibility and reflexes. So all members of the clan are able to predict what is going to happen and react very quickly to the events of the environment.

IST-1999-11683:D6.1 31 March 2002


Figure 3.1 The clan of Air

2. The clan of Fire, associated with magic gem Ruby. Fire is the realm of quality and associated with the physical property of agility. So, all members of this clan are able to move rapidily in the world and reach places the others do not.

Figure 3.2 The Clan of Fire

IST-1999-11683:D6.1 31 March 2002


3. The clan of water, associated with power and strength. All elements of this clan will be able to follow their objectives in a quite proactive way.

Figure 3.3 The clan of Water

4. The clan of Earth is associated with strength and stamina. So all elements of this clan are strong and last longer to the attacks of their enemies.

Figure 3.4 The clan of Earth

Each character from a clan will have both some characteristics of its personality and the physical appearance and capabilities associated with that specific clan.

IST-1999-11683:D6.1 31 March 2002


An auxiliary tool has been developed that will create the characters personality based on the choice of the clan (Air, Fire, Earth, Water) and an astrologic map that relates each day with a specific point in the FantasyA elements sky. Based on such tool, personality factors inspired in the Myers-Briggs model are created for that character. These factors influence the way the emotions will be appraised as well as the emotional decay.

3.1.2 Initial phases in the Game: The clan antechamber and the Elemental room

The game begins in an initial room which is a small antechamber with walls decorated with motives giving cues on the actions to take within it. Most rooms will abide by that simple principle, providing in their inherent design the key for their inner secrets. In this particular room, the character will find two important items and learn the basic principles and prime rules of this Magic sub-world. The user will spot these two items easily. Only after learning these principles will the only massive and heavy door in the room open, clearly the only way out of it.

Once this first door opened, the character will be able to walk into an enormous room at the center of which four doors stand, each one depicting one of the Elements: the Elemental Room (see Figure 3.5). On the outside walls, doors from which will come the adepts from each one of the clans can be seen but not opened from inside the Elemental Room. Other doors exist but remain well hidden (only combining power can those be open).

Figure 3.5. The elemental room

At this point, it is then possible to encounter other will-be Wizards (user-controlled or computer-controlled agents) already standing in the room or entering it from their own antechambers with their staves and gem, or from any nearby room…

IST-1999-11683:D6.1 31 March 2002


3.1.3 Meeting other agents in the world: the Arena and the Emotional Duels

Beyond this initial room, some doors will only open upon the presence of a certain elemental power hardly gatherable by one adept alone. Each magical door has a maximum and minimum value for each of the elements that must be met for the door to open. Shall one of this requisite fail and the door shuts.

By opening and crossing new doors, the students of Magic will learn the value of cooperation and duelling in this world. They will learn that a gem is more powerful in the hands of a trained adept of the related Clan than in the hands of another - the aura around the characters clearly state it - since the staff (s) he possesses will be able to amplify gems in a way that (hers) his can’t. Apart from cooperation, they can obtain the magic gems is by duelling. In duelling, it is the ability to control emotions that puts the character aside from the rest of the world of Mages and marks him as the future Hero of FantasyA. Sometimes this can turn out as a flaw if misused, but well controlled... the player’s character will be able to control/influence (her) his emotions and enhance or inhibit his magical powers. The player has a good advantage over most of the other characters in the game, since (s) he can control (her) his own emotions. With most of the computer-controlled characters, emotions are always a result of the appraisal made to the current state of the game. This kind of interactions will help the player to understand and model other agents as well as other will do regarding (her) him. Emotions are at the center of all inter agent communications. To comprehend emotions is of the uttermost importance to succeed in the game.

3.1.3.1 Emotional Duels

The goal of a duel is to win an opposite wizard, obtain his/her magic gem and continue the game in search of the Clan leader. Duels happen in the Arena (see Figure 3.6). In a duel each player can send attack spells to the opponent and protect (herself)himself from the attacks of the other through defence spells. Each spell, when sent, requires some energy to be used by the issuer of the spell. The spells’ intensity is not only set by the player but also depends on the emotional state of the character. For example, if a character is angry and is attacking the intensity of (her)his spell is intensified. On the other hand, when attacked, because he is angry, he will be more affected by the attack (thus will loose more energy).

IST-1999-11683:D6.1 31 March 2002


Fig 3.6. The Arena

The player will control both the emotional states of the character (in order to control the intensity of the spells and how they affect the character) and the type of spells being sent to the opponent. A duel ends when one of the characters (the looser) reaches an energy stage that is so low that he will not be able to recover in time and thus it will loose his magic gem. So the other character wins the duel and wins the gem that the loser had in his staff.

3.2 Agents in FantasyA

In FantasyA there are three types of agents:

- directly controlled avatars- which are the avatars used for a direct control by the player in the game;

- semi-autonomous avatars- which are the avatars used for a kind of influence control by the player in the game; and

- autonomous agents- which are the other characters that exist in the world with whom the player is going to interact with.

IST-1999-11683:D6.1 31 March 2002


So, when entering the game the player may choose two types of control over his/her character (agent):

- Direct control , where the player controls the emotions of his character directly, with no other aspect influencing the character’s emotions

- Influence, where the player influences the emotions of his character, which can also be influenced by the state of the game itself. That is, the character will have a certain emotional state not simply because the player says so, but also because something is happening in the virtual world that will make the character have a certain emotion.

These different types of character control were designed so that we will be able to evaluate what type of control do players prefer and in particular for different types of players and to give more richness to the game, such that it allows the exploration of semi-autonomous avatars in the game.

Note that controlling the emotions of the characters in a game like this (with the SenToy interface) is a task that needs to be mastered (by appropriately using SenToy). So, players usually start with direct control (to learn how to use the SenToy) and move onwards to the Influence control.

3.2.1 The internal state of an agent in the game

An agent in the game (be it an controlled avatar, a semi-autonomous avatar or an autonomous agent) has an internal state that contains:

- the knowledge about the FantasyA world, including knowledge about the objects that exist in each room, about the gems, and about his own objects;

- goals of the agent in the particular game situation;

- the emotional profile, which is a result of the personality of the character (which results from the initial choice done with the Character Creation tool). This emotional profile characterizes the appraisal parameters of the character and the emotional decay (how easy it if for the agent to get back to the neutral state after an emotional state is active).

- the emotional states that characterize the emotions the agent is experiencing at a given moment (according to the definition of emotion done in D2.1, D4.1 and D3.1);

- energy levels, one level for each type of element in FantasyA;

IST-1999-11683:D6.1 31 March 2002


- recovery rate of that character which determines how the character will recover from an attach in the different energy levels;

- historial of performed actions.

3.2.2 Actions that the agent can perform in a duel

Apart from the typical actions that the agent can perform in the land of FantasyA (such as walk, pick an object and open a door), each agent can perform an extra set of actions in a duel situation. These are:

- change his emotional state (because of the user’s control over the emotional state)

- perform an attack spell (to attack his opponent in the duel)

- perform a defence spell (to defend from his opponent).

These actions can both be controlled by the player (when controlling his/her own character) or by the autonomous agents, when fighting against another character.

3.2.3 Emotional states of the agents

An emotional state of an agent in FantasyA follows the emotional description adopted in Safira. The table below shows the specification for the attributes for an emotion. The attributes considered for the description of an emotion, are according with the OCC theory of emotions [Ortony et al. 1988], which was set to partially supporting theory for the system implementation.

Attribute Description

Class The id of the emotion class being experienced

Valence

Denotes the basic types of emotional response. Neutral, positive or negative value of the reaction.

Subject The id of the agent experiencing the emotion

Target The id of the event/agent/object towards the emotion is directed

Intensity The intensity of the emotion. A logarithmic scale between 0-10

IST-1999-11683:D6.1 31 March 2002


Time-stamp The moment in time when the emotion was felt

Origin The component that originated the emotion.

The attribute Class describing an emotion refers to the type of that emotion. An emotion type represents a family of related emotions differing in terms of their intensity and manifestation, i.e., each emotion type can be realized in a variety of related forms. For example, fear with varying degrees of intensity can be seen as– concern, fright, petrified.

In FantasyA we will only have the following 6 emotional states: 1. Joy/Happiness 2. Distress/Sadness 3. Surprise (which does not exist in the classes of the OCC model) 4. Gloat 5. Anger 6. Fear

The reason why we have only chosen these six emotions (rather than adopting the 24 emotions of the OCC model) was that we needed to select the right emotional states that:

- were possible to recognize in facial and body expressions of our characters, and thus be able to provide immediate feedback to the game player;

- were possible to recognize from the gestures handling the SenToy interface;

- would make sense in a game like situation (FantasyA), so that controlling those emotions would provide some advantage to the player.

The attribute Valence describes the value (positive or negative) for the reaction that originated the emotion. According to this theory, emotions are always a result of positive or negative reactions to events, agents or objects.

The Subject and Target attributes for emotions, define the entities related to them. The Subject defines the agent experiencing the emotion and the Target defines the event, agent or action that originated the emotion.

One must have associated to every emotion, an attribute of Intensity, which is assigned with different values depending of the different situations that gave arise to that particular emotion. The value for the intensity parameter is calculated from the difference between the emotion’s activation minimal threshold (Activation Threshold) and the intensity potential value of the associated emotion directly evaluated from the

IST-1999-11683:D6.1 31 March 2002


eliciting situation (Emotion Potential). Thus, the formula for the intensity parameter for a specific emotion (em) when it is generated, is as follows:

Intensityem = EmotionPotentialem – ActivationThreshold em

Each agent has parameterized its emotion threshold for each emotion and energy level (the threshold from which the emotion becomes active). This emotion threshold, similarly to the decay rate, depends on the personality of the character and is established in the beginning of the game, when the character is created.

3.2.4 Appraisal

The appraisal of these emotions is based on an appraisal structure that depends on the following elements:

- goals of the agent, the goals of the agent will impact on his appraisal in the sense that the proximity of achieving a goal will determine the activation of the emotions fear or surprise.

- The energy state of the agent (which will impact on the achievement of its goals)

- The energy of the opponent (which will impact on the achievement of its goals)

- The events happening in the duel (each attack and each defense).

For example the emotion “anger” will be triggered when the agent has a high level of energy, his goal is to attack the opponent, the opponent has a low level of energy, and the opponent attacked him instead. This is a situation where the agent has the mixed emotions of disproving the action of the opponent, which he wasn’t expecting, and is distressed by the attack. Note that the emotion “anger” in the OCC theory is a compound emotion that combines reproach and distress. Differently the emotion gloat is triggered when the energy of the agent is high, the energy of the opponent turns low and the agent is able to further attack the opponent.

The emotions joy/distress (or Happy and Sad) are emotions that do not depend on the opponent’s state of energy but is a simple reaction to a desirable (successful attack) or undesirable (being attacked) event that happened to the agent.

Concerning the emotion fear, it is triggered when the agent has a low level of energy (so he is expecting something undesirable to occur), his opponent has a quite high level of energy and he has been attacked.

Finally the emotion surprised occurs when the agent has a low level of energy, and the opponent has a high level of energy and the agent is able to attack the opponent successfully.

IST-1999-11683:D6.1 31 March 2002


3.2.5 How do the emotions influence the duel: Physical Capabilities and Action Tendencies

Although partially inspired by the OCC theory of emotions, we had to go further than the OCC when deciding what the emotions meant in the context of the game. That is, how would the emotions, once appraised and triggered influence the behaviour of our agents in the game. To do this we used the work by R. Lazarus [Lazarus 91] which also inspired the architecture of the agent’s provided in D4.1 where the emotional states triggered will have two types of effects:

o physiological effects and body changes; and

o action tendencies.

The body changes are characterised by a differentiation in the animation and body posture of the character. This is done using the affective body expression component developed in WP5. The action tendencies determine the types of actions the character is more inclined to perform and thus actions that are intensified in their performance. For example when experiencing anger the character will attack more (see action tendencies for anger in the Lazarus theory) and the intensity of the attach will be increased. Differently when the character experiences fear, the character will defend itself more and thus his protection will be stronger and thus it will be more resistant to the attacks of the opponent.

Note that the action tendencies resulting from the 6 chosen emotions need to be clear to the player, because he/she will control the emotions through the SenToy and needs to know what will be the result of this control.

3.2.6 Four Characters of FantasyA

For the moment, only four characters are planned: two male ones, one from the Earth Clan and one from the Water clan and two female adepts, one from an Air and one from the Fire Clan. The construction of the characters passed through several phases, where the first ideas were put into paper, to the character choice of name, profile, image, until the modeling using 3D Character studio. Our characters are: Ronin, AlVega, Feronyia and Nack.

3.2.6.1 Ronin: a character from the Earth Clan.

1. Choice of the name and choice of personality.

This was the first character to be developed and we decided that he would belong to the Earth Clan because at the time we felt that we already had gathered a number of references that would allow us to define how a member of this clan would look like and act like. The first question was how to interpret its bond with the Earth, would he be a knowledgeable farmer settled in the vast plains or a nomadic hunter-gatherer

IST-1999-11683:D6.1 31 March 2002


wondering deep inside the forest? These two broad descriptions appeared as the most able to accommodate the characteristics defined in the concept documents “Whispers of the Planes” and “Yertamenh” for the Earth Clan; physical stamina, willpower and social presence. However, they would determine radically different physiognomies and expressive qualities. We chose the latter for its visual potential, since it allowed us to easily merge a wide range of graphic elements from different cultures and eras (which is one of our main guidelines) as to pursue the timeless quality expressed in FantasyA’s concept. This led to the definition of physical traits such as small stature, compact torso and long arms, which are consistent with is natural environment and that provided the basis for establishing his posture and the characteristics of his motions. At this point, we felt that physically the character was already coherent with the proposed description and to achieve the intended degree of believability we would need to give him cultural and emotional references that would add the needed depth to his personality. What we did was to turn to the Japanese culture as source of inspiration; more specifically, the animist cult of Shinto and the romantic ideal of the Samurai. The first accounts for the strong tie with Nature and its manifestations as well as the respect for all living creatures that is the foundation of his character. Also, its warrior side comes with the ethical code of the samurais. Having established the psychological aspects of the character, we came to the point where he only needed a name to come into his own, and once again drawing from the Japanese culture we chose Ronin, a rogue samurai.

2. Some initial sketches.

Figure 3.7- Initial sketches

IST-1999-11683:D6.1 31 March 2002


3. The evolution from an initial design and the actual design.

Figure 3.8- Evolution of Ronin’s model

Figure 3.9- Ronin walking

4. Further design decisions

To design objects that fit into his environment, prepare the model’s face for facial animation and define morph targets in order to animate the transformations caused by every element.

IST-1999-11683:D6.1 31 March 2002


5. Texture.

Figure 3.10 The texture of Ronin

7. Types of animations particular to this character and to this clan

This character, apart from the typical animations such as walking, running, picking an object, attacking and defending, also has a set of actions that are specific of its clan. They are:

- Tree climbing. - Cloaking behavior.

3.2.6.2 Al-Vegha: a character from the Air Clan


We chose a brave personality for this character for the following reasons. First, we needed someone brave, to act like a hero during game play. The forms in which the clan presents itself: Intuition, Reflexes, Awareness and Wits and the environment in which the clan inhabits, the skies, reminded us of liberty and plenitude and is adequate for the personality of a hero. The name Al-vegha came with a twist. The name is based in a character of a Portuguese comic book called Major Alvega. Major Alvega was an ace pilot during World War II, in the British Royal Air Force. Half English, half Portuguese, he fought the nazis with bravery and class, a bit like James Bond. His connection to the air was obvious, and the name had a mysterious touch to it. After altering it to Al-vegha, to blend it into the game look and feel, modeling was much easier, having the Major in mind.

IST-1999-11683:D6.1 31 March 2002


2. Some initial sketches

Figure 3.11. Initial sketches of Al-Vegha

3 The evolution from an initial design and the actual design.

Figure 3.12 evolution of Al-Vegha

IST-1999-11683:D6.1 31 March 2002


Figure 3.13 Al-Vegha walking

4. Texture.

Figure 3.14 Al-Vegha texture

5. Types of animations particular to this character and to this clan:

- Flying (gliding, hovering and diving). - Landing. - Cliff hanging.

IST-1999-11683:D6.1 31 March 2002


3.2.6.3 3. Feronyia: a character from the Fire Clan.


Feronyia is the name of a Sumerian deity of fire and fertility that we found suitable to illustrate our vision of a member of the Fire Clan because of its connotations. Fertility is widely associated with the feminine gender and fire is also frequently linked with knowledge, which is a perfect description of a female of the Clan we agreed would exemplify the technological aspect of FantasyA. The members of this Clan are initiated in the secret art of the blacksmiths, using the power of fire to forge the tools that allow them to endure the harsh conditions of their native environment, the smoking craters of dormant volcanoes, or to dig deep into the Earth searching for substances only known to them. This scenario was developed according to the concept documents “Whispers of the Planes“ and “Yertamenh“ which state that the physical, mental and social traits of the Fire Clan’s members are agility, intelligence and eloquence. That led us to envision a character that would be able to combine a graceful yet solid posture that would express the notion of pride, or even a hint of self-confident superiority, with a pragmatic attitude rooted in a functional perception of the world. To this purpose, we modeled a slim, tall figure with an aggressive hairstyle, which we dressed up with a synthetic-looking outfit mixing medieval craftsmen and contemporary fashion formal references, trying to convey the ideas of elegance and contained strength.

2. Some initial sketches.

Figure 3.15- Initial sketches of Feronyia

IST-1999-11683:D6.1 31 March 2002


3. The evolution from an initial design and the actual design.

Figure 3.16 evolution of Feronyia

IST-1999-11683:D6.1 31 March 2002


4. Texture.

Figure 3.17 Feronyia texture


- Public speaking.

- Objects crafting.

3.2.6.4 Nakk: a character from the Water Clan.

1 Choice of the name and choice of personality.

We chose an evil personality for this character for two main reasons, we needed something bad, to interact with the other characters that were all good people, and the forms in which the clan presents itself: Power, Strength, Perception and discussion. The environment in which the clan inhabits, the ocean, reminded us of the unknown, of what is still to be discovered either beautiful or terrible. After a quick search of deities connected with water, it was fairly easy to choose the name Nakk (a finish evil spirit), also because it’s a short and strong word.

2 Some initial sketches.

Figure 3.18- Initial sketches

3 The evolution from an initial design and the actual design.

IST-1999-11683:D6.1 31 March 2002


Figure 3.18- Evolution of Nack

4 Texture.

Figure 3.20 Texture of Nack


- Swim. - Emerge. - Submerge. - Dive.

IST-1999-11683:D6.1 31 March 2002


3.2.6.5 Technical Details of the characters construction

All characters have been modeled with 3D Studio Max R3 using editable patches and meshes using the following techniques:

- Starting from a box, the character’s body is literally carved by editing the vertices, edges, faces and polygons.

- Designing the clothes with editable patches. First, we design the construction lines and then we create surfaces using these lines as boundaries, the resulting surface is then edited through it’s control points, during this process we can dynamically alter the level of detail, controlling the density of the mesh and it’s polygon count. At this stage, after deciding the level of detail, we turn it into an editable mesh without level of detail or control points.

- Attaching the separate meshes and further editing of vertices, edges, faces and polygons.

- Mapping and adjusting map coordinates using the UVW Map and Unwrap UVW modifiers.

- Defining internal skeleton with a Biped object.

- Adjusting the skeleton to the mesh using the Physique modifier.

- Testing movements.

- Animating with Character Studio.

At this stage, Al-vegha is currently being tested in editable mesh with 1198 polygons. We are using a test map with 1000x1000 pixels in 16 bits color. Currently, one alpha channel is being used to simulate transparency with 320x320 pixels in 16 bits.

Nakk is being tested in editable mesh with 1736 polygons. We are using a test map with 1024x1024 pixels in 16 bits color. Currently, no alpha channel is being used to simulate transparency.

Ronin is being tested in editable mesh with 1908 polygons. We are using a test map with 1024x1024 pixels in 16 bits color. Currently, no alpha channel is being used to simulate transparency. Feronyia is being tested in editable mesh with 1445 polygons. We are using a test map with 1024x1024 pixels in 16 bits color. Currently, no alpha channel is being used to simulate transparency.

IST-1999-11683:D6.1 31 March 2002


3.3 The Game Engine: technical description

To develop the game FantasyA we had two options concerning the choice of platform. The first one would be to use a game engine that would allow for a high level control of the characters and scene development, allowing therefore for a faster development. The other would be to start with a graphics package (DirectX or OpenGL) and build on top our models and our own engine.

To investigate the first option, and given that we started our work in the begin of 2001 we decided to investigate WildTangent and Unreal. To do that, we created a small demo (Wondering in FantasyA- see Annex) that would help us to see how fast and how we could integrate our components in the game.

There were some characteristics in the engine that were essential for the game. They were: - Possibility of integrating the component that handles the affective body expression - Possibility of integration of the SenToy components - Open source - Fast development

Unfortunately these three requirements were impossible to achieve the platforms studied, in particular the first one and the third one. This meant that we would have to develop our own graphics engine on top of OpenGL to support the development of FantasyA game. So, we decided to build a simple engine where the game engine could attach in a fairly simple way.

3.3.1 A simple graphics engine

Every graphical interactive application can be regarded as an infinite cycle with three phases:

1. Phase 1: get input data.

2. Phase 2: process the information and make changes.

3. Phase 3: redraw the scene with the required changes.

In phase 1, the application retrieves input information, such as the keys currently pressed in the keyboard, or the mouse pointer’s screen coordinates.

In phase 2, the information retrieved in phase 1 must be interpreted and, if necessary, changes are made to the appearance of the graphic scene.

In phase 3, the scene is redrawn after applying the required changes to the objects in the scene.

FantasyA’s engine is a little more complex than that but it follows the same philosophy.

IST-1999-11683:D6.1 31 March 2002


FantasyA’s graphics engine was designed to be flexible and easily extensible and, at the same time, fast to allow the game to maintain a good frame rate.

It has a Control Module, which is the core of the engine, that is responsible for the management of all the other modules. Two of these modules have special functions – the Input Module and the Scene Module – which will be described later.

Figure 3.21: FantasyA’s engine

Additional modules expand the engine so that it supports other features such as synthetic characters’ bodies (Affective Body Animation Module) or particle systems (Particle Systems Module). These modules share the use of the Input Module and the Scene Module.

3.3.1.1 The Control Module

The Control Module is the core of the engine and manages all other modules. Every module that is to be integrated in the system must register itself in the Control Module so that is can be queried and updated during the engine cycle. This module initializes all the modules that are registered and calls the update routine of the modules in each engine cycle.

Finally, the Control Module manages the interface sharing between the modules allowing them to communicate with each other.

3.3.1.2 The Input Module

The Input Module is a configurable module with a simple interface that manages the communication between the input devices and the other engine modules.

It was implemented in C++ using Microsoft’s DirectInput API (version 8.1). This API allows great flexibility accessing several types of input devices and the latest technologies.

Due to this fact, the Input Module can use all the input devices supported by the referred API, such as keyboard, mouse and joystick. It is also possible to retrieve data in several ways: Immediate Data, Buffered Data and Action Mapping.

IST-1999-11683:D6.1 31 March 2002


3.3.1.3 Immediate Data

Obtaining data in immediate mode can be seen as taking a picture of the device’s state, i.e. knowing which keyboard keys are pressed or the mouse’s axis coordinates, in a certain moment.

3.3.1.4 Buffered Data

Retrieving data in buffered mode means specifying a data buffer where every change of state of each device object is placed. This means that what is stored is not the state of every key in the keyboard at a given instant, but all the changes in the state of each key since the buffer was last accessed.

3.3.1.5 Action Mapping

Action mapping means binding actions with device objects (i.e. keys, buttons and axis) so that the application retrieves the data from all the input devices simultaneously and transparently. That is, the application is informed of the actions taken and not the buttons that were pressed or the axis that were moved.

Action mapping can be used in immediate or buffered mode. The difference between them is that, in immediate mode, only active actions in a given moment are reported, and in buffered mode, all the actions activated since the buffer was last checked are retrieved.

3.3.1.6 The Scene Module

The Scene Module, together with the Input Module and the Control Module, forms the heart of the engine.

This module’s objective is to store the world’s objects in a hierarchical structure and to draw the scene in each engine cycle. Its interface allows the other modules to retrieve information about the world objects as well as how to change their characteristics.

3.3.1.7 Hierarchical Object Structure

The hierarchical structure is made out of generic objects so that they can all be managed the same way, whether they represent a character, a tree or a light source. This structure is implemented as an ordered list sorted by the object’s unique identification number, granting simplicity and access speed.

Each module tha t manages one or more world objects must begin its execution by registering them in the Scene Module so that they can be placed in the scene management structure. During each cycle, the object’s module can change the object’s properties by accessing the Scene Module’s interface.

IST-1999-11683:D6.1 31 March 2002


3.3.1.8 Drawing the Scene

The scene is drawn when the Scene Module calls the drawing method of each object in the visible area. This way it maintains its independence towards the graphics API.

3.3.2 The core of the Game Engine

Together with the graphics engine, we had to develop a game engine (sitting on top of the graphics one) that would control the game situations and include the SenToy interface. The architecture of the whole system, with the integration of the graphics engine and the game engine looks

Note that the player can control the character through the SenToy interface by making it move in a particular way (by detecting actions such as walk, pick up an object, stop, send a magic spell) and emotions. An initial study of the gestures to be detected was done and previously reported in D3.1 and D7.2.

The game engine is responsible for keeping the state of the game and provide the right requests to the modules of the graphics engine.

IST-1999-11683:D6.1 31 March 2002


Figure 3.22 The general architecture of FantasyA

3.4 The Components from the Safira Toolkit Used

So far we are only developing a small prototype of this demonstrator that will serve to illustrate the components of the toolkit. The components used so far are:

- The SenToy (developed under WP3)

- The affective User Modelling Component (developed under Wp3)

- The body expression component (developed in WP5).

So far we haven’t yet integrated the appraisal compiler (WP4) but we hope to do that in the future.

IST-1999-11683:D6.1 31 March 2002


IST-1999-11683:D6.1 31 March 2002


4. The Influencing Machine

4.1 CONCEPT

Two people enter a small room. Child- like scribbling appears across a wall: jagged lines, circles, spirals, and other shapes build up, overlap, fade away. Scattered throughout the room are postcards with art prints; on a table stands a wooden mailbox. One person picks up a card and tentatively puts it in the box. Unusual and musical sounds begin to play. Drawings change speed, color, pressure, form. The people begin sorting through cards, dropping them in the box and seeing how the graphics and sound change. Over time, new forms appear. The people play, experiment, discuss: “How is this reacting to us?” “How do you think this works?”

Figure 4.1: Setup of the Influencing Machine

4.1.1 Motivation

Affective computation, or the development of computational systems which can be aware of and respond to human emotions, has become the focus of a great deal of attention in the Artificial Intelligence (AI) community. These approaches to affective computation often focus on what one might call an informatics of affect, in which emotions are treated as units of information. Emotions are analysed, classified, discretised, and formulated as units whose purpose is to inform cognition or be communicated. The often-used integrative cognitive theory of emotion of Ortony, Collins and Clore, for example, defines emotions in terms of a set of discrete, basic types and focuses on the cognitions or reasoning which may give rise to them [Ortony et al. 1998]. Once a set of emotional units is defined, input devices can be designed which can turn physiological responses into information. For example, Fernandez, Scheirer, and Picard’s Expression Glasses measure the movement of facial muscles and classify

IST-1999-11683:D6.1 31 March 2002


the resulting expression into a small, discrete set of emotions [Fernandez et al. 1999] Ark’s Emotion Mouse extends a normal computer mouse to deduce users’ emotional states from physiological information such as pulse and galvanic skin response[Ark et al. 1999].

Frequently in this tradition, emotions are subsumed to rationality or effectiveness. Damasio’s influential arguments for the importance of emotion in scientific research, for example, gain currency from the idea that emotion is necessary for true rational behaviour [Damasio 1994]. Similarly, Picard’s ground-breaking work on Affective Computing argues that computers must be able to process emotion in order to function maximally effectively with human beings [Picard 1997].

While defining, classifying, creating logical structure for, and understanding the relationship of rationality to emotions can be useful exercises, bringing with them the pleasures of both computability and scientific respectability, we believe this mindset is in danger of missing a fundamental point: affect is not just a formal, computational construct, but also a human, rich, enigmatic, complex, and ill-defined experience. Rationalizing it may be necessary to make it computable, but an affective computation that truly inspires and incorporates human emotion must include a broader cultural perspective, in which the elusive and nonrational character of emotion does not need to be explained away. From this perspective, computation may be used, not to acquire and reason about user’s emotional states, but rather to create intuitive experiences of affect by the user during interaction. We term this alternative approach to affective computation an enigmatics of affect.

This approach does not, however, liberate us from the constraints of computability. In order to function, technology requires us to discretize, analyse, and formalise. A substantial design challenge in constructing a technical system that supports enigmatic, open interpretation, then, is the need to bridge the rational objectivity of the soft- and hardware with the interpretational complexity of users’ subjective experiences. Doing this well requires combining technology design in the tradition of AI research with cultural analysis of the meaning and metaphors embedded in the interaction, in the spirit of a critical technical practice [Agre 1997].

4.1.2 Description

The Influencing Machine is an interactive installation which explores issues in the enigmatics of affect. In this installation, users influence the emotions of an (invisible) artificial agent, which expresses its emotions by generating real-time, dynamic, child-like scribblings and through an emotionally evocative soundscape. For this system, the underlying technology that the system requires to process emotions and generate expressive behaviour is in the background, with the focus on users having experiences with the system that engage their emotions and critical and interpretive faculties. Users should ask, “Which emotions do I see? Do I agree with the machine? Do I believe these drawings express emotions? What is my relationship to this emotional machine?” In

IST-1999-11683:D6.1 31 March 2002


order to support this questioning, the input and output of the system are deliberately left complex, enigmatic and open to interpretation.

The Influencing Machine functions as a boundary object, floating between the worlds of algorithms and interpretation, between technology design and cultural analysis. It forms itself a cultural commentary on the nature of emotion in the culture of computation, allowing users to take part by experimenting with and reflecting upon their experiences with the machine.

Technically, the system works by using the input postcards to influence an internal emotional model. These internal emotions trigger sounds and the selection of drawing behaviours and their dynamic parameters: speed, colour, size, pressure, etc.

While this technical description is precise and clean, the emotional interpretation of the graphical output and postcards by users is complex, incompletely specifiable, open-ended, and strongly culturally influenced. Rather than forcing users to interact with the system using its internal, clean, unnatural emotional representations, in the design of the Influencing Machine we wanted to enhance users’ intuitive, interpretive experience of the system. The basic problem for this design is the disjunction between the internal representation of discrete emotions that drive graphics and sound and the desired complex, open-ended interpretations of the user

Emotional Model

Graphics

Input

Sound

Figure 4.2: The Influencing Machine technology is organized into layers of cultural complexification, surrounding the core emotional model.

Conceptually, the system is organized into a set of concentric layers as shown in Figure 2, which start out as highly formal, clean and technical, and move out to incorporate more and more complex and ill-defined aspects of user experience. The central layer, the emotional model, forms the rationalized core of the system. To this we add first algorithms for graphical output, which are both technically substantial and begin to evoke emotional interpretation in complex, subtle, and multivalent ways. The design of

IST-1999-11683:D6.1 31 March 2002


the input system moves out further, being based on user interpretation of emotion and bringing this into relationship to the internal emotional model and thereby to the graphics. Finally, we add an intermediate layer of sound, which mediates and further expresses the relationships between the postcard input and the complex output.

4.2 COMPONENTS USED

The Influencing Machine system uses the following components of the SAFIRA toolkit: an affective input object; the developmental model; and affective graphics. Here, we describe how each of the components is used in the demonstrator.

4.2.1 Affective Input

The graphical engine of the Influencing Machine creates dynamic imagery that invites interpretation. This imagery is, however, driven by the internal emotional model, rather than by the emotional interpretation of the user. In order to connect the graphics more directly to the complexities of user experience, we need to use an appropriate input device.

The input device for the Influencing Machine must allow users to influence the behaviour of the system through its internal emotional values while enhancing users’ interpretational experience of the machine and enabling critical engagement. A typical affective computation interface might try to measure and represent users’ unconscious emotional state; this, however, would not support critical engagement, since users may not be aware of how they are interpreted. It would also make multi-user engagement, which supports critical engagement by providing the opportunity for conversation, difficult. On the other hand, a screen interface with sliders would give users precise control of internal emotional values. But this, again, does not enhance the experience of the process: it instead forces users to interact with the system using its internal representation, not using the concepts the user is likely to come up with spontaneously in observing the system’s output.

An additional complication is raised by the unclear ontological nature of the user’s relationship to an emotional machine. Culturally, the notion of an emotional machine is paradoxical; an emotional being is something we relate to, care for, and nurture, while a machine is something we instrumentally use and control. How should humans relate to emotional machines? How should emotional machines in turn understand their relationships to humans?

One answer to these questions inspired by the metaphor of the influencing machine, a paranoid delusion first described by Victor Tausk [Tausk 1992] and extensively described by Bruno Bettelheim in his case study of Joey, a boy who believed he was mechanical [Bettelheim 1972]. People suffering from the influencing machine delusion feel that they are being controlled by a machine that projects hallucinations, produces or removes thoughts, feelings, and physical sensations, and changes one's bodily

IST-1999-11683:D6.1 31 March 2002


composition. This influencing machine is a projection of part of a person's sense of self; i.e. his or her sense of self is split into two parts that share control of the person.

In this way, the influencing machine is a model of what it is like to share control between two entities, a concept that maps to the user-emotional agent relationship. It is easy to imagine that an emotional machine could experience the user as a kind of influencing machine, who is not directly accessible to the machine but causes changes to the machine's actions, thoughts, and emotions. In the Influencing Machine system, the user should not directly control but indirectly influence the system’s behaviour, acting as the influencing machine for the drawing agent.

4.2.2 Design

These observations lead to the following design constraints for input. Input, like perceived output, should be fuzzy, supporting open-ended engagement with the system without imposing a particular, limited set of emotional interpretations. It must allow this rich, analogue interaction while supporting discrete internal processing. The input device should be easy to understand and enjoyable. It should support multiple simultaneous users, who can share their interpretations, reflections, and criticisms. Users must be able to influence, but not control, the agent - and to know they can and how.

To fulfil these requirements, we use a tangible input device, the Mailbox of WP3, which we use with open-ended, imprecise representations. When users enter the Influencing Machine installation, they find physical postcards with emotionally evocative art prints scattered around the room. On a table stands a wooden “mailbox” or “voting box”, into which users can put the postcards, thereby sending emotional messages to the agent. These messages alter the current emotional state of the agent, which in turn alters the soundscape and the form and dynamics of the graphics that are being seen by users.

From a technical perspective, each postcard represents an emotional profile, or a set of emotional values. This profile is designed by hand to match the emotions that the image on the postcard may evoke. It is important to note that input emotions do not directly set the agent’s emotions, but are added to or subtracted from the agent’s current emotional state. Thus, a highly introverted agent cannot be turned into an extraverted agent with one postcard; instead, extraverted postcards must be used multiple times, with the agent’s emotional state gradually becoming more extraverted with each card.

From a human perspective, users pick up postcards and interpret the emotions they contain. By sending a card, users send their personal, emotional reactions to the system, hoping to evoke the same emotions in the agent, or maybe with only a vague idea of the link between card and response. Observing the resulting drawings and sounds, users consider the ir opinion of the card, comparing it to and often commenting on their interpretation of the agent’s behaviour.

IST-1999-11683:D6.1 31 March 2002


The tangibility of the interface brings the experience outside of the machinic and into the physical world of users, who use the device together. It looks whimsical and approachable. Appropriately chosen art postcards cry out for emotional interpretation; they call on users’ natural intuition. Concerned about the potential arbitrariness of interpretation of art prints, we also developed a set of input postcards with one to three solid colors, based on Eva Heller’s empirical theory of the emotional meaning of colors [Heller 1989]. By using sets of emotions with different levels for each card, we avoid a push-button, one-to-one interface, and have the option for nearly arbitrarily many different input objects with different effects. Mapping cards to sets of emotion values is a straightforward way to make the emotional complexity of the postcard imagery computable.

4.2.3 Developmental Model

The goal of the Influencing Machine is to encourage thoughtful interaction and interpretation of emotional content by users. One way in which interaction is made central to the system is through the use of development. With interaction, the drawings users see gradually become more complex. Their development is driven by interaction with the user by using the sequence of emotional input given by the user to trigger developmental change.

In order to do this, the Influencing Machine uses the Developmental Model component of WP4. The Developmental Model maintains a domain- independent representation of development that can be used by other components to reason about and incorporate developmental change. The Developmental Model consists of a set of developmental states organized into a graph. Developmental change is understood as movement along the edges of the graph, which are labelled with predicates. This predicate expresses when developmental change from an earlier state to a later one can take place.

The Influencing Machine uses the Developmental Model in the following way. In the Influencing Machine, a “developmental state” corresponds to a particular drawing form which the affective graphics component is capable of rendering (see Figure 3). The developmental model as used by the Influencing Machine organizes these developmental states into a graph, in which forms are connected developmentally in ways that approximate the ways in which such development is depicted in the literature on children’s drawings (see e.g. [Strauss 1994] [Richter 1997]). Transitions between development states are controlled by predicates on the emotional state, which reflect the ways in which the emotional meaning of that change in state is discussed in the literature. For example, the system transitions from being able to draw a line in a random direction to being able to draw a horizontal or vertical line when the control emotion has reached a level more than 3. It is able to transition from drawing horizontal or vertical lines to being able to draw a ladder when the control emotion is more than 5. In this way, by influencing the emotions of the drawing agent, the user gradually sees more and more complex graphical behavior in the system output.

IST-1999-11683:D6.1 31 March 2002


Figure 4.3: Some of the available forms: circular scribbles, line scribbles, crosses, centred circles, ladders.

Figure 4.4: A developmental state or form may have many different graphical instantiations. These are all line scribbles. The variety in appearance is implemented in the Affective Graphics system.

4.2.4 Affective Graphics

The emotional model internal to the system (see Section 1.3.1 below) is a purely formal, informational construct. The system graphics must begin to translate that clean model into visible forms and dynamics that evoke more open-ended emotional meaning for users. The graphical output of the system should hint at a subjective view of the world. It should be enigmatic but invite interpretation. The fundamental design issue in the Influencing Machine is the need to mediate between the internal, simple emotional model and the complex interpretations of the user. For the graphics of the system, this means the drawings should not be a simple, one-to-one mapping of internal emotion to an observable graphical token (e.g. a 3- inch red triangle that appears whenever the agent is angry). Instead, graphics should be dynamic, be newly generated in real time, and evoke complex and open-ended interpretations of multiple simultaneous emotional states.

In the Influencing Machine, we use the Affective Graphics component of the SAFIRA toolkit in order to generate dynamic, emotionally evocative imagery. The values of the emotional and developmental models drive the choices and execution of drawing behavior. Through its variety of emotionally-driven graphical output, the Affective Graphics component invites absorption, reflection, and emotional interpretation.

IST-1999-11683:D6.1 31 March 2002


4.3 ARCHITECTURE

In addition to the toolkit components, we need several additional components that are specific to the demonstrator. Here, we will first describe the additional components, and then explain how all the components are combined in order to generate the Influencing Machine.

4.3.1 Emotional Model

The internal emotional model is the cornerstone of the system; the other modules function by referring to and changing the emotional model. Internally, emotions are represented using a discrete set of 22 emotions, which are organized as 11 pairs of opposite emotions: anger / peace, transcendent / physical, insecure / assured, primal unity with the world / sense of self, control-stasis / flow-dynamics, warm / cold, happy / sad, aggressive / passive, introverted / extraverted, safe / fearful, energetic / weak. We chose this set of emotions by analysing the emotional terms used in the depth-psychological analysis of children’s drawings [Strauss 1994] [Richter 1997], which corresponds to the sorts of emotions we can express through the graphical output of the machine.

These emotions did not correspond well to commonly available cognitive models of emotion, and we were forced to develop our own model. Our focus is not on developing a complex theory of and structure for the internal emotions, but on how emotions affect the system’s behaviour and thereby user interpretation. Hence, we implemented the emotional model as a minimum-commitment model in which an emotion simply maintains a value between 1 and 10. Opposite emotions are correlated; i.e., when the agent is happy, it cannot also be sad. Emotional levels decay over time, so that, if left alone, the agent tends to a neutral emotional state.

4.3.2 Sound

By themselves, the emotional evocations of the graphics of the system are complex and multivalent, and hence difficult to interpret. The art postcards, too, are richly evocative – and thereby not clearly and straightforwardly classifiable. There is a danger of the user being, not stimulated to autonomous interpretation, but simply confused by the complexities of the input and output of the system.

Because of this difficulty, we decided to add an extra layer of expression through an evocative soundscape between the input and graphical output of the system. These sounds play in the background, providing additional support for disambiguating the potential emotional implications of the dynamic graphics. They also provide additional feedback for input to the system (answering the question, “did that do anything?”), by responding directly and immediately to the emotional implications of the input postcards. They also add to users’ experience simply by incorporating a new and pleasurable sensory dimension.

IST-1999-11683:D6.1 31 March 2002


The sound system is directly triggered by the emotional model. One set of sound samples illustrates the emotional levels of the 4 strongest emotions. The other set of sound samples give direct feedback when there has been a significant change in emotional levels, i.e. a postcard has been input by the user.

The sound system is implemented with a set of rules that map emotions and changes in emotions to particular sound files. The sound system regularly checks the emotional levels in the emotional model to see if any of the pre-configured conditions for a sound response are met. If so, it executes the appropriate sound control commands.

The sound player contains dedicated channels for each pair of opposite emotions, e.g. anger/peace. Every channel only plays the audio representations related to these emotions. The number of emotional audio representations playing at the same time is restricted to a configurable value.

4.3.3 Combined Architecture

The Influencing Machine demonstrator combines the toolkit components and demonstrator-specific components in the following way, as shown in Figure 5. Conceptually, the Mailbox component is plugged as input into the emotional model. Newly generated vectors describing the emotional content of the postcards are combined with the existing emotional values by adding their values to those that are already existing in the system. Changes in the emotional model are communicated to the Developmental Model and to the Sound system, where they immediately alter the sound output of the system. The Developmental Model updates the set of current developmental states. The emotional and resulting developmental changes are communicated to the Affective Graphics system, which continues to draw in a way expressive of the emotional and developmental state of the system.

Mailbox

Emotional Model

Sound

Affective Graphics

Developmental Model

Figure 4.5: Control Flow in the Influencing Machine

The architecture is implemented in a round-robin fashion. All components run in a single thread, and are compiled together as Java code. At each tick of the system, each

IST-1999-11683:D6.1 31 March 2002


of the components runs in turn. The interaction of the components is handled using a centralized loop. In this loop, the following steps happen:

1. Allow stroke component of Affective Graphics to execute current drawing commands.

2. Query Mailbox for new incoming emotional information.

3. Update the emotional model.

4. Generate new sounds based on the updated emotions, if necessary.

5. Update the developmental model based on the updated emotions, if necessary.

6. Run behavioural component of Affective Graphics with the new emotional and developmental information, potentially generating new drawing commands.

Communication between components is simulated using queues. Component output is placed onto queues, where it can be popped and read by other components.

IST-1999-11683:D6.1 31 March 2002


IST-1999-11683:D6.1 31 March 2002


5. Plan for the next six months

During the next six months we will be finalising the demonstrators and produce a “lessons learned” in the form of Framework for affective interactions.

5.1 James the Butler

The target for providing the first functioning prototype of the James the Butler demonstrator is the 1st of June 2002 when it will be evaluated.

The internal functionality of James role as a personal sales assistant is currently complete. James’s complete integration with the web-based interfaces is planned for the 30th of May 2002. Integration requirements for including the affective speech utterance mechanisms of the ASM have been defined and the APIs agreed upon. The actual integration will start on the 15th of April when the module is fully complete. A fully integrated James with the Toolkit components required and possibly the Appraisal component is planned for the 15th of May 2002.

The main functionality of the Winestore domain specific modules will be finished by the 15th of April 2002. Integration between these domain specific modules and included components through CRS will be ready by the 1st of May 2002

5.2 FantasyA

We aim at providing the functioning prototype of FantasyA that includes mainly the Arena, the emotional duels and the integration of SenToy, in the beginning of June. This prototype will be evaluated during June and July and part of August. Any changes in the prototype arising from the evaluation will be done during August and September.

5.3 The Influencing Machine

The construction of the Influencing Machine has been completed and evaluated. Our work in the next 6 months concentrates on demonstrating the Influencing Machine and disseminating its results. The Influencing Machine demonstrator is shown regularly to visitors at the Fraunhofer Institut fuer Medienkommunikation. It will be presented at the conference on Designing Interactive Systems in June, 2002.

IST-1999-11683:D6.1 31 March 2002


IST-1999-11683:D6.1 31 March 2002


6. References

[Agre 1997] Agre, P. Computation and Human Experience. Cambridge UP, Cambridge, 1997.

[Arafa, et. al 1999]. Arafa, Y., Dionisi, D., Fehin, P., Martin, S., Pitt, J., Witkowski, M. MAPPA-D3 Agent to Human Interaction, 1999.

[Ark et al. 1999] Ark, W., Dryer, D., and Lu, D. The emotion mouse. Proceedings of HCI International 1999. Munich, Germany (August 1999).

[Bates, 1994] Bates, Joseph. The Role of Emotion in Believable Agents. Communications of the ACM 37,7, 122-125, 1994.

[Bettelheim 1972] Bettelheim, B. The Empty Fortress: Infantile Autism and the Birth of the Self. Free Press, New York, 1972.

[deBruijn & Spence, 2000] deBruijn, O. & Spence, R. (2000) Rapid Serial Visual Presentation: A space-time trade-off in information presentation, submitted to AVI'2000

[Damasio 1994] Damasio, A. Descartes’ Error: Emotion, Reason, and the Human Brain. Gosset Putnam, New York, 1994.

[Fernandez et al. 1999] Fernandez, R., Scheirer, J. and Picard, R. Expression glasses: a wearable device for facial expression recognition. MIT Media Lab Tech. Rep. 484, 1999.

[Field, 1994] Field, S. 1994. Screenplay: The Foundations of Screenwriting. New York: Bantam Doubleday Dell.

[Heller 1989] Heller, E. Wie Farben Wirken: Farbpsychologie, Farbsymbolik, Kreative Farbgestaltung, Rowohlt Verlag, Reibek bei Hamburg, 1989.

[Lazarus, 1991] Lazarus, R. Emotion and Adaptation, Oxford University Press, 1991.

[Ortony et al. 1988] Ortony, A., Clore, G.L. and Collins, A. The cognitive structure of emotions, Cambridge University Press, N.Y, 1988.

[Picard 1997] Picard, R. Affective Computing, MIT Press, Cambridge, 1997.

[Pokemon]Pokemon. http://www.pokemon.com

[Richter 1997] Richter, H,.-G. Die Kinderzeichnung: Entwicklung, Interpretation, Ästhetik. Cornelsen, Berlin 1997.

[Reeves & Nass, 1996] Reeves, B. & Nass, C. (1996) The Media Equation. CSLI Publications, Cambridge University Press.

IST-1999-11683:D6.1 31 March 2002


[Spence, 1999] Spence, R. (1999) Int. J. of Human Computer Studies., Vol 51.

[Strauss 1994] Strauss, M. Von der Zeichensprache des kleinen Kindes: Spuren der Menschwerdung. Verlag Freies Geistesleben, Stuttgart, 1994.

[Tausk 1992] Tausk, V. The influencing machine, in Incorporations, ed. J. Crary and S. Kwinter. Zone Books, New York 1992.

[Thomas & Johnson, 1981] Thomas, F., and Johnson, O. 1981. The Illusion of Life: Disney Animation. New York: Norton.

[Vargas, M] Vargas, M. About Retail Loyalty. http://retailindustry.about.com

[Wooldridge & Jennings, 1995] Wooldridge, M. and Jennings, N. (eds.) 1995. Intelligent Agents: Theory and Practice. The Knowledge Engineering Review 10(2):115-152, 1995.

Documents

Deliverable 6 - Gaips at Inesc-Idgaips.inesc-id.pt/safira/publications/D-SAFIRA-WP6-D6.1.pdf · The SAFIRA application uses profile ownership, Quality of Service adaptation to individualised