ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

Neurocomputing 0 0 0 (2017) 1–13

Contents lists available at ScienceDirect

Neurocomputing

journal homepage: www.elsevier.com/locate/neucom

Personality affected robotic emotional model with associative memory

for human-robot interaction

Naoki Masuyama

a , Chu Kiong Loo

a , ∗, Manjeevan Seera

b

a Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia b Faculty of Engineering, Computing and Science, Swinburne University of Technology (Sarawak Campus), Malaysia

a r t i c l e i n f o

Article history:

Received 19 April 2016

Revised 16 September 2016

Accepted 28 June 2017

Available online xxx

Communicated by Bo Shen

Keywords:

Associative memory

Emotional model

Human-robot interaction

Personality

a b s t r a c t

The decision making process in communication is affected by internal and external factors from dynamic

environments. Humans can perform a variety of behaviors in a similar situation, unlike robots. This paper

discusses human psychological phenomena during communication from the point of view of internal and

external factors, such as perception, memory, and emotional information. Based on these, we introduce

the personality affected robotic emotional model and the emotion affected associative memory model for

the robot. We organize an interactive robot system to provide suitable decisions for the robot. Results

from interactive communication experiments indicate that the robot is able to perform different actions

based on internal and external factors.

© 2017 Elsevier B.V. All rights reserved.

1

d

v

v

o

p

k

t

[

r

i

w

h

e

c

p

t

t

t

s

a

a

s

h

l

e

a

w

c

o

m

f

o

r

m

d

s

r

t

d

p

c

s

p

h

0

. Introduction

Communication is a fundamental action for humans. In past

ecades, researchers in the field of psychology have tried to re-

eal human psychological functions such as neuropsychology, de-

elopmental psychology, and cognitive psychology [1,2] . On the

ther hand, computer scientists have attempted to establish human

sychological functions on the computer, based on psychological

nowledge. Specifically, in order to acquire the human functions,

he research in intelligence as neural networks and fuzzy systems

3] , and cognition and perception as image processing and voice

ecognition have been developed [4] .

Facial and gesture expressions, being elements of multi-modal

nformation depict a significant role to focus attention on subjects,

hile sharing the cognitive environment during during human-

uman interaction. In general, capability in sharing the cognitive

nvironment with others is vital for a smooth communication. The

ontinuity and relevance of subjects are among other factors in ex-

anding the cognitive environment [5] . In the human-human in-

eraction, associative memory is a functional and vital brain func-

ion in handling continuity and relevance of the subject. To mimic

his effective brain function, several types of artificial neural as-

ociative memories with improvements have been introduced, and

nalyzed mathematically for the memory capacity, noise tolerance

∗ Corresponding author.

E-mail address: ckloo.um@um.edu.my (C.K. Loo).

c

d

c

a

ttp://dx.doi.org/10.1016/j.neucom.2017.06.069

925-2312/© 2017 Elsevier B.V. All rights reserved.

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

nd stability of network [6] . These models however are not con-

idered to influence of other functions of human brain. In human-

uman communication, the decision making process affects the

ogical thinking factors and also the emotional factors [7] . This

motional effect is one of the key differences between a human

nd a robot.

The significance of interaction between emotion and memory,

ith its mutual relationships is discussed in [8] . In general, in

ommunication, the recalled information might change depending

n emotional effects. This psychological phenomena is called

ood-congruency effect [9] . It is presumed that the emotional

actors perform significant roles in human-robot interaction. In

ther words, we assume that the robot can provide suitable

eactions to situations, if the robot performs multi-modal com-

unication based on emotion affected associative memory. In

iscussing humanity from the psychology point of view, one of the

ignificant elements is personality. The concept of personality is

egarded as one of the essential factors of emotional response in

he psychological field. In other words, personality gives individual

ifferences among people in behavior patterns and cognitive

rocess [10] . Based on differences in age and gender of human, it

an be regarded that the impact and reaction based on external

timulus have diversity [11] . Sensibility of person is normally

redicted based on their appearance, such as an elderly person is

alm and a young female is more emotional. We assume that it

epends on the communication partner, where the person would

hange the his/her behaviors unconsciously, based on the partner’s

ppearance to perform the appropriate responses as a kind of

otic emotional model with associative memory for human-robot

2017.0 6.0 69

2 N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

b

p

K

c

o

p

i

a

o

T

t

i

n

s

e

s

m

c

t

2

f

s

s

t

A

C

a

p

h

v

p

[

n

c

t

T

i

e

p

i

a

s

P

e

e

b

c

s

d

e

C

i

e

n

2

b

T

social interaction. Due to the settings of personality factors, it

is possible to arrange the unique and suitable robot to specific

conditions. This results in a more active communication between

the robot and the human would be more active.

If the robot only takes into account the external stimulus, the

reactions from the robot are always the same, corresponding to

the stimulus. In contrast, considering the internal state (emotional

state in this manuscript) as another type of stimulus, the robot is

able to perform the different reactions based on history of stim-

ulus. Due to the personality factors in the emotional model, we

can easily arrange the individual/unique robot. From the viewpoint

of natural and smooth communication, several studies [12,13] have

shown that the emotional information during communication per-

forms significant roles, such as behavior selection, which makes

the robot more attractive and changes the meaning of the cognitive

information.

In this paper, we propose a human-robot interaction system

with associative memory that is dominated by personality affected

robotic emotional model. The proposed system is able to handle

several modalities, in order to communicate with humans. In ad-

dition, based on multi-modal inputs, the robotic emotional infor-

mation will be generated, and recalled information in associative

memory will be changed based on internal states in the robot. We

regard this is one of the functional implementations of associative

memory from the point of view of the psychology.

The contribution of this paper is the development of a robotic

decision making system, based on human psychological phenom-

ena during communication. We take into account internal and ex-

ternal factors, such as perception, memory, and emotional informa-

tion. The proposed system is able to assign the preferable person-

ality to the robot, based on the estimated human biometric infor-

mation. These personality factors make significant differences for

the decision making process for the robot, which is defined by the

emotion affected associative memory which is characterized by the

mood congruency effect. As a result, the proposed system is able to

provide the human-like behaviors for the robot during interactions

with humans.

This paper is organized as follows. A literature review on asso-

ciative memory models, emotional models, and its applications are

first presented in Section 2 . Section 3 discusses the relationships

between memory and emotion in the human brain from the psy-

chology and brain science viewpoints. Section 4 presents the com-

putational models of memory and emotion. In Section 5 , configu-

ration of proposed interactive robot system and its cognitive intel-

ligence are detailed. Experimental results under several conditions

are discussed in Section 6 . Concluding remarks are finally given in

Section 7 .

2. Literature review

In this section, we present representative models of artificial

neural associative memory models and emotional models for hu-

manoids. In addition, we introduce the several interactive robot

systems which integrate memory and emotion functions.

2.1. Artificial neural associative memories

Fundamental structures of associative memory such as Hopfield

Associative Memory (HAM), Bi-/Multi-directional Associative Mem-

ory (BAM, MAM) are introduced by Hopfield [14] , Kosko [15] , and

Hagiwara [16] , respectively. Based on the complex-valued artificial

neuron model [17] , Complex-Valued HAM (CHAM), BAM (CBAM)

and MAM (CMAM) are introduced by Jankowski [18] , Donq-Liang

[19] , and Kobayashi [20] , respectively. These fundamental mod-

els however suffered from low memory capacity and recall re-

liability. To improve the abilities of model, several studies have

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

een introduced. Lee et al. [21] showed that the projection matrix

roposed by Personnaz can be generalized to a complex domain.

obayashi [22] applied a pseudo-relaxation learning algorithm. The

omplex-valued chaotic behavior models are also considered based

n real-valued models [23,24] . Though these models show the su-

erior abilities, the complexity of the model structures are greatly

ncreased.

In another approach, the concept of quantum mechanics is

pplied to artificial neural networks [25] . Based on the features

f quantum mechanics as parallelism and unitarity, Rigatos and

zafestas [26] proposed the Quantum-Inspired Hopfield Associa-

ive Memory model (QHAM). This model demonstrates quantum

nformation processing in neural structures results in an expo-

ential increase in storage capacity, and can explain the exten-

ive memory and inferencing capabilities of humans. It is how-

ver limited to auto-association and binary state processing. Ma-

uyama and Loo have solved the problems of QHAM by introducing

ultidirectional architecture and complex-valued neuron model,

alled Quantum-Inspired Complex-Valued Multidirectional Associa-

ive Memory (QCMAM) [27] .

.2. Emotional models

Emotion is a complex human function which can be discussed

rom physiological, cognitive, and motivational processes. Several

tudies of emotional phenomena in psychological field operate as-

uming a fixed number of emotions [28] , the regions or proto-

ypical trajectories based on the results from human studies [29] .

simplified method of the emotional system, Ortony, Clore, and

ollins (OCC) [30] introduced the OCC model to identify emotional

ttribution of events, object desirability and praiseworthiness.

In terms of emotion, personality is an important factors. In the

ast, several psychologists have discussed on relationships between

uman emotional factor and personality factor [31,32] . From the

iew point of behaviors [33] , various rule-based models [34] and

robabilistic models [35] have been introduced. Costa and McCrae

36] introduced the OCEAN model based on five factors, i.e. open-

ess, conscientiousness, extraversion, agreeableness, and neuroti-

ism. Mehrabian utilized the five factors of personality to represent

he Pleasant-Arousal-Dominance (PAD) temperament model [37] .

he relationship between five factors of personality and PAD model

s derived through the linear regression analysis [38] .

From studies in psychological fields, several computational

motion models have been introduced [39] . Han et al. [40] em-

loyed five factors of personality to a 2D (pleasure-arousal) scal-

ng model, introduced by Russell and Bullock [41] to represent

robotic emotional model. This model generates a robot mood

tate from the human facial expression information. Smith and

etty [42] described that human beings have tendency to recall

motions from information based on their knowledge and experi-

nce. As mentioned in Section 1 , human emotional information and

ehaviors are affected stimulus from the environment and the

ommunication partner. Naveh et al. [43] studied the effects to as-

ociation process from not only emotional factor, but also the gen-

er and age factors. Rosenthal et al. [44] studied the differences of

motional reactions towards a robot based on specific situations.

hen et al. [45] developed the model that is able to understand the

ntention in human-robot interaction, which is mainly obtained by

motion, with identification information such as age, gender, and

ationality.

.3. Interaction robot systems

It is considered that associative memory function in the human

rain plays a significant role in the human-human communication.

he association process is affected by internal and external factors,

otic emotional model with associative memory for human-robot

2017.0 6.0 69

N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13 3

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

s

i

s

h

t

y

c

b

r

a

r

t

a

c

[

m

a

a

e

c

m

e

e

t

i

t

t

I

o

w

t

w

t

p

3

h

t

i

3

c

u

n

e

m

j

i

t

i

i

t

f

[

t

t

f

i

e

o

s

t

t

c

s

t

K

o

n

[

t

t

s

m

h

a

s

a

t

p

3

t

b

t

t

c

m

e

o

s

s

b

a

m

e

e

(

I

B

s

t

m

t

c

o

m

t

t

B

3

m

w

a

t

s

uch as emotion [8] . Conventionally, several types of robot system,

n which the emotion affected associative memory selects the

uitable robot behavior have been introduced to improve the

uman-robot interaction. Hiolle et al. [46] developed the system

o elicit care-giving behavior in human-robot interaction with

ounger/adults people using associative memory model. The robot

ollects the multi-modal information to associate the suitable

ehavior for the situations. Rumbell et al. [47] discussed and

eviewed about emotional mechanisms that are often used in

rtificial agents as a method of improving action selection.

Itoh et al. [48] developed an emotion expression humanoid

obot and its interactive system. The system is able to associate

he emotional expression for robot from human behavior using

chaotic complex-valued associative memory that the output is

ontrolled by the mental model and the robot personality. Yi et al.

49] regarded that associative memory is essential to realize man-

achine cooperation in the natural interaction between human

nd robot. They developed the emotional robot platform with the

ssociative memory model which is controlled by the dynamic

motional states. Valverde et al. [50] studied and proposed the

omputation model for associative memory and emotional infor-

ation, using ideas inspired by neuroscience research into neural-

ndocrine systems interaction. The model is able to create the

motional memory of the robot. The emotional memory is utilized

o predict future emotional states based on past experiences, and

t would be affected to select the emotional response actions by

he robot.

In previous work, we have developed an interactive robot sys-

em which is based on QCMAM and discrete emotion model [51] .

n [51] , we need to predefine a number of sensitive parameters, in

rder to define the individuality of robot. However, in this study,

e introduce the universal emotion model with personality factors

o the robot system, which is detailed in Section 4.2 . Furthermore,

e integrate the biometric information for improving the ability

o understand the environment of interaction space. The details of

roposed interactive robot system are presented in Section 5 .

. Memory and emotion in brain

In this section, the effects of emotion to memory function in

uman brain from the psychological view are introduced. In addi-

ion, the functions of emotion affected associative memory model

s discussed.

.1. Associative memory with emotional effect

The feature of the research on emotion and memory in the psy-

hological field not only considers emotional valence of the stim-

lus, but also on the emotional state of the subject itself, simulta-

eously. The pioneering research by Bower [52] demonstrates the

motional state influences to the human memory. In the experi-

ent, subjects are artificially set to have Happy or Sad state. Sub-

ects then tried to memorize the specific words, associate the event

n the past, and remember sentences they wrote in a diary. From

he series of experiments, it is revealed that we are feeling a pos-

tive emotion, we are likely to recall positive information depend-

ng on individual knowledge and experience. Likewise, we have

he tendency to recall the negative information during negative

eelings.

The relationships between memory and emotion presented in

8] provide noteworthy perspective to associative memory func-

ion. In short, the memory information can be recalled using emo-

ional information. Likewise, emotional information can be recalled

rom memory information. This kind of psychological phenomena

s known as the mood-congruency effect [9] .

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

In the past, several studies have shown the influence and prop-

rties of a mood-congruency effect in the human activities based

n the analysis of the brain signal processing and several human

tudies. Lewis et al. [53] assumed that the mood congruent facili-

ation is due to the mood-related reactivation at retrieval of emo-

ional responses which are linked to valenced information at en-

oding in the associative memory. In the experiment, they pre-

ented subjects with positive and negative words and manipulated

heir mood while monitoring brain activity by fMRI. Pierce and

ensinger [54] studied the effects of emotional valence and arousal

n associative connection based on human study by using the

egative, positive, and neutral word pairs. Murray and Kensinger

55] considered the relationships of associative memory and emo-

ion that takes into account the factors of age and gender by using

he word pairs, in the case of emotional and non-emotional, re-

pectively. Egidi and Nusbaum [56] utilized visual and sound infor-

ation to study a mood-congruency effect for a discourse compre-

ension based on analysis of EEG signal during experiment. Ravaja

nd Kätsyri [57] revealed the relationships between facial expres-

ion and mood congruency effect based on facial electromyography

nalysis. From these, it can be noted that the impact and effec-

iveness of a mood-congruency effect to the association and recall

rocesses with the multi-modal information in the human brain.

.2. Associative memory based on mood-congruency effect

In regards to memory, Collins and Loftus [58] have insisted

hat the memory can be represented by the topological network

ased on nodes (memory) and connections between nodes (rela-

ionships). From the psychological perspective, the emotional con-

ext and memory predict that mood-related memory facilitation

an be explained as an associative memory effect [52] . Further-

ore, Lewis et al. [53] have reported its validity from the brain sci-

nce perspective.

Based on the studies in psychology and brain science, the mem-

ry can be mapped by memory nodes and its connective relation-

hips as depicted in Fig. 1 (a). In Fig. 1 (a), each information is repre-

ented by node, and relationships between nodes are represented

y edges. Here, the memory nodes, which are relating to Red Ball,

re focused as an example. Taking into account the emotion infor-

ation, the memory can be illustrated as Fig. 1 (b). In general, the

motional attribution of each node is defined by individual knowl-

dge and experience.

From the associative memory viewpoint, the key information

Red Ball) is able to associate the information in connected nodes.

n case of Fig. 1 (a), all nodes which are connected with the Red

all have the same possibility to be associated. Furthermore, con-

idering the emotion information, each node can be labeled to

he positive, negative or neutral attribution based on experience,

emory and feelings of individuals as Fig. 1 (b). In this condition,

he different association possibility can be assigned to each node

orresponding to the emotion information based on the function

f mood-congruency effect. For instance, the human who has the

emory as Fig. 1 (b), and has the positive emotional information,

he nodes of Apple and Fruits are more likely to be associated than

he nodes of Blood and Knife from the key information as the Red

all.

.3. Affinity of emotional information and complex-valued associative

emory model

In psychology, emotional information can be considered as

aveform information [59] , e.g. the emotion state is regarded as

short-term emotional state, and mood state is regarded as a long

erm emotional state, respectively. On the other hand, a number of

tudies of the human brain indicate that the neurons in the brain

otic emotional model with associative memory for human-robot

2017.0 6.0 69

4 N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

Fig. 1. Region for Red Ball in association map; (a) without emotional factors, (b) with emotional factors. (For interpretation of the references to color in this figure legend,

the reader is referred to the web version of this article).

a

H

a

t

i

l

m

X

r

r

o

p=1 j=1 i =1

transmit electrical activity, which is the basis of neural oscillation

[60] . Several studies have shown that the brain activity and emo-

tional information are closely related each other [61] .

As shown in Section 3.1 , the association process with a mood-

congruency effect can be seen from the signals of brain activi-

ties. Thus, it is acceptable to represent the emotional informa-

tion by the state of the neurons. In terms of the artificial neuron

model, the complex-valued model is regarded as one of the oscilla-

tor models, due to the phase information. In general, the complex-

valued model has the rich expressive power than the real-valued

model. Thus, from the point of view of the functions of neuron in

the brain, the complex-valued model, that has the ability to han-

dle the oscillator model, is an appropriate and affinity model as

the neuron in brain than the real-valued model.

Based on these discussions, it is clear that the interactions

between memory and emotion functions provide interesting and

significant functions for the humans. In this paper, we apply

the complex-valued associative memory model and the emotional

model to simulate a mood-congruency effect for an interactive

robot system, which is presented in Section 5 , to improve the hu-

manity and communication ability of the robot. In the system,

the emotional information is handled by the phase information of

the neurons in the complex-valued associative memory model. It

depends on the emotional information, the association process is

controlled similar with a mood-congruency effect.

4. Computational models of memory and emotion

This section presents the computational models of memory and

emotion for the proposed interactive robot system. In terms of as-

sociative memory, we utilize the QCMAM [27] due to its supe-

rior memory capacity and noise tolerance. In regards to emotion

model, we introduce the personality affected universal emotion

model based on PAD architecture, which is detailed in Section 4.2 .

4.1. Fundamentals of quantum-inspired complex-valued

multidirectional associative memory

We utilized a QCMAM to an association module for its su-

perior abilities in terms of the memory capacity and the noise

tolerance. The model shows quantum information processing in

neural structures, which results in an exponential growth in its

storage capacity. It can also describe inferencing capabilities and

extensive memory of humans. The model is applied on fuzzy infer-

ence to weight matrix, determined by one-shot learning as Hebb-

like learning to satisfy parallelism and unitarity [27] .

In this section, details of the Quantum-Inspired Complex-Valued

Multidirectional Associative Memory (QCMAM) is presented. In

general, multidirectional model is regarded as a multiple combi-

nation of bidirectional models. Thus, the following presentations

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

re focused the network between the αth layer and the βth layer.

ere, the model that has L -layers is considered, and the associ-

ted layer is defined as αth layer, and the other layers are referred

o be βth layers (β = 1 , 2 , . . . , L ; α � = β), respectively. The neurons

n each layer continue to be subject to cyclic updates until the

ayer reaches an equilibrium. Let k of the original complex-valued

emory pairs {

X

(k ) (1)

, X

(k ) (2)

, . . . , X

(k ) (L )

}are stored in QCMAM, where

=

[p 1 + jq 1 , p 2 + j q 2 , . . . , p N (l)

+ j q (l)

]( p, q ∈ R, l = 1 , 2 , . . . , L ) , L

epresents the number of layers, N ( l ) denotes the number of neu-

ons in l th layer, and subscript j denotes the imaginary unit. Based

n above conditions, QCMAM is formalized as follows;

• αth layer to βth layers ⎧ ⎪ ⎪ ⎪ ⎨

⎪ ⎪ ⎪ ⎩

S (k ) (β)

=

L ∑

α=1 α � = β

N (β) ∑

j=1

M (α) ∑

i =1

W

∗i j(αβ) x

(k ) i (α)

, (β = 1 , 2 , . . . , L ;β � = α) (1a)

X

(k ) (β)

=φ(

S (k ) (β)

)(1b)

• βth layers to αth layer

⎧ ⎪ ⎪ ⎨

⎪ ⎪ ⎩

U

(k ) (α)

=

L ∑

β=1 β � = α

M (α) ∑

i =1

N (β) ∑

j=1

W i j(αβ) x (k ) j(β)

, (α = 1 , 2 , . . . , L ;α � = β) (2a)

X

(k ) (α)

=φ(U

(k ) (α)

)(2b)

where, S and U denote temporal states of associated patterns

in αth layer and βth layer, respectively. The exponential aster-

isk denotes the conjugate transpose operation. The φ( ·) denotes

an activation function based on a complex unit circle that is de-

picted as Fig. 2 . The activation function for the complex-valued

model is formalized as follows;

φ(Z) =

⎧ ⎪ ⎨

⎪ ⎩

exp ( j2 πn/q ) ,

If ∣∣Arg

{Z

exp ( j2 πn/q )

}∣∣ < π/q and Z � = 0

previous state , If Z = 0

(3)

where, Arg( ·) denotes the phase angle which is taken to range

over (−π, π) . r denotes quantization value on the complex unit

circle, n takes an integer. Assume that z 0 , z 1 , . . . , z r−1 , are r

quantized values. Here, the complex number Z will be defined

as z 1 that is closest to Z .

Here, the weight connections W

∗ and W are as follows;

• αth layer to βth layers

W

∗(αβ) =

1

k

k ∑

N (β) ∑

M (α) ∑

s (p) ∗j(β)

s (p) i (α)

. (4)

otic emotional model with associative memory for human-robot

2017.0 6.0 69

N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13 5

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

Fig. 2. Discrete complex unit circle.

w

k

a

t

l

l

b

t

a

4

t

t

e

i

I

t

a

a

4

m

t

a

a

fl

r

s

T

4

u

D

v

Table 1

Mapping of prototype emotions based on pleasant-arousal-dominance space

[37] .

Factor (Pleasant, arousal, dominance)

Happy (0.81, 0.67, 0.46)

Sad ( −0.63, −0.27, −0.13)

Anger ( −0.51, 0.59, 0.25)

Disgust ( −0.60, 0.35, 0.11)

Surprise (0.40, 0.67, −0.13)

Table 2

Five factors of personality [36] .

Factor Descriptions

Openness Open mindedness, interest in culture.

Conscientiousness Organized, persistent in achieving goals.

Extraversion Preference for and behavior in social situations.

Agreeableness Interactions with others.

Neuroticism Tendency to experience negative thoughts.

a

a

a

e

M

t

i

T

[

r

4

P

T

w

P

P

P

w

(

t

fi

s

a

o

4

f

m

e

a

w

e

P

a

p

m

a

t

o

i

• βth layers to αth layer

W (αβ) =

1

k

k ∑

p=1

M (α) ∑

i =1

N (β) ∑

j=1

s (p) ∗i (α)

s (p) j(β)

. (5)

here, the exponential asterisk denotes transpose operation.

denotes the number of complex-valued memory pairs. s ( α)

nd s ( β) represent orthonormalized complex-valued memory vec-

ors X in αth layer and βth layer, respectively, that are calcu-

ated by complex-valued Gram-Schmidt orthogonalization as fol-

ows; a 1 = A 1 / ‖ A 1 ‖ (p = 1) , b p = A p −∑ k −1

i = p−1 ( a i , A i ) a i and a p = p / ‖ b p ‖ (2 ≤ p ≤ k ), where A denotes complex-valued memory vec-

or, a and b denote the orthonormalized complex-valued vector

nd the orthogonalized complex-valued vector, respectively.

.2. Robotic emotional model with personality factors

The psychological background of relationships between emo-

ion and memory, and personality factors is presented in this sec-

ion. The three stages (i.e. core affect, emotion, and mood) robotic

motional model based on 3D (pleasure-arousal-dominance) scal-

ng model with OCEAN model as personality factors is introduced.

n this model, we consider the OCC model to appraise the emo-

ional information of objects, events, praiseworthiness, and desir-

bility to handle not only human facial expression information, but

lso the several modalities.

.2.1. Pleasure-arousal-dominance model

Mehrabian [37] proposed a Pleasure-Arousal-Dominance (PAD)

odel to describe a large variety of emotional state. In this model,

he pleasure defined as positive versus negative affective state, the

rousal defined in term of level of mental alertness and physical

ctivity, and the dominance is defined as feeling of control and in-

uences others. Based on the result of human study to reveal the

elationship between emotional states and PAD space, the six ba-

ic emotions are positioned as in Table 1 . We utilize the states in

able 1 as the weighting factor for updating the emotion states.

.2.2. Five factors of personality

Personality is one of the key factors to construct the individ-

al differences, such as perception, motivation, and cognition [62] .

ifferences of personality will influence and intervention to indi-

idual psychological phenomena, for instance, perceives emotion

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

nd emotional behaviors. In the past, several models of person-

lity have been introduced. One of the widely accepted person-

lity models is the five factors (i.e. openness, conscientiousness,

xtraversion, agreeableness, and neuroticism) model, proposed by

cCrae and Costa [63] . The five factors of personality were created

hrough a statistical procedure, which is used to analyze how rat-

ngs of various personality traits are correlated for general humans.

able 2 shows the five factors of personality and its descriptions

36] . The five factors model is used in this paper to represent the

obotic personality.

.2.3. Personality affected emotional factors

The relationships between the five factors of personality and

AD model are derived through the linear regression analysis [38] .

his result is summarized as three equations of temperament,

hich includes pleasure, arousal, and dominance as follows;

α = 0 . 21 E + 0 . 59 A + 0 . 19 N (6)

β = 0 . 15 O + 0 . 30 A − 0 . 57 N (7)

γ = 0 . 25 O + 0 . 17 C + 0 . 60 E − 0 . 32 A (8)

here P α , P β and P γ represent the value for pleasant axis

α-axis), arousal axis ( β-axis) and dominance axis ( γ -axis), respec-

ively. O, C, E, A , and N ( where , O, C, E, A, N ∈ [ −1 , 1] ) represent the

ve factors of personality as openness, conscientiousness, extraver-

ion, agreeableness and neuroticism, respectively. We utilize the

bove factors as the parameters to make an individual differences

f emotional reactions.

.2.4. Mathematical descriptions of robotic emotional model

In general, human emotional states are generated not only from

acial expression, but also from several stimulus in the environ-

ent. In addition, researches in the human psychology field have

xpected that the human emotional function is the result of core

ffect, emotion, and mood state composition [64] . In this paper,

e propose a three stage (core affect, emotion, and mood) robotic

motional model in which the emotion states are represented by

AD space. In this model, the OCC model [30] is considered for

ppraisal the emotional information of events, objects, desirability,

raiseworthiness to handle not only human facial expression infor-

ation, but also the several modalities, instead of the raw signal

nalysis, such as tone and volume of speech, to extract the emo-

ional information [65] . From the different standpoint, the concept

f Kansei engineering can be applied for translating feelings and

mpressions of information into arbitrary parameters [66] .

otic emotional model with associative memory for human-robot

2017.0 6.0 69

6 N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

U

Fig. 3. Configuration of interactive robot system.

I

I

I

h

a

m

a

e

m

a

a

s

s

i

t

s

i

fi

I

w

T

fi

5

s

o

c

t

o

t

o

i

The appraisal model based on OCC model can be shown as a

following vector;

� =

⎡

⎢ ⎢ ⎣

ω 1

. . . ω m

⎤

⎥ ⎥ ⎦

, ∀ i ∈ [1 , m ] : ω i ∈ [0 , 1] (9)

where m is the number of basic emotions. ω is defined based on

OCC model for any information.

First, the system observes multi-modal inputs I MI (t)

, such as

visual, sound and contextual information;

I MI (t) =

⎡

⎢ ⎢ ⎣

u 1

. . . u m

⎤

⎥ ⎥ ⎦

, ∀ i ∈ [1 , m ] : u i ∈ [0 , 1] (10)

where m is the number of basic emotions with each modal hav-

ing the same vector form. As a next step, the OCC model extracts

emotional intensities of each information as;

i j(t) = I MI i (t) �

T j(t) (11)

where � denotes emotional intensities that are determined by OCC

model. Exponential T denotes a transpose matrix. Diagonal ele-

ments of U ij ( t ) are utilized as the state of core affect I CA (t)

.

I CA (t) =

⎡

⎢ ⎢ ⎣

U

I 11(t)

. . .

U

I mm (t)

⎤

⎥ ⎥ ⎦

, ∀ i ∈ [1 , m ] : U i ∈ [0 , 1] (12)

In this paper, six basic emotions (i.e. Happy, Sad, Anger, Fear,

Disgust, and Surprise) are used. Therefore, core affect I CA (t)

will be

written as follows;

I CA (t)

=

⎡

⎢ ⎢ ⎢ ⎢ ⎣

ca H

ca S

ca A

ca F

ca D

ca Sur

⎤

⎥ ⎥ ⎥ ⎥ ⎦

=

⎡

⎢ ⎢ ⎢ ⎢ ⎣

intensity of Happy intensity of Sad

intensity of Anger intensity of Fear intensity of Disgust intensity of Surprise

⎤

⎥ ⎥ ⎥ ⎥ ⎦

,

ca ∈ [0 , 1]

(13)

Han et al. [40] proposed interactive robotic emotional variables

( α, β), which represents the reaction from current emotional

intensities on the pleasant-arousal plane. These variables are based

on neutral intensity, happiness intensity, anger intensity and sad-

ness intensity. Based on the above concept, we extend the emo-

tional variables ( α, β , γ ) for PAD model, such that;

αCA t = 0 . 81 ca H − 0 . 63 ca S − 0 . 51 ca A

− 0 . 64 ca F − 0 . 60 ca D + 0 . 40 ca Sur (14)

βCA t = 0 . 67 ca H − 0 . 27 ca S + 0 . 59 ca A

+ 0 . 70 ca F − 0 . 35 ca D + 0 . 67 ca Sur (15)

γ CA t = 0 . 46 ca H − 0 . 33 ca S + 0 . 25 ca A

−0 . 43 ca F + 0 . 11 ca D − 0 . 13 ca Sur (16)

where, variable α, β and γ represent the value for pleasant

axis ( α-axis), arousal axis ( β-axis) and dominance axis ( γ -axis),

respectively. In addition, coefficient of each emotional intensity is

determined by Table 1 .

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

The state of emotion I E (t)

is calculated for each axis as follows;

E α(t)

= tanh

[γ M

(I E α(t−a )

+ P α · αCA (t−a )

)](17)

E β(t)

= tanh

[ γ M

(I E β(t−a )

+ P β · βCA (t−a )

)] (18)

E γ(t)

= tanh

[ γ M

(I E γ(t−a )

+ P γ · γ CA (t−a )

)] (19)

ere, E α , E β and E γ denote a pleasant, an arousal and a dominance

xis, respectively. γ M (0 < γ M ≤ 1.0) is the suppression rate from

ood state. Depending on the combination of the current mood

nd core affect, the value of γ M will change (e.g. If the mood and

motional attribution are positive, γ M takes a high value. If the

ood is positive, but emotional attribution is negative, γ M takes

low value). a takes an arbitrary value as a time delay. P α , P βnd P γ are defined as Eqs. (6) , (7) and (8) , respectively. Here, the

tate of I E (t)

is mapped on PAD space, and it will be moved to the

pecific position based on the effect from core affect. For instance,

f the state of I E (t)

is close to position (α, β, γ ) = (0 . 81 , 0 . 67 , 0 . 46) ,

he intensity of Happy is higher than other emotions.

In general, the mood state will be taken positive or negative

tate [67] . We assume that it can be represented on pleasant axis

n Table 1 . Therefore, emotion-mood transfer coefficient αE t is de-

ned as follows;

αE t = I E α

(t) (20)

Finally, the mood state is determined as follows;

M

(t) = tanh

[γ M

(I M

(t−a ) + P α · αE (t−a )

)](21)

here, γ M (0 < γ M ≤ 1.0) is the suppression rate from mood state.

he variable a takes an arbitrary value as a time delay. P α is de-

ned as in Eq. (6) .

. Interactive robot system

This section presents the details of proposed interactive robot

ystem. The main objective of the proposed system is the devel-

pment of a robotic decision making system based on the mood-

ongruency effect, combining the associative memory and emo-

ional information. We discussed the importance and usefulness

f mood-congruency effect, and information affinity between emo-

ional information and complex-valued model in Section 3 . Based

n above discussion, we developed the robot system as depicted

n Fig. 3 .

otic emotional model with associative memory for human-robot

2017.0 6.0 69

N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13 7

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

Table 3

Comparison of conventional human-robot interaction systems.

System Emotion model Mood state Personality Input Biometric Association Functions of

modality information model robot

Proposed PAD model Positive OCEAN Gesture, Age, Multidirectional Facial display

system (universal emotion) and model object, gender (one-to-many) and arms

negative voice,

facial Exp.

Previous Ekman model Positive None Gesture, None Bidirectional Facial display

work [51] (6 basic emotions) and object, (one-to-one) and arms

negative voice,

facial Exp.

Yorita and None None None Gesture, None Multidirectional Facial display

Kubota [69] object, (many-to-one) and arms

voice,

Hiolle et al. Arousal model None None Touch, None Multidirectional Animal robot

[46] (self-defined) gesture (many-to-one)

Itoh et al. Pleasant-arousal Positive None Voice None Bidirectional Upper body

[48] -certainty model and (one-to-one) of humanoid

(self-defined) negative (WE-4RII)

Yi et al. Emotional energy Positive None Object, None Bidirectional Facial robot

[49] model and voice

(self-defined) negative

Fig. 4. iPhonoid and example of face templates.

5

c

i

r

t

(

g

i

f

a

t

m

p

e

i

o

f

c

s

s

t

t

O

u

a

p

g

m

s

t

i

T

m

s

s

o

p

s

5

i

i

c

g

5

b

n

g

t

d

t

t

p

t

a

e

e

n

p

5

i

t

t

.1. System configuration

The system is composed of the robot, a Microsoft Kinect, a mi-

rophone, and a server Personal Computer (PC). We utilized an

Phonoid as the robot, developed by Kubota and Toda [68] . The

obot is made up of an iPhone with four servo motors. It simul-

aneously outputs audio and display, together with arm motion

Fig. 4 (a)).

From the multi-modal information, several cognitive intelli-

ence extract the meaningful information, and it will be utilized

n the emotion model and associative memory model. Specifically,

acial expression, object, gesture, voice and biometric information

re extracted as the cognitive information from the raw data cap-

ured by a Kinect and a microphone. Based on the cognitive infor-

ation, the emotion states for robot are generated depending on

ersonality factors. Finally, utilizing the cognitive information and

motion states, associative memory model recalls the robot behav-

ors based on the predefined relationships. The brief descriptions

f cognitive intelligences are presented in Section 5.2 .

Majority of the conventional human-robot interaction systems

ocus on specific modality, such as gesture-based, voice-based, fa-

ial expression based, or combination of voice and face-based

ystem. Therefore, these systems are able to handle the limited

ituations in communication. Most of systems are focused on de-

ecting the human emotional information to select the actions of

he robot, not to generate the emotional information for the robot.

n the other hand, the proposed system is able to generate the

niversal emotion information for the robot from the multi-modal

nd biometric information to select the actions of the robot. The

roposed system also accepts other sophisticated cognitive intelli-

ences, due to the definitions of emotional model and associative

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

emory model. The functional differences between conventional

ystems and the proposed system are summarized in Table 3 .

The ability to handle the large number of multi-modal informa-

ion is a significant factor to evaluate the usefulness and adaptabil-

ty of the interactive robot system. According to the comparison in

able 3 , the proposed system has better functionality, both infor-

ation cognition and processing. Furthermore, in order to empha-

ize the justification of proposed system, we provided the discus-

ion about the significance of reproducing the mutual relationship

f memory and emotion from the psychology and brain science

erspective in Section 3 . Therefore, we regard that the proposed

ystem would have a wider applicability than other systems.

.2. Cognitive intelligence for robot

We utilized a number of existing studies to collect the external

nformation for the proposed system. In this section, the cognitive

ntelligences for the multi-modal information (object, gesture, fa-

ial expression and voice) and the biometric information (age and

ender) are briefly introduced.

.2.1. Object recognition

In regards of object recognition, various types of algorithm have

een proposed, such as the template matching methods by dy-

amic programming (DP) [70] , cellular neural network [71] , and

enetic algorithm (GA) [72] . While cellular neural network requires

he exact templates for target, DP and GA can detect the targets

epending on similarity or distance based cost functions as an op-

imization problem. Basically, GA can be divided into a genera-

ional model (standard GA) and steady-state model (SSGA). SSGA

artially replaces a few individuals with offspring in a genera-

ion, not all individuals as standard GA. In particular, SSGA is suit-

ble to solve optimization problems in the dynamic or changing

nvironments [73] .

In general, the interactions are performed under the dynamic

nvironments. Furthermore, we utilized only the color information,

ot topological information for object recognition. Thus, we ap-

lied SSGA based object recognition [68] in the proposed system.

.2.2. Gesture recognition

In general, the context of hand gesture is quite flexible depend-

ng on the hand speed and its movement. Thus, it can be regarded

hat the significance of gesture recognition is to extract the spatio-

emporal information from the hand movements. It is well known

otic emotional model with associative memory for human-robot

2017.0 6.0 69

8 N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

Table 4

Emotional intensity � of multi-modal information.

Attribution Emotional intensity �

Happy Sad Anger Fear Disgust Surprise

Object: Red Cir 0.7 0.0 0.0 0.0 0.0 0.1

Blue Rec 0.0 0.6 0.0 0.1 0.1 0.0

Gesture: Circle 0.6 0.0 0.0 0.0 0.0 0.1

Bye-Bye 0.0 0.1 0.0 0.7 0.1 0.0

Voice: Positive 0.7 0.0 0.0 0.0 0.1 0.0

Negative 0.0 0.1 0.0 0.1 0.7 0.0

Facial rep.: Happy Face 0.7 0.0 0.0 0.0 0.0 0.1

Sad Face 0.0 0.7 0.0 0.1 0.1 0.0

Table 5

Parameter settings of suppression ratio γ M .

Input Positive mood Negative mood

γ M Positive mood 0.95 0.60

Negative mood 0.60 0.95

Table 6

Four types of personality.

Factor Personality

Type 1 Type 2 Type 3 Type 4

Openness 0.50 0.90 0.50 0.20

Extraversion 0.90 0.70 0.20 0.10

Agreeableness 0.50 0.50 0.20 0.30

Neuroticism 0.20 0.30 −0 . 40 −0 . 40

Table 7

Determination of personality type based on biometric information.

Attribution Gender:

Male Female

Age: Young Type 1 Type 2

Adult Type 3 Type 4

6

6

t

m

m

t

r

d

l

i

H

t

r

6

p

F

t

s

f

r

D

p

i

that a Spiking Neural Network (SNN) is able to handle the spatial

and temporal context [74] . In addition, due to the spike response

model, it is able to reduce the computational cost comparing with

integrate-and-fire model. We utilized the SNN based hand gesture

recognition method that is introduced by Kubota and Toda [68] to

the proposed system.

5.2.3. Facial expression recognition

Facial expression plays an important role in communication to

express one’s emotional state directly. The robot system in this pa-

per utilizes Constrained Local Model (CLM) [75] based facial fea-

ture tracking framework [76] . The framework applied two clus-

tering algorithms, namely LeaderP [77] and Topological Gaussian

Adaptive Resonance Theory algorithm (TGART) [78] , for patch clus-

tering and shape clustering, respectively. The above CLM with

above clustering algorithms provide the human dynamic facial

features with superior accuracy, and reducing recognition errors

throughout tracking.

5.2.4. Voice recognition

Verbal communication is one of the essential things for human.

In the past, various studied have introduced. One of the established

open source software is Julius [79] , which runs in real time. In a

20,0 0 0-word reading test, it’s recognition accuracy rates was over

90%. In this paper, Julius is applied with Japanese language model

for voice recognition.

5.2.5. Biometric recognition

Biometric information is a significant factor in human emo-

tional information and behaviors. Specifically, we have tendency

to change our own attitudes/reactions based on facial appearances

of communication partner. Thus, we regard that this ability will

support in obtaining unique characteristics for robot. Eidinger et al.

[80] developed the age and gender estimation algorithm. The al-

gorithm is applied on Local Binary Patterns (LBP) to extract fea-

tures, and classification is performed using standard linear Sup-

port Vector Machine (SVM) that is trained by feature vectors of

LBP. Eidinger et al. [80] report extensive tests analyzing both the

difficulty levels of contemporary benchmarks as well as the capa-

bilities of their algorithm. These show the algorithm to outperform

state-of-the-art by a wide margin. In this paper, the algorithm is

customized to only detect male/female as gender and young/adult

as age for minimum configuration.

6. Experiment of interactive robot system

This section presents experimental results of the proposed in-

teractive robot system. The proposed system performs a number

of behaviors due to robot personality and biometric information

of communication partner. The following subsections describe ex-

perimental conditions that are related to the emotional model and

associative memory. Due to the limitations of cognitive abilities

of system, the practical relationships between each information or

movements are ignored. Thus, the simple symbolic information is

utilized for association. In practical terms, relationships are defined

from actual information, facts, common senses or personal inten-

sities. Here, we consider that the main focus of this experiment is

the association result from associative memory changes depending

on emotional factors and biometric information of communication

partner, which is characterized as a mood-congruency effect.

The experiment is divided to two parts; first, the processing of

multi-modal information into emotional information in the emo-

tion model is simulated. Next, based on multi-modal information

and emotional information, which are came from first one, associ-

ation process will be performed to determine the robot behaviors.

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

.1. Experimental conditions

.1.1. Conditions of personality affected emotional model

The behavior of robot is controlled by modules of robotic emo-

ional model and associative memory based on multi-modal infor-

ation and biometric information, as shown in Fig. 3 .

In each module, there are several predefined information. Each

ulti-modal input is assumed to be assigned the specific emo-

ional intensity � by OCC model, as in Table 4 . The suppression

atio γ M is defined as in Table 5 . Four types of personalities are

efined as in Table 6 . These personality factors are used to calcu-

ate Eqs. (6) and (7) . In the proposed system, biometric information

s affected to determine the types of personality factors as Table 7 .

ere, we show the four types of results based on personality.

Note that the parameters in Tables 4 –7 are all involved in emo-

ion model. It is worth noting that in order to define the unique

obot, we need only to change four parameters in Table 4 .

.1.2. Conditions of association process for robot behaviors

In this section, all inputs of associative memory module are

rovided from emotional model in the previous section, namely,

igs. 6 (a) and 8 . We utilize only male/female and young/adult as

he biometric information for simplicity of conditions. In the as-

ociative memory module, relationships between multi-modal in-

ormation and robot action are defined as shown in Table 8 . Each

elationship is labeled by ID for associative memory (A.M._ID).

epending on the association result and mood state, the robot

erforms several behaviors which identifying Act._ID as shown

n Table 9 . For instance, the robot recognizes a “Red Circle” as

otic emotional model with associative memory for human-robot

2017.0 6.0 69

N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13 9

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

Table 8

Information relationships for QCMAM (A.M._ID).

A.M._ID Moods Input Input Associated information Act._ID

attribution Object Gesture Voice

0 P/N – No Input – – – 0

1 Positive Object Red Circle – Circle Happy 1

2 Negative Object Red Circle – Circle Sad 3

3 Positive Object Blue Rectangle – Bye-Bye Happy 2

4 Negative Object Blue Rectangle – Bye-Bye Sad 4

5 Positive Gesture Circle Red Circle – Happy 1

6 Negative Gesture Circle Red Circle – Sad 3

7 Positive Gesture Bye-Bye Blue Rectangle – Happy 2

8 Negative Gesture Bye-Bye Blue Rectangle – Sad 4

9 Positive Voice Happy Red Circle Circle – 1

10 Negative Voice Happy Red Circle Circle – 3

11 Positive Voice Sad Blue Rectangle Bye-Bye – 2

12 Negative Voice Sad Blue Rectangle Bye-Bye – 4

Fig. 5. Example of robot actions with a neutral face.

Table 9

Definitions of robot action ID (Act._ID).

Act._ID Robot action

Face Gesture Voice

0 Neutral – –

1 Happy Circle Happy

2 Sad Up&Down –

3 Happy Up&Down –

4 Sad Bye-Bye Sad

a

“

t

a

T

p

t

I

a

t

Table 10

Definitions of input information ID (IN_ID).

IN_ID Information attribution

Face Object Gesture Voice

0 Neutral No info. No info. No info.

1 Happy Red Circle Circle Happy

2 Sad Blue Rectangle Bye-Bye Sad

6

6

a

F

s

b

t

{

f

(

g

t

c

s

s

h

r

a

s

i

c

d

n object information with positive mood state, the robot recalls

Circle” and “Happy” as gestural and voice information, respec-

ively. The robot then performs circle-wise gesture by his arm,

nd utter a “Happy” with displaying a Happy face as Act._ID 1 in

able 9 .

Fig. 5 shows gestural actions examples by the robot. In this ex-

eriment, we assigned specific ID (IN_ID) to multi-modal informa-

ion as in Table 10 for visualization. Here, information belonging to

N_ID 0 has a neutral attribution, IN_ID 1 has a positive emotional

ttribution, IN_ID 2 has a negative emotional attribution, respec-

ively.

Fig. 6. Internal states of emotional model. (a)

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

.2. Experimental results

.2.1. Results of emotion information processing in personality

ffected emotional model

The history of multi-modal information is shown in Fig. 6 (a). In

ig. 6 (a), IN_ID denotes the types of multi-modal information as

hown in Table 10 .

Fig. 6 (b) shows the intensity of core affect, which is calculated

y the pre-defined parameters in Table 4 . Thus, as an example,

he first stimuli in Figs. 6 (b) has the intensity of core affect as

Sad : 0.6, Fear : 0.1, Disgust : 0.1} at same time, which are calculated

rom the parameters in Table 4 corresponding to the Blue Rectangle

first stimuli in Fig. 6 (a)).

Based on the intensity of core affect, the emotion states are

enerated as in Fig. 7 . Due to the influence of personality factors,

he four types of results can be defined from the same intensity of

ore affect. Furthermore, from the emotion information, the mood

tates are generated as in Fig. 8 corresponding to 4 types of per-

onality.

As defined in Tables 6 and 7 , we assume that the young woman

as the rich emotional sensitivity, a high adaptability to the envi-

onment, while the adult man has a calm character, and tolerance

bout influence from the environment. From the definitions of per-

onality factors as Table 6 , we assume that Type 1 and 2 personal-

ties indicate the similar properties, as well as Type 3 and 4. Here,

omparing with Type 2 and 3 personalities as in Fig. 7 (b) and (c),

ue to Type 2 has high value of Openness and Extraversion than

Multi-modal inputs and (b) Core affect.

otic emotional model with associative memory for human-robot

2017.0 6.0 69

10 N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

Fig. 7. Trajectory of emotion states based on four types of personality.

Fig. 8. Trajectory of mood states based on four types of personality.

T

t

t

b

m

T

o

i

Section 6.2.2 .

Type 3, Type 2 personality shows the wide range of trajectory than

Type 3 one from the same input information.

The mood state is calculated based on emotion state by

Eq. (21) . Fig. 8 shows the trajectory of mood state. Due to the dif-

ferences of emotion states and personality factors, the mood state

in each types also generate different outputs. As mentioned, Type

2 personality has a sensitive tendency in the stimulus. Thus, the

mood state of Type 2 personality changes quickly and violently.

In contrast, Type 3 personality shows resistance to the stimulus.

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

herefore, the mood state of Type 3 personality changes with long

erm period than other types. Furthermore, the personality fac-

ors bring the noteworthy feature of emotional model which can

e seen at ranges (i), (ii) and (iii) in Fig. 8 . In these ranges, the

ood state of Type 1 and 2 indicate the positive state, while

ype 3 and 4 indicate the negative one. Due to this differences

f mood state, the association process is affected, which is shown

n Section 6.2.2 . The positions (a) to (f) are also mentioned in

otic emotional model with associative memory for human-robot

2017.0 6.0 69

N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13 11

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

Fig. 9. Association results of information relationships and corresponding robot actions.

p

f

p

c

a

6

t

a

s

c

m

t

(

i

s

f

i

p

t

I

t

r

p

i

c

i

7

a

w

g

t

t

g

m

s

t

o

p

f

f

i

c

a

c

r

a

t

t

A

I

C

From the results, it is clear that personality factors play an im-

ortant role to make the different emotional properties, and the

unctions of personality have successfully integrated in the pro-

osed emotional model. We consider that the robot is able to

hange the emotional reactions based on communication partner

ppearance for providing the appropriate responses.

.2.2. Results of robot interaction based on mood-congruency effect

This section presents the association process of the robot ac-

ion depending on the emotional factor. Fig. 9 (a) shows the associ-

tion results (A.M._ID) in associative memory module, and Fig. 9 (b)

hows the corresponding robot actions (Act._ID) based on asso-

iation, respectively. The associations are performed using multi-

odal inputs and emotional information based on predefined rela-

ionships as in Table 8 . In Figs. 9 (a) and 9 (b), the positions of (a) to

f) are plotted in same step point in Fig. 8 , and these positions are

ncluded in ranges (i), (ii) and (iii) as Fig. 8 , respectively.

Due to the differences of mood attribution, the association re-

ults are affected. In this paper, from the definitions of personality

actors as in Table 6 , it is assumed that Type 1 and 2 personalities

ndicate the similar properties, as well as Type 3 and 4. Thus, the

ositions (a), (c), (d) and (e) of Type 1 and 2 personalities show

he same association results, which is different from Type 3 and 4.

n particular, the positions (b) and (f) of Type 3 personality show

he unique results, respectively. In the same time, corresponding

obot actions are also affected as Fig. 9 (b).

From these results, it can be considered that the difference of

ersonality is able to manage the different behaviors of robot. It

s assumed that the suitable and actual information relationships

an be prepared, the robot is able to provide preferable reactions

n any situations.

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

. Conclusions

In this paper, the relationships between associative memory

nd emotional factors from the point of view of the psychology

ith its effect in communication are discussed. Cognitive intelli-

ences, associative memory model and personality affected emo-

ional model are introduced for the interactive robot system. From

he experimental results, it is regarded that the robot is able to

enerate the different emotional response from multi-modal infor-

ation depending on personality factors. In addition, results also

how that the output of associative memory is affected from emo-

ional factors. Thus, the individual robot can perform various types

f responses. Even if the experimental conditions are not under

ractical conditions (applied symbolic information and simple in-

ormation relationships), the system shows that it can generate dif-

erent the emotional information and responses from multi-modal

nformation based on predefined conditions. We assume that if the

ognitive intelligences are well developed, there is a possibility to

pply the system with the complex practical information, fact and

ommon sense. As a result, the interactions between human and

obot would be more natural and active. For future work, work on

n interaction system with multiple robots will be investigated. In

his situation, we are able to assign the specific roles and charac-

ers to individual robots.

cknowledgments

This research is supported by Fellowship Scheme under High

mpact Research UM.C/625/1/HIR/MOHE/FCSIT/10 and UM Grand

hallenge Grant GC003A-14HTM from the University of Malaya.

otic emotional model with associative memory for human-robot

2017.0 6.0 69

12 N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

[

References

[1] M. Minsky , The Society of Mind, Simon and Schuster, New York, 1986 .

[2] R. Pfeifer , C. Scheier , Understanding Intelligence, The MIT Press, 1999 .

[3] A.J. Maren , C.T. Harston , R.M. Pap , Handbook of Neural Computing Applica-tions, Academic Press, 2014 .

[4] L. Besacier , E. Barnard , A. Karpov , T. Schultz , Automatic speech recognition forunder-resourced languages: a survey, Speech Commun. 56 (2014) 85–100 .

[5] D. Sperber , D. Wilson , Relevance – Communication and Cognition, Oxford Uni-versity Press, 1995 .

[6] H. Zhang , Z. Wang , D. Liu , A comprehensive review of stability analysis of con-

tinuous-time recurrent neural networks, IEEE Trans. Neural Netw. Learn. Syst.25 (7) (2014) 1229–1262 .

[7] A. Bechara , H. Damasio , A.R. Damasio , Emotion, decision making and the or-bitofrontal cortex, Cereb. Cortex 10 (3) (20 0 0) 295–307 .

[8] D.E. Reisberg , P.E. Hertel , Memory and Emotion, Oxford University Press, 2004 .[9] P. Ekkekakis , The Measurement of Affect, Mood, and Emotion: A Guide for

Health-behavioral Research, Cambridge University Press, 2013 . [10] W. Mischel , Introduction to Personality, vol. 250, London, 1993 .

[11] K.J. Michalska , K.D. Kinzler , J. Decety , Age-related sex differences in explicit

measures of empathy do not predict brain responses across childhood andadolescence, Dev. Cognit. Neurosci. 3 (2013) 22–32 .

[12] S.C. Gadanho , Learning behavior-selection by emotions and cognition in amulti-goal robot task, J. Mach. Learn. Res. 4 (Jul) (2003) 385–412 .

[13] J.-M. Fellous , M.A. Arbib , Who Needs Emotions?: The Brain Meets the Robot,Oxford University Press, 2005 .

[14] J.J. Hopfield , Neural networks and physical systems with emergent collective

computational abilities, in: Proceedings of the National Academy of Sciences79 (8) (1982) 2554–2558 .

[15] B. Kosko , Constructing an associative memory, Byte 12 (10) (1987) 137–144 . [16] M. Hagiwara , Multidirectional associative memory, in: Proceedings of the In-

ternational Joint Conference on Neural Networks, vol. 1, 1990, pp. 3–6 . [17] N. Aizenberg , Y.L. Ivaskiv , D. Pospelov , G. Hudiakov , Multiple-valued thresh-

old functions. II. Synthesis of the multi-valued threshold elements, Kibernetika

(Cybern.) 1 (1973) 53–66 . [18] S. Jankowski , A. Lozowski , J. Zurada , Complex-valued multistate neural associa-

tive memory, IEEE Trans. Neural Netw. 7 (6) (1996) 1491–1496 . [19] L. Donq-Liang , W. Wen-June , A multivalued bidirectional associative mem-

ory operating on a complex domain, Neural Netw. 11 (9) (1998) 1623–1635 .

[20] M. Kobayashi , H. Yamazaki , Complex-valued multidirectional associative mem-

ory, Electr. Eng. Jpn. 159 (1) (2007) 39–45 . [21] D.L. Lee , Improvements of complex-valued Hopfield associative memory by

using generalized projection rules, IEEE Trans. Neural Netw. 17 (5) (2006)1341–1347 .

[22] M. Kobayashi , Pseudo-relaxation learning algorithm for complex-valued asso-ciative memory, Int. J. Neural Syst. 18 (02) (2008) 147–156 .

[23] R.S. Lee , A transient-chaotic autoassociative network (TCAN) based on Lee os-

cillators, IEEE Trans. Neural Netw. 15 (5) (2004) 1228–1243 . [24] A. Yoshida , Y. Osana , Chaotic complex-valued multidirectional associative

memory with variable scaling factor, in: Proceedings of the InternationalConference on Artificial Neural Networks and Machine Learning–ICANN 2011,

Springer, 2011, pp. 266–274 . [25] V. Gandhi , G. Prasad , D. Coyle , L. Behera , T. McGinnity , Quantum neural

network-based EEG filtering for a brain; computer interface, IEEE Trans. Neural

Netw. Learn. Syst. 25 (2) (2014) 278–288 . [26] G.G. Rigatos , S.G. Tzafestas , Quantum learning for neural associative memories,

Fuzzy Sets Syst. 157 (13) (2006) 1797–1813 . [27] N. Masuyama , C.K. Loo , Quantum-inspired complex-valued multidirectional as-

sociative memory, in: Proceedings of the IEEE International Joint Conferenceon Neural Networks (IJCNN), IEEE, 2015, pp. 1–8 .

[28] P. Ekman , An argument for basic emotions, Cognition &amp; Emotion 6 (3–4)(1992) 169–200 .

[29] J.A. Russell , A circumplex model of affect, J. Pers. Soc. Psychol. 39 (6) (1980)

1161 . [30] A. Ortony , The Cognitive Structure of Emotions, Cambridge University Press,

1990 . [31] J.A. Gray , The neuropsychology of emotion and personality, Zh Vyssh Nerv

Deiat lm l P. 37 (6) (1987) 1011–1024 . [32] J.A. Russell , L.F. Barrett , Core affect, prototypical emotional episodes, and other

things called emotion: dissecting the elephant, J. Pers. Soc Psychol. 76 (5)

(1999) 805 . [33] M. Johns , B.G. Silverman , How emotions and personality effect the utility of al-

ternative decisions: a terrorist target selection case study, Center Hum. Model.Simul. (2001) 10 .

[34] E. André, M. Klesen , P. Gebhard , S. Allen , T. Rist , Integrating models of person-ality and emotions into lifelike characters, in: Affective Interactions, Springer,

20 0 0, pp. 150–165 .

[35] G. Ball , J. Breese , Emotion and personality in a conversational agent, EmbodiedConversational Agents, MIT Press, Cambridge, MA , USA , 20 0 0, pp. 189–219 .

[36] P.T. Costa , R.R. McCrae , Normal personality assessment in clinical practice: theNEO personality inventory, Psychol. Assess. 4 (1) (1992) 5 .

[37] A. Mehrabian , Analysis of the big-five personality factors in terms of the PADtemperament model, Aust. J. Psychol. 48 (2) (1996) 86–92 .

[38] L.R. Goldberg , The development of markers for the big-five factor structure.,

Psychol. Assess. 4 (1) (1992) 26 .

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

[39] M. Schneider , J. Adamy , Towards modelling affect and emotions in autonomousagents with recurrent fuzzy systems, in: Proceedings of the 2014 IEEE Interna-

tional Conference on Systems, Man and Cybernetics (SMC), IEEE, 2014, pp. 31–38 .

[40] M.J. Han , C.H. Lin , K.T. Song , Robotic emotional expression generation basedon mood transition and personality model, IEEE Trans. Cybern. 43 (4) (2013)

1290–1303 . [41] J.A. Russell , M. Bullock , Multidimensional scaling of emotional facial expres-

sions: similarity from preschoolers to adults, J. Pers. Soc. Psychol. 48 (5) (1985)

1290 . [42] S.M. Smith , R.E. Petty , Personality moderators of mood congruency effects on

cognition: the role of self-esteem and negative mood regulation, J. Pers. Soc.Psychol. 68 (6) (1995) 1092 .

[43] M. Naveh-Benjamin , G.B. Maddox , P. Jones , S. Old , A. Kilb , The effects of emo-tional arousal and gender on the associative memory deficit of older adults,

Mem. Cognit. 40 (4) (2012) 551–566 .

44] A.M. Rosenthal-von der Pütten , N.C. Krämer , L. Hoffmann , S. Sobieraj , S.C. Eim-ler , An experimental study on emotional reactions towards a robot, Int. J. Soc.

Robotics 5 (1) (2013) 17–34 . [45] L.-F. Chen , Z.-T. Liu , M. Wu , M. Ding , F.-Y. Dong , K. Hirota , Emotion-age-gen-

der-nationality based intention understanding in human–robot interaction us-ing two-layer fuzzy support vector regression, Int. J. Soc. Robotics (2015) 1–21 .

[46] A. Hiolle , L. Canamero , M. Davila Ross , K.A. Bard , Eliciting caregiving behav-

ior in dyadic human-robot attachment-like interactions, ACM Trans. Interact.Intell. Syst. (TiiS) 2 (1) (2012) 3 .

[47] T. Rumbell , J. Barnden , S. Denham , T. Wennekers , Emotions in autonomousagents: comparative analysis of mechanisms and functions, Autonomous

Agents Multi-Agent Syst. 25 (1) (2012) 1–45 . [48] K. Itoh , H. Miwa , H. Takanobu , A. Takanishi , Application of neural network to

humanoid robots—development of co-associative memory model, Neural Netw.

18 (5) (2005) 666–673 . [49] W. Yi , W. Zhi liang , W. Wei , Research on associative memory models of emo-

tional robots, Adv. Mech. Eng. 6 (2014) 208153 . [50] R. Valverde Ibanez , M.U. Keysermann , P. Vargas , Emotional memories in au-

tonomous robots, in: Proceedings of the 23rd IEEE International Symposiumon Robot and Human Interactive Communication, IEEE, 2014, pp. 405–410 .

[51] N. Masuyama , M. Islam , M. Seera , C. Loo , Application of emotion affected as-

sociative memory based on mood congruency effects for a humanoid, NeuralComput. Appl. (2015) 1–16 .

[52] G.H. Bower , Mood and memory, Am. Psychol. 36 (2) (1981) 129 . [53] P. Lewis , H. Critchley , A. Smith , R. Dolan , Brain mechanisms for mood congru-

ent memory facilitation, Neuroimage 25 (4) (2005) 1214–1223 . [54] B.H. Pierce , E.A. Kensinger , Effects of emotion on associative recognition: va-

lence and retention interval matter, Emotion 11 (1) (2011) 139 .

[55] B.D. Murray , E.A. Kensinger , Age-related changes in associative memory foremotional and nonemotional integrative representations, Psychol. Aging 28 (4)

(2013) 969 . [56] G. Egidi , H.C. Nusbaum , Emotional language processing: how mood affects

integration processes during discourse comprehension, Brain Lang. 122 (3)(2012) 199–210 .

[57] N. Ravaja , J. Kätsyri , Suboptimal facial expression primes in textual media mes-sages: evidence for the affective congruency effect, Comput. Human Behav. 40

(2014) 64–77 .

[58] A.M. Collins , E.F. Loftus , A spreading-activation theory of semantic processing,Psychol. Rev. 82 (6) (1975) 407 .

[59] J. Rottenberg , J.J. Gross , When emotion goes wrong: realizing the promise ofaffective science, Clinical Psychol.: Sci. Pract. 10 (2) (2003) 227–232 .

[60] E. Ba ̧s ar , Brain Function and Oscillations: Volume II: Integrative Brain Function.Neurophysiology and Cognitive Processes, Springer Science & Business Media,

2012 .

[61] C.I. Hooker , L. Bruce , M. Fisher , S.C. Verosky , A. Miyakawa , S. Vinogradov , Neu-ral activity during emotion recognition after combined cognitive plus social

cognitive training in schizophrenia, Schizophrenia Res. 139 (1) (2012) 53–59 . [62] A. Mehrabian , Basic dimensions for a general psychological theory: Impli-

cations for personality, social, environmental, and developmental studies,Oelgeschlager, Gunn & Hain, Cambridge, MA, 1980 .

[63] R.R. McCrae , P.T. Costa , Validation of the five-factor model of personality across

instruments and observers, J. Pers. Soc. Psychol. 52 (1) (1987) 81 . [64] J.A. Russell , Core affect and the psychological construction of emotion, Psychol.

Rev. 110 (1) (2003) 145 . [65] Z. Esmaileyan , H. Marvi , Recognition of emotion in speech using variogram

based features, Malays. J. Comput. Sci. 27 (3) (2014) . [66] M. Nagamachi , A.M. Lokman , Innovations of Kansei Engineering, CRC Press,

2016 .

[67] P. Ekman , Basic Emotions, Handbook of Cognition and Emotion, 98, 1999,pp. 45–60 .

[68] N. Kubota , Y. Toda , Multimodal communication for human-friendly robot part-ners in informationally structured space, IEEE Trans. Syst. Man Cybern. Part C:

Appl. Rev. 42 (6) (2012) 1142–1151 . [69] A. Yorita , N. Kubota , Cognitive development in partner robots for information

support to elderly people, IEEE Trans. Autonomous Ment. Dev. 3 (1) (2011)

64–73 . [70] D. Bertsekas , Dynamic Programming: Deterministic and Stochastic Models,

Prentice-Hall, 1987 . [71] J.A. Anderson , Neurocomputing, vol. 2, MIT Press, 1993 .

otic emotional model with associative memory for human-robot

2017.0 6.0 69

N. Masuyama et al. / Neurocomputing 0 0 0 (2017) 1–13 13

ARTICLE IN PRESS

JID: NEUCOM [m5G; July 10, 2017;22:10 ]

[

[

[

[

[72] T. Back , D.B. Fogel , Z. Michalewicz , Handbook of Evolutionary Computation, IOPPublishing Ltd., 1997 .

[73] N. Kubota , T. Fukuda , Ecological model of virus-evolutionary genetic algorithm,Fundamenta Informaticae 37 (1, 2) (1999) 51–70 .

[74] W. Maass , Networks of spiking neurons: the third generation of neural net-work models, Neural Netw. 10 (9) (1997) 1659–1671 .

75] D. Cristinacce , T. Cootes , Automatic feature localisation with constrained localmodels, Pattern Recognit. 41 (10) (2008) 3054–3067 .

[76] M.N. Islam , C.K. Loo , Geometric feature-based facial emotion recognition using

two-stage fuzzy reasoning model, in: Neural Information Processing, Springer,2014, pp. 344–351 .

Please cite this article as: N. Masuyama et al., Personality affected rob

interaction, Neurocomputing (2017), http://dx.doi.org/10.1016/j.neucom.

[77] J. Nuevo , L.M. Bergasa , P. Jiménez , RSMAT: robust simultaneous modeling andtracking, Pattern Recognit. Lett. 31 (16) (2010) 2455–2463 .

78] F. Dawood , C.K. Loo , W.H. Chin , Incremental on-line learning of human mo-tion using gaussian adaptive resonance hidden Markov model, in: Proceedings

of the 2013 International Joint Conference on Neural Networks (IJCNN), IEEE,2013, pp. 1–7 .

79] A. Lee , T. Kawahara , Recent development of open-source speech recognitionengine Julius, in: Proceedings: APSIPA ASC, 2009, pp. 131–137 .

80] E. Eidinger , R. Enbar , T. Hassner , Age and gender estimation of unfiltered faces,

IEEE Trans. Inf. Forensics Secur. 9 (12) (2014) 2170–2179 .

otic emotional model with associative memory for human-robot

2017.0 6.0 69