Research on Emotion Recognition and the Usage ofEmoticons in Social Media

Xiao HanUniversity of Birmingham

xxh339@student.bham.ac.uk

Abstract—Recent research indicate that nowadays, thestudy of emotion has become an important module in thedomain of interaction design. The first part of thisresearch will introduce the main idea of “emotion” inhuman-computer interaction and effective ways to detectemotions. And after explaining the connection between theexpression and recognition of emotion, it will talk abouthow can people communicate better with the help ofemoticons by using social applications on systems ordevices like mobile phones. Also, discussions about futurework will be presented at the end of this report.

Keywords—Emotion; detecting emotion; emoticons;communicate; social media; human-computer interaction

I. INTRODUCTIONEmotion is playing an important part of our daily life.

People express and present themselves with all different kindsof emotions. For example, simple emoticons can basicallyexpress all kinds of facial expressions while other expressionslike bodily expressions can be presented by complexemoticons, for strong emotions are always involved indifferent kinds of emoticons.

The first part of this paper will introduce human emotion(which is quite complicated and associated) and the way thatpeople usually choose to express their emotions(consciously orunconsciously). Then it will show the modern techniques todetect human emotion to present the psychological state ofhuman more correctly and intuitively.

The aim of this research is to investigate recent method todetect emotions and analyse how do emoticons play animportant part in nonverbal communication. Moreover, byunderstanding the existing problems and potential usage ofemoticons, people may able to create better communicationenvironment for social media in the future.

II. EMOTION

A. What is “emotion”?Emotion refer to many different domains, such as

neuroscience, cognitive science, psychology, computer

science, etc[10]. We usually mean the instant responseexpressed under a certain situation when we talk about theword “emotion”, which is the product of cognition andprogress of consciousness. Lots of researchers have giventheir opinions on what emotion is, however, an integrateddefinition of emotion is still needed[9].

There are six basic emotions of human: happiness, sadness,surprise, anger, disgust and fear[3,4]. Here are some examplesof those basic emotions(Figure 1):

Figure 1: The image of six basic emotions[1]

We can see details form the photograph showing that withthe switch of emotion, some part of human face would changetogether at the same time(e.g. eyebrows, lips and eyelids).Also the basic emotions can be somehow combined withfacial expressions.

Basic emotions are rather easy for other people todistinguish and recognize, however, during the time ofcommunicating with other people(or even systems anddevices) in our real life, people tend to present emotionswhich are always much more complicated than the basic onesmentioned above[5]. For example: guilt, awkward, shyness

and excitation. That is because the six basic emotions are ableto combine and interact with each other and create multiplesubtle and complex effective states according to Hatice in2010[10]. Those emotions are usually learned whenindividuals are exposed to a specific circumstance[4], onecomplex emotion can be elected immediately by manydifferent kinds of culture. For instance, you may feel nervousnot only by the time you meet your parents in law at the firsttime but also when you are about to take an importantinterview. You may present exactly the same behaviour suchas sweating and hand-wringing under both of the twodifferent situations. Additionally, complex emotions can beexpressed into various signals such as facial expression,verbal expression and audio expression which lead moredifficulty to the recognition of emotion.

B. Expressions of emotionIndividuals always have their own ways to express

themselves. There could be many different ways to let otherpeople or systems know their emotions: facialexpression[8,13], body expression[11,12], voiceexpression[14], textual expression[15] and even bio-potentialsignals[16].

Facial expression, body expression and voice expressionare the main way to communicate with others in human-human interactions. Coulson(2004) found out that basically,human can recognize emotion from bodily expression aseffective as facial expressions and audio expressions[12]. Theresearch made by Van don Stock, Righart and De Gelder(2007)also indicate that all emotions can be recognized from bodypostures[11]. The expressions are neither rigid nor stable inreal life. For example, there could be many different kind ofexpression to present the emotion “exciting”. People maychoose one of the actions to behave but they may also presentall the actions below in the same time: yell out, jump high,wave their arms etc(Figure 2). In all, what people present toothers really depends on the different personality, cognitionchoices and surrounding situations of individuals.

experience

Figure 2: Getting excited

There is an argue that human are actually not that good atrecognizing emotions of other people or even them selves,which means they may misunderstood the real meaning of theexpressions[7]. For the situations change all the time andmake it really difficult for people to judge the emotion at themoment and mostly people won’t realize that the reaction of acertain emotion differs from people. Meanwhile, someresearchers consider emotion as a kind of stimuli that willtrigger emotional behaviour[3] while Jeremy R. Gray etal.(2002) indicate that emotions controls human thoughts andbehaviour while cognition controls emotions, which meansemotion and cognition interact each other and play a balancedrole in our daily life[2]. With the control of cognition, someemotions can be hidden(people choose not to present thecorresponding behaviour), which could make others miss thereal emotion and take it differently. For example, someonecan even appear to be calm while his is actually very angry.

C. Emotions in human-computer interactionSince most emotions are presented unconsciously,

effective recognition of human emotion in human-computerinteraction become more important. What’s more, there couldbe a word of difference between the posed expression andautomatic(natural) expression, and it is hard to distinguishthem for human, which lead to another reason for thedevelopment of computerized emotion detection.

People have different personality and habits, each personcommunicate and interact with others in their own differentways, which means the detection of emotion would not be easyfor us. Computerized emotion detection and recognition makeit possible to establish better emotional interaction betweenhuman and computers[8].

The main ways to communicate between human anddevice when people need to communicate with others are totype, speak and facial recognition. However, typing needsusers to play lots of attentions on the virtual keyboard whichwould cost concentration and high degree of participation[7].During the experience of typing on mobile phones, peoplewith big fingers would find it inconvenient with poor usingexperience.

The research of them also shows that there would be apotential option to take the place of typing and speaking in thefuture, which is called Brain Computer Interface(BCI)[7].

According to them, BCI would be a system or device which isable to detect and receive signals from human brains directlyand explain the signals into instructions for the platform ofsystem and device. But the problem of privacy concern mustbe considered under that situation, for where would be a directinterface of human and computer.

III. EMOTION DETECTION AND RECOGNITION

This part of research will talk about how researchers havebeen collecting emotional data from the interaction betweenhuman and devices under different kinds of situations.

One research have showed that “humans are not alwaysefficient at detecting emotion in others or even withinthemselves.[6]” Which means sometimes people need the help ofsensitive devices or systems to detect emotions for them.

With the development of emotion detection, it becomepossible for people to recognize, understand and expressemotions of other people under the multi lingual and cultureenvironment, which may also provide feasibility for creatingmultiple valuable applications.

Hatice provided lots of devices which are used to collectemotional data form our daily life in the research. For example:cameras for image data(usually for PC and mobile phones);microphones for voice recording; infrared cameras for thermalimagery and other sensors for bio-potential signalrecording(Figure 3)[10].

Figure 3: Devices to detect emotional data: (a) cameras forimage data, (b) microphones for voice recording, (c)infrared cameras for thermal imagery

There are also some other methods to detect emotions,which will be introduced below:

Audio emotion detectionWe can usually perceive emotions through the “explicit

message” and “implicit message” expressed by other peoplewhen we are talking to each other[10]. Both “implicitmessage” and “explicit message” are very important for therecognition of emotion. Words usually have double meanings,for instance, while a native English speaker says “Veryinteresting.”, he may actually means “That is clearlynonsense.”, but what others understand is still “Veryinteresting”.

Ortony and Turner(1990) mentioned that “Different peoplechoose different words to express exactly the same things”[18].

After that, Ambady and Rosenthal(1992) indicate thatactually, the investigation of language and voice information isnot the most effective way to detect and forecasting humanemotion and behaviour[17]. For there could be a lot ofproblems trying to predict someone’s intention and expression.

Behaviour emotion detectionThe “nonverbal behaviour” was mentioned in the research

of Irina in 2008. Research explored the reactions(emotionalbehaviour) of users to different kinds of piratical applicationsin the interactive systems. Irina points out that emotionalaspect would do helps on the system of integrating people withdisabilities[4], which mainly because people with disabilitiesusually face lots of block during their daily life(e.g.communication problems, understand problems et al.),especially when facing new technology and unfamiliardomains. More specifically, assume there is a person withspeak disorders, a nice body expression detector would bereally helpful on expressing

Thermal signals detectionThermal signals detection is quite different from other

emotional detection. Researches show that there exist someassociations between bio-body temperature and theiremotional state[10]. For example, the experiment made byNakayama et.al in 2005 found out that the face temperature ofmonkeys would drop down when they felt scared orterrified[19]. And in the same year, another research made byVianna and Carrive found out that some part of monkeys’body temperature would drop down while other part of whichwould increase[20].

Differed from other kind of emotional detection, thermalcamera do not rely on getting in touch directly with theorganism and will not be influence by other emotional signalssuch as facial experiences and body behaviour[10].

Text based emotion detectionEmotions always hidden in text like books, papers, stories

and even the words and expressions people post online. Ashort sentence may implied lots of complex emotions, whichalso means one world could have double or even multiplemeanings.

Chengqin and Fuji (2014)indicate that text emotiondetection can be classified into 2 basic branches[21]:

(1)Sentiment analysis(opinion extract)[22]

This branch extract the attitude and opinions of peoplefrom the text posted online and focus on the valence of textinstead of classify the text into a certain emotionalcategory[21].

(2)Emotion analysis

This part of detection classify human mood extract fromtext into emotion category[21].

Chengqin and Fuji also declare that text based emotionaldetection can refer to many aspects such as headlines of news,microbloggs, message(text), stories and spoken dialog[21].

On the other hand, Weiyuan and Hua(2014) have made aresearch on one of the most famous Chinese social networkapplication called “Microblogging”(which is similar asFacebook). They discussed the relationship between emotionand cause event then made deep classifications to emotions byextracting large amount of emotional data from the text peopleposted online[15].

The results of the experiment shows that the “emotioncause extraction technique”do help on finding moreinformative tings, which proves there would be great potentialto add other domains into the traditional emotion classificationmodels and make emotion to be classified more clearly.However, Weiyuan and Hua think there is still a challenge thatcomplicated linguistic patterns would still be quite difficult forsystems to recognize [15].

Facial emotion detectionAs was mentioned in the first part of research, facial

expressions are defined to be a very important way to presenthuman emotion.

In recent years, facial expression detection has been usedin lots of areas like “lie detection, surveillance, criminalinvestigations, security and forensic applications”[23] byanalyzing human emotions form the facial expression.

IV. CONVEY EMOTIONS THROUGH SOCIAL APPLICATIONS

A. BackgroundWhen we use the applications in systems or devices like

computers or mobile phones, the expressions are limited butstill can be attractive using “Nonverbal communication”[4]like lively text expression and the usage of emoticon(whichalso can be recognized as a kind of facial expression) .

In this part of research, we are starting to focus on theeffect of emoticon expression through those social network

applications. People start to choose different applications tocommunicate with others on their phones and computers.Which choices depend mostly on their own grown up context ,culture and most importantly, how user-friendly andconvenient the applications are.

“It’s still hard to observe which messaging apps dominatein each part of the world.”[25]

In China, most people(even the elderly), have start usingsocial network applications instead of text message to connectwith families and friends. For text message do not provideenough functions to express emotion. For example, althoughyou can send pictures though text message, in China it wouldcost extra money and the amount of emoticons kind are ratherpoorly. There is an instant messenger application calledWeChat produced by Tencent company, which is China’s topsocial application platform with 71.8% of users, well ahead ofother two popular applications called Qzone (53.0%) andWeibo(Microbloggs) (39.9%)[24]. Which increase can also beseen clearly in Figure 4 :

Figure 4: The increasing trend of WeChat users, Qzoneusers and Weibo users[24].

There is no doubt that WeChat has become the mostpopular social media in China., and up until the beginning of2015, there has been 468 million active users from all over theword according to the WeChat official announcement. You arenot only able to send all kinds of messages to your friends, butalso post picture, words, emotions and videos on the platformjust like Facebook.

Research of Martin also shows that most usage of WeChatare on phones, which means people find themselves morewillingly(or conveniently) communicating to other people onmobile devices(Figure 5).

Figure 5: The usage of WeChat on different devices(phone,pad and computer)[24].

It seems that the limitations for people to communicatethrough social media(e.g. People can not capture theimmediate reactions or body/face expressions of others) do nothave much influence on their communication.

The next part of research will talk about the usage ofemoticons and how it help people communicate on socialapplications(which could be one of the reasons why WeChathas taken the place of system text message).

B. About usage of emoticons

The ubiquitous emoticonsWe can see many kinds of emoticons online, basically each

company would design several series of emoticon groups fortheir own applications. As can be seen below in figures:

Figure 6: Facebook

Figure 7: “Emoji” in iPhone

Figure 8 Weibo

Figure 9: WeChat

As can be seen above, the six basic emotions(happiness,sadness, surprise, fear, anger and disgust[3,4]) are alwaysincluded in the groups, although those four emoticon groupslook somewhat different from each other. Also, the emoticonsin Facebook are quite laconic and simple, while emoticonsdesigned from other cultures(e.g. Japan and China) can get abit more complicated.

Do you really understand what he/she means?There is an interesting thing that one emotion can be

expressed by many different types of emoticons. Here aresome examples, from each group of which you can easilyrecognize the similar emotions:

Figure 10: Happiness

Figure 11: Anger

Most of those emoticons are rather easy to recognizebecause designers have exaggerated those facial expressionsand make the emotions quite clear to understand. However,one emoticon also can be explained into two or even morekinds of emotion states, which means you may misunderstandwhat they are really trying to express. And if you don’t get theidea of him/her, you would miss a lot fun during the chat.

Here are some typical examples of this kind of situation inChinese social media communication. It is interesting thatbasically, no similar irony meanings are found in other groupsof emoticons.

Figure 12: The real meaning of smile face in China

(FromWeChat)

Figure 12: The real meaning of goodbye face in China

(FromWeChat)

Figure 13: The real meaning of painting nails in China

(From Emoji)

As can bee seen above, there could be lots of problems forpeople who don’t understand the real meaning of thoseemoticons. This kind of situation would easily occur betweenpeople from different cultures and even different age groups,the recognition of complex emoticons still remains to be aproblem.

How do emoticons brighten up our communications?While using social media on mobiles or computers, people

lost the chance to interact with others directly, which measpeople can not get the immediate facial or bodily signals and

have to recognize emotions through text message, voicemessage, emoticons and some other indirect expressions.

Thus, the communication would be boring and serious without the usage of emoticons, moreover, this kind of visualexpressions can easily elicit the resonance of people involvedin the conversation.

Basic emoticons have been introduced above, this part ofresearch will introduce some image emoticons that looks muchmore complicated which may include not only facialexpressions, but also bodily expressions. Those emoticons areso lively that people can easily be influenced.

Figure 14: Image emoticons expressing happiness

Figure 15: Image emoticons expressing happiness

The usage of those image emoticons differs from cultures,age groups and individual choices, which make thecommunication in social medias a lot more attractive.

V. SUMMARY OF PROBLEMS AND LIMITATION

Human emotion itself is quite complicated andassociated, which means the expression of emotions(e.g.facial expression, body expression, voice expression et al.) arenot accurate and would change randomly under differentcontext. Both emotion detection and recognition are stillchallenging works.

For instance, voice expression is considered as animportant part of the out put of human emotion, however, thedetection on which does not seem to be effective and appliedenough according to the research mentioned before.

On the other hand, Even though emoticons have beenproved to be able to bring multiple vivid emotionalexpressions to our communication in social media, there stillremains some problems. For instance, apart from those mostcommon used emoticons, different age groups of peoplewould find it really hard to understand some of the emoticonexpressions(they probably prefer the old style of expressing),which problems exist especially among elderly and youngpeople. People from different country or different culturecould also face the similar problems, which leave lots ofrestriction to the chat. Thus, how to recognize emotionalsignals correctly still needs to be found out.

VI. CONCLUSION AND FUTURE WORK

In this report, we looked at several methods to detecthuman emotion and focus on the importance of the usage ofemoticons in mobile social media. Although theunderstanding of emoticon may differ form cultures and agegroups, it still become ubiquitous in our socialcommunications in recent years.

People start to notice the importance of emotionaldetection through different kind of sensors in devices orsystems in recent years, several sensors have already existedin our mobile phones(e.g. camera, microphone, distancesensor, light sensor, gravity sensor et al.). Researchers shouldfocus more on how to make better use of all those sensors toprovide more useful information of human emotion. Forexample, find some way to detect how much strength usersput their phones on tables or other objects, which could be animportant part of user emotions(when people get angry, theymay drop their phones roughly and that strength/concussionshould be record). The developing techniques of emotiondetection might help solving the problems mentioned above inthe near future, for emoticon makes the expressions of ouremotions in social media much more interesting and attractive.

By using the data collected from sensors, we will be ableto analyse as much emotional message from users as possible

and help people recognize emotions from other people(oreven themselves) more correctly in the future， all to makebetter communication in both human-human interaction andhuman-computer interaction.

REFERENCES

[1] Ernest W. Semerda. " Microexpressions: The truth is written allover our faces. " The Road to Silicon Valley. 19 Sep. 2013. 9Mar. 2015. <http://www.theroadtosiliconvalley.com/personal-development/microexpresions-lie-to-me>.

[2] J. R. Gray, T. S. Braver, and M. E. Raichle, "Integration ofemotion and cognition in the lateral prefrontal cortex,"Proceedings of the National Academy of Sciences, vol. 99, pp.4115-4120, 2002.

[3] D. Keltner and P. Ekman, "Facial expression of emotion,"Handbook of emotions, vol. 2, pp. 2-17, 2000.

[4] I. Cristescu, "Emotions in human-computer interaction: the roleof nonverbal behavior in interactive systems," RevistaInformatica Economică nr, vol. 2, pp. 110-116, 2008.

[5] S. B. Cohen, O. Golan, S. Wheelwright, and J. Hill, "Mindreading: The interactive guide to emotions," London: JessicaKingsley, 2004.

[6] N. Naveed, T. Gottron, J. Kunegis, and A. C. Alhadi, "Badnews travel fast: A content-based analysis of interestingness ontwitter," in Proceedings of the 3rd International Web ScienceConference, 2011, p. 8.

[7] A. Almehmadi, M. Bourque, and K. El-Khatib, "A Tweet of theMind: Automated Emotion Detection for Social Media UsingBrain Wave Pattern Analysis," in Social Computing(SocialCom), 2013 International Conference on, 2013, pp. 987-991.

[8] M. Ilbeygi and H. Shah-Hosseini, "A novel fuzzy facialexpression recognition system based on facial feature extractionfrom color face images," Engineering Applications of ArtificialIntelligence, vol. 25, pp. 130-146, 2012.

[9] P. E. Griffiths, What emotions really are: The problem ofpsychological categories: Cambridge Univ Press, 1997.

[10] H. Gunes, "Automatic, dimensional and continuous emotionrecognition," 2010.

[11] J. Van den Stock, R. Righart, and B. De Gelder, "Bodyexpressions influence recognition of emotions in the face andvoice," Emotion, vol. 7, p. 487, 2007.

[12] M. Coulson, "Attributing emotion to static body postures:Recognition accuracy, confusions, and viewpoint dependence,"Journal of nonverbal behavior, vol. 28, pp. 117-139, 2004.

[13] A. Sánchez, J. V. Ruiz, A. B. Moreno, A. S. Montemayor, J.Hernández, and J. J. Pantrigo, "Differential optical flow appliedto automatic facial expression recognition," Neurocomputing,vol. 74, pp. 1272-1282, 2011.

[14] J. Hornak, E. Rolls, and D. Wade, "Face and voice expressionidentification in patients with emotional and behaviouralchanges following ventral frontal lobe damage,"Neuropsychologia, vol. 34, pp. 247-261, 1996.

[15] W. Li and H. Xu, "Text-based emotion classification usingemotion cause extraction," Expert Systems with Applications,vol. 41, pp. 1742-1749, 2014.

[16] K. Takahashi, "Remarks on emotion recognition from multi-modal bio-potential signals," in Industrial Technology, 2004.IEEE ICIT'04. 2004 IEEE International Conference on, 2004,pp. 1138-1143.

[17] N. Ambady and R. Rosenthal, "Thin slices of expressivebehavior as predictors of interpersonal consequences: A meta-analysis," Psychological bulletin, vol. 111, p. 256, 1992.

[18] A. Ortony and T. J. Turner, "What's basic about basicemotions?," Psychological review, vol. 97, p. 315, 1990.

[19] K. Nakayama, S. Goto, K. Kuraoka, and K. Nakamura,"Decrease in nasal temperature of rhesus monkeys (Macacamulatta) in negative emotional state," Physiology & behavior,vol. 84, pp. 783-790, 2005.

[20] D. M. Vianna and P. Carrive, "Changes in cutaneous and bodytemperature during and after conditioned fear to context in therat," European Journal of Neuroscience, vol. 21, pp. 2505-2512,2005.

[21] C. Quan and F. Ren, "Textual emotion recognition forenhancing enterprise computing," Enterprise InformationSystems, pp. 1-22, 2014.

[22] A. Neviarouskaya, H. Prendinger, and M. Ishizuka, "SentiFul:A lexicon for sentiment analysis," Affective Computing, IEEETransactions on, vol. 2, pp. 22-36, 2011.

[23] A. Besinger, T. Sztynda, S. Lal, C. Duthoit, J. Agbinya, B. Jap,et al., "Optical flow based analyses to detect emotion fromhuman facial image data," Expert Systems with Applications,vol. 37, pp. 8897-8902, 2010.

[24] Martin Guo. " Chinese enthusiasm for social media tails offsharply. " KANTAR. 3 Mar. 2015. 9 Mar. 2015. <http://cn-en.kantar.com/media/social/2015/kantar-china-social-media-impact-report-2015/>.

[25] Steven Millward. " WeChat sees resurgent growth, now has 355million active users." TECHINASIA. 19 Mar. 2014. 9 Mar.2015. <http://cn-en.kantar.com/media/social/2015/kantar-china-social-media-impact-report-2015/>.