Affective Personalization - from Psychology to Algorithms

Affective Personalization - from Psychology to Algorithms

Marko Tkalčič, Free University of Bozen-Bolzanohttp://markotkalcic.com

Talk at the Alpen-Adria-Universität Klagenfurt21. December 2017

Marko Tkalčič, AAU 1/53

Table of Contents

Introduction

Models of Emotion and Personality

Acquisition of Personality

Usage of personality for personalization

Acquisition of Emotions

Usage of emotions for personalization

Conclusion


Who am I?

Marko Tkalčič

• 2017 : habilitation as Associate Professor in Italy• 2016 - now : assistant professor at Free University of

Bozen-Bolzano• 2013 - 2015: postdoc at Johannes Kepler University,

Linz• 2011 - 2012: postdoc at University of Ljubljana• 2008 - 2010: PhD student at University of Ljubljana

My research explores ways in which psychologically-motivated user characteristics,such as emotions and personality, can be used to improve recommender systems(personalized systems in general). It employs methods such as user studies andmachine learning.


Book, 2016

• Tkalčič, M., Carolis, B. De, Gemmis, M. de, Odić, A., &Košir, A. (Eds.). (2016). Emotions and Personality inPersonalized Services. Springer International Publishing.https://doi.org/10.1007/978-3-319-31413-6

• Authors from• Stanford, Cambridge, Imperial College, UCL . . .

• topics:• psychological models• acquisition of emotions/personality• personalization techniques

• http://www.springer.com/gp/book/9783319314112


Personalized systems

The research I do is about Personalization

Most frequently Recommender Systems


Personalized systems

The research I do is about Personalization

Most frequently Recommender Systems


Recommender systems


Recommender Systems are not Perfect


Why do recommender systems make mistakes?

What is Netflix recommending us?

Movies/films . . . really?

“I want to watch a funny movie tonight”

Funny is all you want?




















But there’s more!!

Question:Can (rating/genre/year/director)summarize that rollercoaster?

Thanks to Shlomo Berkovski for the inspiring example from the EMPIRE 2015 keynote.

Image source: http://yhvh.name/?w=2646


But there’s more!!

Question:Can (rating/genre/year/director)summarize that rollercoaster?

Thanks to Shlomo Berkovski for the inspiring example from the EMPIRE 2015 keynote.

Image source: http://yhvh.name/?w=2646


Decision making and emotions - Damasio

• physiological/evolutionary aspect• emotional processes guide (or bias) behavior, particularly decision-making• changes in both body and brain states in response to different stimuli• these physiological signals (or somatic markers) and their evoked emotion are

consciously or unconsciously associated with their past outcomes and biasdecision-making

ReferencesDamasio, A. (1994) Descartes’ Error: Emotion, Reason, and the Human Brain


Kahneman Tversky two systems

• Decision making:• System 1: fast, intuitive, emotion-driven• System 2: slow, rational

ReferencesKahneman, D. (2003). A perspective on judgment and choice: mapping bounded rationality. The American Psychologist, 58(9),697–720.


Personality and preferences

Personality traits (extraverted/introverted, open/conservative etc.) are linked to musicgenre preferences (Rentfrow et al, 2003)

ReferencesRentfrow, P. J., and Gosling, S. D. (2003). The do re mi’s of everyday life: The structure and personality correlates of musicpreferences. Journal of Personality and Social Psychology, 84(6), 1236–1256.Tkalčič, M., Ferwerda, B., Hauger, D., and Schedl, M. (2015). Personality Correlates for Digital Concert Program Notes. In UMAP2015, Lecture Notes On Computer Science 9146 (Vol. 9146, pp. 364–369).


Emotions are related to other things as well

Why we choose to consume some kind of content?

One of the main reasons why people consume music (Lonsdale, 2011) and films (Oliver,2008) is emotion regulation.

ReferencesLonsdale, A. J., and North, A. C. (2011). Why do we listen to music? A uses and gratifications analysis. British Journal of Psychology(London, England : 1953), 102(1), 108–34. https://doi.org/10.1348/000712610X506831

Oliver, M. B. (2008). Tender affective states as predictors of entertainment preference. Journal of Communication, 58(1), 40–61.https://doi.org/10.1111/j.1460-2466.2007.00373.x


Emotions are related to other things as well

Why we choose to consume some kind of content?

One of the main reasons why people consume music (Lonsdale, 2011) and films (Oliver,2008) is emotion regulation.

ReferencesLonsdale, A. J., and North, A. C. (2011). Why do we listen to music? A uses and gratifications analysis. British Journal of Psychology(London, England : 1953), 102(1), 108–34. https://doi.org/10.1348/000712610X506831

Oliver, M. B. (2008). Tender affective states as predictors of entertainment preference. Journal of Communication, 58(1), 40–61.https://doi.org/10.1111/j.1460-2466.2007.00373.x


Bottomline

Ratings alone are unlikely to capture the user preferences and decision-making.


Table of Contents

Introduction






Conclusion


Personality, Mood and Emotions












Personality

• What is personality?• accounts for individual differences ( = explains the variance in users) in our enduring

emotional, interpersonal, experiential, attitudinal, and motivational styles


Models of Personality - Big5

• The five factor model (FFM) – Big5:• Extraversion• Agreeableness• Conscientousness• Neuroticism• Openness (to new experiences)

The inverse of Neuroticism is sometimes referred to as Emotional Stability,

ReferencesMcCraMcCrae, R. R., and John, O. P. (1992). An Introduction to the Five-Factor Model and its Applications. Journal of Personality,60(2), p175 – 215.


Measuring the FFM

• Extensive questionnaires (from 5 to several 100s questions)• BFI: 44 questions• TIPI : 10 questions• NEO-IPIP: 300 questions

• For each user u a five tuple b = (b1, b2, b3, b4, b5)


Emotion vs. Mood vs. Sentiment

Let’s clear some terminology

• Affect : umbrella term for describing the topics of emotion, feelings, and moods• Emotion:

• brief in duration• consist of a coordinated set of responses (verbal, physiological, behavioral, and neural

mechanisms)• triggered

• Mood:• last longer• less intense than emotions• no trigger

• Sentiment:• towards an object• positive/negative


Models of Emotions

• Emotions are complex human experiences• Evolutionary based• Several definitions, we take with simple models, easy to incorporate in computers:

• Basic emotions• Dimensional model


Basic Emotions

• Discrete classes model• Different sets• Darwin: Expression of emotions in man and animal

• Ekman definition (6 + neutral):• Happiness• Anger• Fear• Sadness• Disgust• Surprise


Dimensional model of Emotions

Three continuous dimensions

• Valence/Pleasure (positive-negative)

• Arousal (high-low )• Dominance (high-low )

Each emotion is a point in the VAD space

Self-Assessment Manikin (SAM)

ReferencesBradley, M. M., and Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. Journal ofBehavior Therapy and Experimental Psychiatry, 25(1), 49–59.




• Valence/Pleasure (positive-negative)• Arousal (high-low )

• Dominance (high-low )







• Valence/Pleasure (positive-negative)• Arousal (high-low )• Dominance (high-low )



















Table of Contents

Introduction






Conclusion


Personality Detection

Our online behaviour is influenced by our personality.

Hence, our traces in social media should reflect our personality.

It is enough to acquire personality once.












Kosinki - personality from FB

• personality prediction from Facebook

ReferencesKosinski, M., Stillwell, D., and Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior.Proceedings of the National Academy of Sciences of the United States of America, 110(15), 5802–5.https://doi.org/10.1073/pnas.1218772110


Kosinki - personality from FB

Selected most predictive likes for openness


Personality from Instagram

• N=113 (AMT)• 22398 pictures• BFI• features

• low-level image features(Hue-Value-Saturation)

• filters• presence of people

ReferencesSkowron, M., Ferwerda, B., Tkalčič, M., and Schedl, M. (2016). Fusing Social Media Cues : Personality Prediction from Twitter andInstagram. WWW’16 Companion, 2–3. https://doi.org/10.1145/2872518.2889368


Unobtrusive Personality Detection wo Social Media

• When users don’t want to disclose their social media activity?• Relationship between personality traits and disclosure behavior• 126 subjects• Facebook simulation web app• BFI 44 items (1-5)

4. RESULTSTo find the relationship between personality traits and

disclosure behavior, we dichotomized the responses of thedisclosure scale (To everybody, To friends only, Custom set-ting, Don’t know the setting, or Don’t disclose). Althoughwe asked participants for their disclosure setting, providinga third-party application access to one’s profile disregardsthat. An application will have access to the sections thata user granted access to, regardless of the disclosure settingin the profile. Hence, we recoded the responses ”To every-body,” ”To friends only,” ”Custom setting,” ”Don’t know thesetting,” to 1, as this means that participants had somethingfilled in. The ”Don’t disclose” responses were recoded as 0.

We performed a correlation analysis to indicate the rela-tionship between personality traits and disclosure behavior(see Table 2). Point-biserial correlation (r ε [-1,1]) is re-ported as the correlation coefficient. 1 We discuss the resultsrelated to each personality trait below.

Openness to experience. The openness to experiencefactor correlates with several items in the ”About” section.We found negative correlations with the ”Current city” (r=-.24, p=.02), ”Hometown” (r=-.25, p=.01), ”Mobile phone”(r=-.22, p=.03), ”Website” (r=-.22, p=.03), and ”Address”(r=-.24, p=.02). Additionally, we found a relationship ofopenness to experience with ”Birth date” (r=-.018, p=.08).The negative relationship between openness to experienceand disclosing behavior, indicate that those scoring high inthis personality dimension are less prone to disclose the in-formation of the respective items.

Conscientiousness. For the conscientiousness personalitytrait we found some relationships with items in the ”About”section. We found correlations with ”Current city” (r=-.20,p=.05), ”Hometown” (r=-.18, p=.07), and ”Birth date” (r=-.018, p=.07). Additionally, we found a correlation with the”Other” item in the ”Like” section (r=-.19, p=.06). Also forthe conscientiousness personality trait, a negative relation-ship was found for disclosing behavior. This indicates thatconscientious participants indicated to be less likely to dis-close such information in their profile.

Extraversion. Significant correlations were found inthe ”About” section and extraversion. We found correla-tions with ”Email” (r=.23, p=.02), and ”Birth date” (r=-.22,p=.03). Additionally, we found several positive correlationswith items in the ”Like” section and extraversion: ”Restau-rant” (r=.22, p=.03), ”Games” (r=.18, p=.08), ”Activities”(r=.21, p=.04), ”Interests” (r=.17, p=.09), ”Food” (r=.24,p=.02), and ”Clothing” (r=.19, p=.06). Except for emailand birth date disclosure, the items show a positive rela-tionship with extraversion. This indicate that in generalextraverts are more inclined to disclose this kind of informa-tion in their profile.

Agreeableness. The only correlation we found with theagreeableness personality factor is with ”Places lived” in the”About” section (r=-.20, p=.04). Agreeable participants in-dicated that the are less likely to disclose the places wherethey have lived before.

1The magnitude of the reported correlations are commonlyseen in personality related research [4, 6, 7, 11, 14, 12].

O C E A N4 Current city -.24* -.20ˆ -.08 -.08 .015 Hometown -.25* -.18ˆ -.08 -.13 -.056 Places lived -.12 -.12 -.08 -.20* -.017 Mobile phone -.22* -.12 -.01 -.05 .108 Website -.22* .01 .16 .02 -.169 Email -.16 .09 -.23* .13 -.1310 Address -.24* -.02 .14 -.04 -.1511 Birth date -.18ˆ -.18ˆ -.22* -.12 .17ˆ32 Restaurant .03 -.06 .22* -.06 .0933 Games .10 .01 .18ˆ .02 -.1334 Activities .05 .03 .21* .06 -.0835 Interests .09 -.04 .17ˆ -.06 -.0537 Foods .01 -.18 .24* .01 -.1138 Clothing -.05 -.06 .19ˆ .01 -.0940 Other -.05 -.19ˆ .08 -.09 .02

Note. ˆp<0.1, *p<0.05

Table 2: Correlation Matrix of the profileitems disclosure against the personality traits:(O)penness, (C)onscientiousness, (E)xtraversion,(A)greeableness, (N)euroticism. Only items thatshow significant levels of p<0.1 are reported.

Neuroticism. A correlation was found between ”Birthdate” and neuroticism (r=.17, p=.09). The positive coef-ficient indicate a positive relationship with disclosing birthdate and the neuroticism trait. In other words, neurotic par-ticipants indicated that they are more likely to disclose thebirth date in their user profile.

5. PERSONALITY PREDICTIONAs we found significant correlations between personality

traits and disclosure behavior, we explored personality pre-diction based on disclosure behavior. We trained a 10-foldcross-validation regression model with 10 iterations by us-ing the Radial Basis Function. To indicate the differencesbetween the predicted and observed values, we report theroot-mean-square error (RMSE; see Table 3). The RMSE ofeach personality trait relates to an [1,5] scale.

Personality RMSE 1 2Openness to experience 0.73 0.73 0.69Conscientiousness 0.73 0.69 0.76Extraversion 0.99 0.95 0.88Agreeableness 0.73 0.74 0.79Neuroticism 0.83 0.95 0.85

Table 3: Personality prediction with the root-mean-square error (RMSE). Left RMSE column shows theresults of the current study. Columns numbered 1and 2 show RSME scores of Ferwerda et al. [4] andQuercia et al. [12] respectively.

To see how well our prediction performs, we compared ourresults with prior work of Ferwerda et al. [4], and Quercia etal. [12], as they used a similar approach for their analyses.Ferwerda et al. [4] extracted personality using characteristicsof Instagram (e.g., how users apply filters), and Quercia etal. [12] uses Twitter users’ characteristics (e.g., popularity,highly read; see Table 3). By disregarding content and only

ReferencesFerwerda, B., Schedl, M., and Tkalčič, M. (2016). Personality Traits and the Relationship with ( Non- ) Disclosure Behavior onFacebook. WWW’16 Companion. https://doi.org/10.1145/2872518.2890085


Table of Contents

Introduction






Conclusion


Personality for mood regulation

• high on openness, extraversion, and agreeableness more inclined to listen to happymusic when they are feeling sad.

• high on neuroticism listen to more sad songs when feeling disgusted (neuroticpeople choose to increase their level of worry)

ReferencesFerwerda, B., Schedl, M., and Tkalcic, M. (2015). Personality and Emotional States : Understanding Users ’ Music Listening Needs. InA. Cristea, J. Masthoff, A. Said, and N. Tintarev (Eds.), UMAP 2015 Extended Proceedings.


Personality and music browsing styles

• personality is correlated with music browsing styles

ReferencesFerwerda, B., Yang, E., Schedl, M., and Tkalčič, M. (2015). Personality Traits Predict Music Taxonomy Preferences. In Proceedings ofthe 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’15 (pp. 2241–2246).https://doi.org/10.1145/2702613.2732754


Personality as user similarity

• new user problem• N = 52• images = 70• neighborhood-based RS: Euclidian distance −1

ReferencesTkalčič, M., Kunaver, M., Košir, A., and Tasič, J. (2011). Addressing the new user problem with a personality based user similaritymeasure. In F. Ricci, G. Semeraro, M. de Gemmis, P. Lops, J. Masthoff, F. Grasso, J. Ham (Eds.), Joint Proceedings of the Workshopon Decision Making and Recommendation Acceptance Issues in Recommender Systems (DEMRA 2011) and the 2nd Workshop on UserModels for Motivational Systems: The affective and the rational routes to persuasion (UMMS 2011).


Personality in Matrix Factorization

• in (Elahi et al., 2013) and (Fernández-Tobías, 2016)• injection of personality factors in MF as additional (latent) features (a la SVD++)

rui = qi (pu +∑

a∈A(u)

ya)

• personality u = (2.3, 4.0, 3.6, 5.0, 1.2) maps to A(u) = {ope2, con4, ext4, agr5,neu1}.

• (Fernández-Tobías, 2016) is a very comprehensive paper• iMF = (Hu et al., 2008)

ReferencesElahi, M., Braunhofer, M., Ricci, F., and Tkalčič, M. (2013). Personality-based active learning for collaborative filtering recommendersystems. In M. Baldoni, C. Baroglio, G. Boella, and O. Micalizio (Eds.), AI*IA 2013: Advances in Artificial Intelligence (pp. 360–371).

Fernández-Tobías, I., Braunhofer, M., Elahi, M., Ricci, F., and Cantador, I. (2016). Alleviating the new user problem in collaborativefiltering by exploiting personality information. User Modeling and User-Adapted Interaction, 26(2), 1–35.https://doi.org/10.1007/s11257-016-9172-z


UMUAI Special Issue

• UMUAI, June 2016• Special Issue on Personality in Personalized Systems• Issue Editors:

• Marko Tkalcic,• Daniele Quercia,• Sabine Graf

ReferencesTkalčič, M., Quercia, D., and Graf, S. (2016). Preface to the special issue on personality in personalized systems. User Modeling andUser-Adapted Interaction, 26(2–3), 103–107. https://doi.org/10.1007/s11257-016-9175-9


Table of Contents

Introduction






Conclusion


Multimodal Emotion Detection

• Emotions consist of a coordinated set of responses (verbal, physiological,behavioral, and neural mechanisms)

• These responses can be used to measure the emotions.• Affective Computing


Overview

• modalities• audio• language• visual - videos of faces (action units)• physiology• brain signals

• target emotion• discrete (classification)• continuous (regression)

ReferencesSchuller, B. W. (2016). Acquisition of Affect. In M. Tkalčič, B. De Carolis, M. de Gemmis, A. Odić, and A. Košir (Eds.), Emotions andPersonality in Personalized Services: Models, Evaluation and Applications (pp. 57–80). Cham: Springer International Publishing.


Unobtrusive Emotion Detection from Facial Videos

• 2 datasets:• Posed (Kanade Cohn)• Spontaneous (LDOS-PerAff-1)

• video streams of facial expressions asresponses to visual stimuli

• distinct classes• Gabor features• kNN classifier

• Posed dataset: accuracy = 92 %• Spontaneous dataset: accuracy = 62%• Reasons for bad results:

• Weak learning supervision• Non optimal video acquisition (face

rotation, occlusions, changing lightning. . . )

• Non extreme facial expressions


Sad Reality


Sad Reality


Table of Contents

Introduction






Conclusion


Model of usage of emotions in Recommender Systems

ReferencesTkalčič, M., Košir, A., Tasič, J., and Kunaver, M. (2011). Affective recommender systems: the role of emotions in recommendersystems. In A. Felfernig, L. Chen, M. Mandl, M. Willemsen, D. Bollen, and M. Ekstrand (Eds.), Joint proceedings of the RecSys 2011Workshop on Human Decision Making in Recommender Systems (Decisions@RecSys’11) and User-Centric Evaluation of RecommenderSystems and Their Interfaces-2 (UCERSTI 2) affiliated with the 5th ACM Conference on Recommender (pp. 9–13).


Affective User Modeling

• Multimedia content ELICITS (induces) emotions• Underlying assumption: users differ in their preferences for emotions


Affective User Modeling

ReferencesTkalčič, M., Burnik, U., and Košir, A. (2010). Using affective parameters in a content-based recommender system for images. UserModeling and User-Adapted Interaction, 20(4), 279–311. doi:10.1007/s11257-010-9079-z

Tkalčič, M., Odić, A., Košir, A., and Tasič, J. (2013). Affective labeling in a content-based recommender system for images. IEEETransactions on Multimedia, 15(2), 391–400. https://doi.org/10.1109/TMM.2012.2229970


Emotions as feedback

• video-on-demand scenario• usage of hesitation as feedback• 4 recommendations, 1 selection

• control group: recommend similar• hesitation group: recommend similar/diverse

• quality of experience (QoE) is improved when hesitation is taken into account

ReferencesVodlan, T., Tkalčič, M., and Košir, A. (2015). The impact of hesitation, a social signal, on a user’s quality of experience in multimediacontent retrieval. Multimedia Tools and Applications. doi:10.1007/s11042-014-1933-2


Table of Contents

Introduction






Conclusion


Conclusion

• emotions and personality account for differences in user behavior• research is still scattered


Open Issues

• lack of awareness• this talk will hopefully help

• lack of data• mostly small-scale data gathered through user studies• exceptions:

• Movielens+personality• myPersonality

• unobtrusive annotation of content• subtitles?

• privacy


Technology

Affective Personalization - from Psychology to Algorithms