Session, focus and engagement

Session, focus and engagement

Mounia Lalmas

Yahoo! Research Barcelona

mounia@acm.org

A bit about myself

1999-2008: Lecturer (assistant professor) to Professor at Queen Mary, University of London

2008-2010 Microsoft Research/RAEng Research Professor at the University of Glasgow (and lived outside London)

2011- Visiting Principal Scientist at Yahoo! Research Barcelona

Research topics XML retrieval and evaluation (INEX) Quantum theory to model interactive information retrieval Aggregated search Bridging the digital divide Models and measures of user engagement

Message and Outline

Interaction and search Beyond result relevance Beyond search session

Towards “engagement”

1. Motivations2. Engagement3. Future directions

1. Outline

1. Motivations• Relevance in multimedia search• Relevance in focused retrieval• Online multi-tasking

2. Engagement

1. Future directions

Information Retrieval Over Query Sessions

Retrieval Models & Ranking: How to analyze/model/predict user interactions and use these findings to improve retrieval performance? How can we adapt ranking/retrieval models and IR theory in the light of a sequence of user interactions.

Evaluation & Test Collections: How can we evaluate retrieval system performance over entire query sessions? How can we build reusable test collections to study this IR task? How can we model/simulate user interactions over a session?

User Interaction & Interfaces: How can we model user interactions so we can predict and improve the user experience over sessions? How can we design and perform user studies that reveal new information about users? How can we make use of implicit feedback from users?

Multimedia search activities often driven by entertainment needs, not by information needs

Relevance in multimedia search

M. Slaney, Precision-Recall Is Wrong for Multimedia, IEEE Multimedia Magazine, 2011

Relevance in focused retrievalRelevance in context

Table of Content Focused retrieval is about putting results (element, fact, passage) in context, to understand and trust them

Courtesy Jaap Kamps, Zoltan Szlavik, Norbert Goevert

Beyond search session

On month browsing data, sample of Yahoo! sites

On month browsing data, sample of sites

(INT=Yahoo site,EXT=non Yahoo site)

Courtesy of Janette Lehmann

users spend more and more of their online session multi-tasking, e.g. emailing, reading news, searching for information ONLINE MULTI-TASKING navigating between sites, using browser tabs, etc seamless integration of social networks platforms into many services

Interactive IR …

P Ingwersen, Human Aspects in IR, ESSIR 2011.

2. Outline

1. Motivations

2. (User) Engagement• Definition• Characteristics• Measuring• Models

1. Future directions

User Engagement – connecting three sides User engagement is a quality of user experience that emphasizes the positive aspects of

interaction – in particular the fact of being captivated by the technology.

Successful technologies are not just used, they are engaged with.

user feelings: happy, sad,excited, bored, …

The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource

user interactions: click, readcomment, recommend, buy, …

user mental states: concentrated,challenged, lost, interested …

Characteristics of user engagement (I)

S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.

Characteristics of user engagement (II)

S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.

The four I’s

Measuring Engagement, Forrester Research, June 2008

Measuring user engagement

Objective measures – Online activities

Proxy of user engagement

Models of user engagementOnline sites differ concerning their engagement!

GamesUsers spend much time per visit

SearchUsers come frequently and do not stay long

Social mediaUsers come frequently and stay long

SpecialUsers come on average once per time considered

NewsUsers comeperiodically

ServiceUsers visit site, when needed

Is it possible to model these differences?

Data and Metrics

Interaction data, 2M users, July 2011, 80 US sites

Popularity #Users Number of distinct users

#Visits Number of visits

#Clicks Number of clicks

Activity ClickDepth Average number of page views per visit.

DwellTimeA Average time per visit

Loyalty ActiveDays Number of days a user visited the site

ReturnRate Number of times a user visited the site

DwellTimeL Average time a user spend on the site.

Diversity in user engagement

Users and Loyalty Sites have different user groups Proportion of user groups is site-

dependent

Time and Popularity Site engagement can be periodic

or contains peaks

Engagement of a site depends on users and time

mail, social media

shopping, entertainment

media(special events)

daily activity,navigation

media,entertainment

Methodology

General models User-based models Time-based modelsDimensions

8 metrics5 user groups8 metrics per user group

weekdays, weekend8 metrics per time span

#Dimensions 8 40 16

Kernel k-means with Kendall tau rank correlation kernel

Nb of clusters based on eigenvalue distribution of kernel matrixSignificant metric values with Kruskal-Wallis/Bonferonni

#Clusters (Models) 6 7 5

Analysing cluster centroids = models

Models of user engagement

• 6 general models

• Popularity, activity and loyalty are independent from each other

• Popularity and loyalty are influenced by external and internal factors e.g. frequency of publishing new

information, events, personal interests

• Activity depends on the structure of the site

Models based on engagement metrics

interest-specific

e-commerce,configuration

periodicmedia

Models of user engagement

User-based [7 models] Models based on engagement per

user group

Time-based [5 models] Models based on engagement

over weekdays and weekend

Models based on engagement metrics, user and time

navigation game, sporthobbies,interest-specific

daily news

Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model

The groups of models describe different aspects of engagement, i.e. they are independent from each other

Recap & NextUser engagement is complex and standard

metrics capture only a part of itFirst step towards a taxonomy of models of user

engagement … and associated metrics

NextInteraction between modelsInteraction between sites (multi-tasking)User demographics, time of the day, geo-location, etc

J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.

3. Outline

1. Motivations

2. Engagement

1. Future directions1. The three sides of user engagement2. Interactive IR3. Towards engagement

+ layout +links+ saliency + content

user engagement within and across site Measurements and methodologies

+ online analytics metrics (dwell time, CTR, …) + complex networks metrics

+ questionnaires, surveys, … + crowd-sourcing

+ biometrics (eye tracking, mouse tracking, …)

Goals + Models of user engagement + Metrics of user engagement

The three sides

+ emotional

+ cognitive

+ behavioral

Let us revisit … connecting three sides

Let us revisit … Interactive IR

P Ingwersen, Human Aspects in IR, ESSIR 2011.

session, interaction, multi-tasking, network, search, relevance, …

•I Aapakis, K Athanasakos, J Jose, A comparison of general vs personalised affective models for the prediction of topical relevance, SIGIR 2010.•J Huang, R White, S Dumais, No clicks, no problem: using cursor movements to understand and improve search, CHI 2011.• P Ingwersen & K Järvelin, The turn: integration of information seeking and retrieval in context, 2005.

TOWARDS ENGAGEMENT

Information Retrieval Over Query Sessions

Retrieval Models & Ranking: How to analyze/model/predict user interactions and use these findings to improve retrieval performance? How can we adapt ranking/retrieval models and IR theory in the light of a sequence of user interactions.

Evaluation & Test Collections: How can we evaluate retrieval system performance over entire query sessions? How can we build reusable test collections to study this IR task? How can we model/simulate user interactions over a session?

User Interaction & Interfaces: How can we model user interactions so we can predict and improve the user experience over sessions? How can we design and perform user studies that reveal new information about users? How can we make use of implicit feedback from users?

TOWARDS ENGAGEMENT

beyond session and relevance

Thank you

mounia@acm.org

www.dcs.gla.ac.uk/~mounia

TOWARDS ENGAGEMENT

beyond session and relevance