Mobile Attention Management Identifying and Harnessing Factors that Determine Reaction to Mobile...

Mobile Attention Management Identifying and Harnessing Factors that Determine

Reaction to Mobile Notifications

Dr Veljko Pejović Faculty of Computer and Information Science

University of Ljubljana, Slovenia

FreeUniversityofBozen-Bolzano,September2016

Mobile Notifications •  Mobile phone is the most

direct point of contact •  Increasingly interactive lives •  For recipients, notifications

are a means of real-time information awareness

•  For senders, a way to initiate remote communication

Mobile Notifications •  If not managed properly:

–  Interrupt ongoing tasks resulting in poor completion rates, errors

–  Interfere with users’ lifestyle –  Arrive at socially inappropriate

moments –  Increase frustration –  May result in negative brand perception –  Get ignored

Design and develop a system for intelligent notification scheduling on a mobile device

Goal

Premise: notification timing

is the key!

Identify opportune moments to deliver information to a mobile user

But first…

Sensing Interruptibility •  Hypothesis: sensed context reveals interruptibility

Sensing Interruptibility •  Hypothesis: sensed context reveals interruptibility •  Measuring interruptibility

–  Reaction presence ? t

Sensing Interruptibility •  Hypothesis: sensed context reveals interruptibility •  Measuring interruptibility

–  Reaction presence –  Time to reaction

ttreaction

Sensing Interruptibility •  Hypothesis: sensed context reveals interruptibility •  Measuring interruptibility

–  Reaction presence –  Time to reaction –  Sentiment I hate you

t

Sensing Interruptibility •  Test with data:

–  SampleMe: Android experience sampling app •  (Reactions to) notifications •  Sensor data •  20 users, two weeks, approx

eight notifications per day

V. Pejovic and M. Musolesi, InterruptMe: Designing Intelligent Prompting Mechanisms for Pervasive Applications, ACM UbiComp’14, Seattle, WA, USA

Modeling Interruptibility

Context at the time of

notifying (tN)

Classifiers: -  Batch, online -  Naïve

Bayesian, Tree-based, Boosting

Response presence

Response time

Sentiment

•  Findings: –  Reaction presence

•  Precision is important

Classifier Precision Recall

AdaBoost 0.64 0.41

BayesNet 0.58 0.37

NaïveBayes 0.57 0.52

Baseline 0.39 0.38

Modeling Interruptibility

•  Findings: –  Reaction presence

•  Precision is important –  Time to reaction

•  Coarse-grain inference

00.10.20.30.40.50.60.7

5 1015202530354045505560

Prec

isio

n

Answer threshold [min]

Baseline Naïve Bayes Bayes Net Boosting

Modeling Interruptibility

Modeling Interruptibility •  Findings:

–  Reaction presence •  Precision is important

–  Time to reaction •  Coarse-grain inference

–  Sentiment •  Difficult to identify very

much suitable moments to interrupt

0

0.2

0.4

0.6

0.8

1

1 2 3

Prec

isio

n

Min acceptable sentiment [1-"a bit", 2-"some", 3-"very much"]

Baseline Naïve Bayes Bayes Net Boosting

Design and develop a system for intelligent notification scheduling on a mobile device

Goal

Towards real-time interruptibility recognition on

the mobile!

InterruptMe •  Interruptibility lib:

–  Connect sensed context with user interruptibility

–  Build personalised or shared models

–  Ensure persistence

ApplicaHon

AndroidOS

InterruptMe

NoHficaHon

Sensorstream

InterrupHonmanager

Machinelearninglib

Predictedoutcome

Sensinglib

Sensorfeatures

Decision

Outcomefeedback

Trainclassifier

InterruptMe publicclassNo9fica9onServiceextendsServiceimplementsIntelligentTriggerReceiver{IntelligentTriggerManagerinterrupHonMngr;publicvoidonCreate(){interrupHonMngr=IntelligentTriggerManager.getTriggerManager(this); }publicvoidonTriggerReceived(Stringa_triggerID,ArrayList<LearnerResultBundle>a_bundles){//…sendanoHficaHonusingNoHficaHonManager}

//…incaseuserrespondstoanoHficaHoninterrupHonMngr.trainLearnerFromFeedback(Constants.MOD_INTERRUPTIBILITY,

currentTriggerNo,"yes");

InterruptMe Evaluation •  SampleMe + InterruptMe:

–  Context-aware experience sampling –  N-of-1 trial

(random vs intelligent notification triggering)

–  Ten users for one month

InterruptMe Evaluation •  Faster response to

InterruptMe-based notifications

0

0.2

0.4

0.6

0.8

1

0 50 100 150 Response time [min]

Random InterruptMe

InterruptMe Evaluation •  Faster response to

InterruptMe-based notifications

•  No significant difference in the sentiment towards being interrupted

Very much Some A bit Not at all

0.00

0.20

0.40

0.60

0.80

1.00

CDF of min reported sentiment

Random InterruptMe

InterruptMe Evaluation •  Faster response to

InterruptMe-based notifications

•  No significant difference in the sentiment towards being interrupted

•  Amount of interruptibility 0

0.2 0.4 0.6 0.8

1 1.2 1.4

0 1 2 3 4 5 6 7 8 9 10

Sent

imen

t

Number of notifications in previous two hours

InterruptMe Evaluation •  Isolated InterruptMe

notifications received better than random notifications

Very much Some A bit Not at all 0.00

0.20

0.40

0.60

0.80

1.00

CDF of min reported sentiment

Random InterruptMe

InterruptMe – the first context-aware real-time mobile notification management system that leads to faster responses and better received

notifications.

Conclusion

However…

… no significant effects of notification

scheduling on the usage of a

behavioural change intervention app

Conclusion

What Did We Miss? •  We had a limited dataset – only a single

type of a notification

•  We did not take content into account

•  We overlooked the role of a sender-receiver relationship

NotifyMe Dataset •  Sensed context •  Reaction to a notification •  Notification data

–  Category –  Sender ID

•  User preferences: –  Where and when would you like

to receive notifications with similar content A. Mehrotra, M. Musolesi, R. Hendley and V. Pejovic,

Designing Content-Driven Intelligent Notification Mechanisms for Mobile Applications, ACM UbiComp’15, Osaka, Japan.

NotifyMe Dataset •  Participants: 35 •  Study duration: 3 weeks •  Notification samples: 70,000 (approx) •  Questionnaire responses: 4,069

NotifyMe Dataset •  Application name and

notification title extracted –  Manual app name

classification

NotifyMe Dataset •  Recipient-sender

relationship –  Users classified their

notifications to: Family, Friend, Work, Other

Chat and email divided into eight

sub-categories

The Impact of Content/Sender •  Notification click

count differs between application types (i.e. content type) and sender-receiver relations

Predicting Opportune Moments •  Moments in which a user reacts quickly and accepts a

notification •  Predictor flavours (depending on features):

1.  User-defined rules and context data 2.  Context data 3.  Information type and context data 4.  Social circle, information type and context data

•  Machine learning algorithms: –  Naïve Bayes, Ada Boost, Random Forest

Prediction Results

By using information type and social circle we were able to predict the acceptance of a notification within 10 minutes from its arrival time with an average sensitivity of 70% and a specificity of 80%.

We identified a number of factors that impact the acceptance of a notification. However,

reactions to notifications are diverse. How exactly will a user react to a notification?

Detailed Analysis of User Reactions

swipe

click

seen time decision time

Seen Time: time from the notification arrival until the notification was seen by the user

A. Mehrotra, V. Pejovic, J. Vermeulen, R. Hendley and M. Musolesi, My Phone and Me: Understanding People’s Receptivity to Mobile Notifications

ACM CHI’16, San Jose, CA, USA.

Detailed Analysis of User Reactions

swipe

click

seen time decision time

Decision Time: time from the moment a user saw a notification until the time they acted upon it (click or dismiss) A. Mehrotra, V. Pejovic, J. Vermeulen, R. Hendley and M. Musolesi,

My Phone and Me: Understanding People’s Receptivity to Mobile Notifications ACM CHI’16, San Jose, CA, USA.

My Phone And Me App •  Automated logging:

–  Notification time of arrival, seen, removal –  Notification response –  Notification details (title, app) –  Alert type –  Context (physical activity, location, etc.)

•  Experience sampling: –  Questions about how the notification was

handled, sender-receiver relationship, cognitive context, personality

A B

C D

Key Results – Reaction Times •  Alert modality, current task type and complexity impact

the seen time –  We are faster to react when engaged in a complex task

•  Relationship with the sender impacts the decision time –  We are faster to decide on a message that comes from a close

person (partners, immediate family members), than on a message that come from a less familiar person (extended family, service providers)

Key Results – Disruption •  Sender/receiver relationship:

messages from subordinates and system messages are considered as most disruptive. Extended family members are considered as the least disruptive.

•  Task engagement: more disruptive if it arrives when the user is in the middle of or finishing a task. Perceived disruption increases with the complexity of an ongoing task.

Confirmed in another study: V. Pejovic, A. Mehrotra and M. Musolesi,

Investigating The Role of Task Engagement in Mobile Interruptibility,

Smarttention workshop with ACM Ubicomp’15.

Key Results – Personality •  Data is limited as only 11 user

have fully completed the personality test

•  Hints towards extroverts being more inclined to be disrupted by a notification

Linear regression with the average disruption as a dependent variable and

personality traits as independent variables

How does a thought get disrupted?

Theory of Multitasking Resources •  Perceptual and motor •  Cognitive

–  Procedural memory –  Declarative memory

Mechanisms •  Resource use is exclusive

– one task at a time per resource

•  Multiple problem threads run in parallel, but processing is still serial [Salvucci2008]

Theory of Multitasking •  Interference when two or more threads ask for the same

resource at a time

Example from [Borst2010]

Theory of Multitasking •  Complex tasks require problem state saving/retrieving

Example from [Borst2015]

Practical Implications on Mobile Attention Management

•  Interruptions are more disruptive if they require problem state switching

•  Make them less disruptive by interrupting: –  At moments when a task is not fully active (e.g. just starting, or

just finished) –  At moments when a task does not require a problem state –  At moments when a user is working on a task that is well

practiced, a routine

Can we automatically infer task engagement?

G. Urh and V. Pejovic, TaskyApp: Inferring Task Engagement via Smartphone Sensing

Ubittention workshop with ACM UbiComp’16, Heidelberg, Germany.

TaskyApp •  Can smartphones sense that their users

are busy (in an office setting)? •  TaskyApp data collection app

–  Background sensing of device movement, ambient sound, collocation with other devices

–  Data labelling via experience sampling and retroactive assisted labelling

–  Recruited eight office workers for five weeks •  232 labelled instances (3035 unlabelled) •  Most data between 8am and 6pm

TaskyApp – Data Analysis •  Linear regression fit with sensed

features as independent variables and task difficulty (1-5 on a Likert scale) as a dependent variable –  Movement data gives the most

informative features –  The regression explains only a small

part of the data

TaskyApp – Data Analysis •  Classify a task engagement moment as either “easy” or

“difficult” depending on the sensed features –  We experimented with different classifiers but Naïve Bayes

seems to work best (probably due to the low amount of data) •  62.5% accuracy compared to 52.8% baseline •  Also, leads to “favourable” errors –

few difficult tasks predicted as easy

Task Engagement Inference •  Even in a restricted office setting smartphone-based task

inference is difficult •  Movement features seem to be the most informative •  Next step – wearables

–  Sense heart rate and skin temperature

Conclusions •  Notification success depends on the content, sender-

receiver relationship, channel, context in which a user is •  Our reaction and the disturbance levels depend on the

alert type, sender, existing task engagement •  User behaviour is highly personalised (our personalised

data have performed better than the general ones) •  Personality may play a role in the way we perceive

notifications •  Task engagement inference requires additional sensors

References •  [Pejovic2014] V. Pejovic and M. Musolesi, InterruptMe: Designing Intelligent Prompting Mechanisms for

Pervasive Applications, ACM UbiComp’14, Seattle, WA, USA •  [Mehrotra2015] A. Mehrotra, M. Musolesi, R. Hendley and V. Pejovic, Designing Content-Driven Intelligent

Notification Mechanisms for Mobile Applications, ACM UbiComp’15, Osaka, Japan. •  [Pejovic2015] V. Pejovic, A. Mehrotra and M. Musolesi, Investigating The Role of Task Engagement in Mobile

Interruptibility, Smarttention workshop with ACM Ubicomp’15. •  [Mehrotra2016] A. Mehrotra, V. Pejovic, J. Vermeulen, R. Hendley and M. Musolesi, My Phone and Me:

Understanding People’s Receptivity to Mobile Notifications, ACM CHI’16, San Jose, CA, USA. •  [Urh2016] G. Urh and V. Pejovic, TaskyApp: Inferring Task Engagement via Smartphone Sensing, Ubittention

workshop with ACM UbiComp’16, Heidelberg, Germany. •  [Salvucci2008] Salvucci, Dario D., and Niels A. Taatgen. "Threaded cognition: an integrated theory of

concurrent multitasking." Psychological review 115.1 (2008): 101. •  [Borst2010] Borst, Jelmer P., Niels A. Taatgen, and Hedderik van Rijn. "The problem state: a cognitive

bottleneck in multitasking." Journal of Experimental Psychology: Learning, memory, and cognition 36.2 (2010): 363.

•  [Borst2015] Borst, Jelmer P., Niels A. Taatgen, and Hedderik van Rijn. "What Makes Interruptions Disruptive?: A Process-Model Account of the Effects of the Problem State Bottleneck on Task Interruption and Resumption." CHI’15, 2015.

Collaborators Abhinav Mehrotra University of Birmingham UK

Dr Mirco Musolesi University College London UK

Gašper Urh University of

Ljubljana Slovenia

Prof Robert Hendley University of Birmingham

UK

Dr Jo Vermeulen University of Calgary

Canada

Thank you! Veljko Pejovic

University of Ljubljana Veljko.Pejovic@fri.uni-lj.si

http://lrss.fri.uni-lj.si/Veljko @veljkoveljko

bitbucket.org/veljkop/intelligenttrigger github.com/vpejovic/MachineLearningToolkit